search

ail-project / ail-framework

AIL framework - Analysis Information Leak framework
GNU Affero General Public License v3.0
561 stars 80 forks source link

Current Issues on Ubuntu 24.04 #199

Open SteveClement opened 8 months ago

SteveClement commented 8 months ago

pycld3 currently fails with Python 3.11(.7), one workaround, use 3.10

Another workaround (cython):

source ./AILENV/bin/activate
git clone https://github.com/bsolomon1124/pycld3.git
cd pycld3
pip install cython
USE_CYTHON=1 python setup.py install

Error

  Building wheel for pycld3 (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [36 lines of output]
      /home/ail-stable/ail-framework/AILENV/lib/python3.11/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
      !!

              ********************************************************************************
              The license_file parameter is deprecated, use license_files instead.

              This deprecation is overdue, please update your project and remove deprecated
              calls to avoid build errors in the future.

              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
              ********************************************************************************

      !!
        parsed = self.parsers.get(option_name, lambda x: x)(value)
      running bdist_wheel
      running build
      Running protoc sentence.proto feature_extractor.proto task_spec.proto --cpp_out=/tmp/pip-install-xotyzcea/pycld3_7b35e5846d8c4ec8a39eb3d6a665a80d/src/cld_3/protos/
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-311
      creating build/lib.linux-x86_64-cpython-311/cld3
      copying cld3/__init__.py -> build/lib.linux-x86_64-cpython-311/cld3
      running build_ext
      building 'cld3._cld3' extension
      creating build/temp.linux-x86_64-cpython-311
      creating build/temp.linux-x86_64-cpython-311/cld3
      creating build/temp.linux-x86_64-cpython-311/src
      creating build/temp.linux-x86_64-cpython-311/src/cld_3
      creating build/temp.linux-x86_64-cpython-311/src/cld_3/protos
      creating build/temp.linux-x86_64-cpython-311/src/script_span
      x86_64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -I/usr/local/include/ -I/tmp/pip-install-xotyzcea/pycld3_7b35e5846d8c4ec8a39eb3d6a665a80d/src/ -I/tmp/pip-install-xotyzcea/pycld3_7b35e5846d8c4ec8a39eb3d6a665a80d/src/cld_3/protos/ -I/home/ail-stable/ail-framework/AILENV/include -I/usr/include/python3.11 -c cld3/pycld3.cpp -o build/temp.linux-x86_64-cpython-311/cld3/pycld3.o -std=c++11
      cld3/pycld3.cpp:250:12: fatal error: longintrepr.h: No such file or directory
        250 |   #include "longintrepr.h"
            |            ^~~~~~~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pycld3
  Running setup.py clean for pycld3
  Building wheel for pylibinjection (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [71 lines of output]
      src/pylibinjection.c: In function ‘__Pyx_InitCachedConstants’:
      src/pylibinjection.c:882:241: warning: passing argument 14 of ‘PyCode_New’ makes pointer from integer without a cast [-Wint-conversion]
        882 |   __pyx_codeobj__2 = (PyObject*)__Pyx_PyCode_New(1, 0, 4, 0, 0, __pyx_empty_bytes, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_tuple_, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_kp_s_home_glaslos_workspace_pylibinj, __pyx_n_s_detect_sqli, 25, __pyx_empty_bytes); if (unlikely(!__pyx_codeobj__2)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 25; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
            |                                                                                                                                                                                                                                                 ^~
            |                                                                                                                                                                                                                                                 |
            |                                                                                                                                                                                                                                                 int
      src/pylibinjection.c:124:72: note: in definition of macro ‘__Pyx_PyCode_New’
        124 |           PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
            |                                                                        ^~~~~
      In file included from /usr/include/python3.11/Python.h:71,
                       from src/pylibinjection.c:16:
      /usr/include/python3.11/cpython/code.h:151:33: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘int’
        151 |         PyObject *, PyObject *, PyObject *, int, PyObject *,
            |                                 ^~~~~~~~~~
      src/pylibinjection.c:882:245: warning: passing argument 15 of ‘PyCode_New’ makes integer from pointer without a cast [-Wint-conversion]
        882 |   __pyx_codeobj__2 = (PyObject*)__Pyx_PyCode_New(1, 0, 4, 0, 0, __pyx_empty_bytes, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_tuple_, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_kp_s_home_glaslos_workspace_pylibinj, __pyx_n_s_detect_sqli, 25, __pyx_empty_bytes); if (unlikely(!__pyx_codeobj__2)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 25; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
            |                                                                                                                                                                                                                                                     ^~~~~~~~~~~~~~~~~
            |                                                                                                                                                                                                                                                     |
            |                                                                                                                                                                                                                                                     PyObject * {aka struct _object *}
      src/pylibinjection.c:124:79: note: in definition of macro ‘__Pyx_PyCode_New’
        124 |           PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
            |                                                                               ^~~~
      /usr/include/python3.11/cpython/code.h:151:45: note: expected ‘int’ but argument is of type ‘PyObject *’ {aka ‘struct _object *’}
        151 |         PyObject *, PyObject *, PyObject *, int, PyObject *,
            |                                             ^~~
      src/pylibinjection.c:124:11: error: too few arguments to function ‘PyCode_New’
        124 |           PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
            |           ^~~~~~~~~~
      src/pylibinjection.c:882:33: note: in expansion of macro ‘__Pyx_PyCode_New’
        882 |   __pyx_codeobj__2 = (PyObject*)__Pyx_PyCode_New(1, 0, 4, 0, 0, __pyx_empty_bytes, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_tuple_, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_kp_s_home_glaslos_workspace_pylibinj, __pyx_n_s_detect_sqli, 25, __pyx_empty_bytes); if (unlikely(!__pyx_codeobj__2)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 25; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
            |                                 ^~~~~~~~~~~~~~~~
      /usr/include/python3.11/cpython/code.h:148:28: note: declared here
        148 | PyAPI_FUNC(PyCodeObject *) PyCode_New(
            |                            ^~~~~~~~~~
      src/pylibinjection.c: In function ‘__Pyx_CreateCodeObjectForTraceback’:
      src/pylibinjection.c:1457:9: warning: passing argument 14 of ‘PyCode_New’ makes pointer from integer without a cast [-Wint-conversion]
       1457 |         py_line,      /*int firstlineno,*/
            |         ^~~~~~~
            |         |
            |         int
      src/pylibinjection.c:124:72: note: in definition of macro ‘__Pyx_PyCode_New’
        124 |           PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
            |                                                                        ^~~~~
      /usr/include/python3.11/cpython/code.h:151:33: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘int’
        151 |         PyObject *, PyObject *, PyObject *, int, PyObject *,
            |                                 ^~~~~~~~~~
      src/pylibinjection.c:1458:9: warning: passing argument 15 of ‘PyCode_New’ makes integer from pointer without a cast [-Wint-conversion]
       1458 |         __pyx_empty_bytes  /*PyObject *lnotab*/
            |         ^~~~~~~~~~~~~~~~~
            |         |
            |         PyObject * {aka struct _object *}
      src/pylibinjection.c:124:79: note: in definition of macro ‘__Pyx_PyCode_New’
        124 |           PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
            |                                                                               ^~~~
      /usr/include/python3.11/cpython/code.h:151:45: note: expected ‘int’ but argument is of type ‘PyObject *’ {aka ‘struct _object *’}
        151 |         PyObject *, PyObject *, PyObject *, int, PyObject *,
            |                                             ^~~
      src/pylibinjection.c:124:11: error: too few arguments to function ‘PyCode_New’
        124 |           PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
            |           ^~~~~~~~~~
      src/pylibinjection.c:1443:15: note: in expansion of macro ‘__Pyx_PyCode_New’
       1443 |     py_code = __Pyx_PyCode_New(
            |               ^~~~~~~~~~~~~~~~
      /usr/include/python3.11/cpython/code.h:148:28: note: declared here
        148 | PyAPI_FUNC(PyCodeObject *) PyCode_New(
            |                            ^~~~~~~~~~
      src/pylibinjection.c: In function ‘__Pyx_AddTraceback’:
      src/pylibinjection.c:1489:13: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’}
       1489 |     py_frame->f_lineno = py_line;
            |             ^~
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pylibinjection
  Running setup.py clean for pylibinjection
  Building wheel for langid (setup.py) ... done
  Created wheel for langid: filename=langid-1.1.6-py3-none-any.whl size=1942290 sha256=55351e85cf19518dcb696fba408257c71088c7d94a1b7e71030fc95acbdfeb2c
  Stored in directory: /tmp/pip-ephem-wheel-cache-gjt1vwfa/wheels/d1/98/ca/efb42df1369f894e668d2b2f01fd9899fb780ef6863b9b80d5
  Building wheel for future (setup.py) ... done
  Created wheel for future: filename=future-0.18.3-py3-none-any.whl size=492024 sha256=4f19a087cca6e4e464490f343dff450ae0736759b389183f969c25150526ec1a
  Stored in directory: /home/ail-stable/.cache/pip/wheels/da/19/ca/9d8c44cd311a955509d7e13da3f0bea42400c469ef825b580b
  Building wheel for IPy (setup.py) ... done
  Created wheel for IPy: filename=IPy-1.1-py3-none-any.whl size=19440 sha256=2e96755c846947eaccd671bd74709b8bd5c537bed2859fcd544cf42fb63004ca
  Stored in directory: /home/ail-stable/.cache/pip/wheels/53/c9/6c/cb41f2821952f1539a00922cdcfba23ab4bd6c287160b34699
Successfully built yara-python crcmod ssdeep zmq langid future IPy
Failed to build pycld3 pylibinjection
ERROR: Could not build wheels for pycld3, pylibinjection, which is required to install pyproject.toml-based projects

Known error: https://github.com/bsolomon1124/pycld3/issues/35 But seems to be not maintained anymore.

cpython fix: https://github.com/python/cpython/pull/28968

~~

pylibinjection compile error

Workaround: use cython

Error

  Building wheel for pylibinjection (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [71 lines of output]
      src/pylibinjection.c: In function ‘__Pyx_InitCachedConstants’:
      src/pylibinjection.c:882:241: warning: passing argument 14 of ‘PyCode_New’ makes pointer from integer without a cast [-Wint-conversion]
        882 |   __pyx_codeobj__2 = (PyObject*)__Pyx_PyCode_New(1, 0, 4, 0, 0, __pyx_empty_bytes, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_tuple_, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_kp_s_home_glaslos_workspace_pylibinj, __pyx_n_s_detect_sqli, 25, __pyx_empty_bytes); if (unlikely(!__pyx_codeobj__2)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 25; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
            |                                                                                                                                                                                                                                                 ^~
            |                                                                                                                                                                                                                                                 |
            |                                                                                                                                                                                                                                                 int
      src/pylibinjection.c:124:72: note: in definition of macro ‘__Pyx_PyCode_New’
        124 |           PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
            |                                                                        ^~~~~
      In file included from /usr/include/python3.11/Python.h:71,
                       from src/pylibinjection.c:16:
      /usr/include/python3.11/cpython/code.h:151:33: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘int’
        151 |         PyObject *, PyObject *, PyObject *, int, PyObject *,
            |                                 ^~~~~~~~~~
      src/pylibinjection.c:882:245: warning: passing argument 15 of ‘PyCode_New’ makes integer from pointer without a cast [-Wint-conversion]
        882 |   __pyx_codeobj__2 = (PyObject*)__Pyx_PyCode_New(1, 0, 4, 0, 0, __pyx_empty_bytes, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_tuple_, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_kp_s_home_glaslos_workspace_pylibinj, __pyx_n_s_detect_sqli, 25, __pyx_empty_bytes); if (unlikely(!__pyx_codeobj__2)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 25; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
            |                                                                                                                                                                                                                                                     ^~~~~~~~~~~~~~~~~
            |                                                                                                                                                                                                                                                     |
            |                                                                                                                                                                                                                                                     PyObject * {aka struct _object *}
      src/pylibinjection.c:124:79: note: in definition of macro ‘__Pyx_PyCode_New’
        124 |           PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
            |                                                                               ^~~~
      /usr/include/python3.11/cpython/code.h:151:45: note: expected ‘int’ but argument is of type ‘PyObject *’ {aka ‘struct _object *’}
        151 |         PyObject *, PyObject *, PyObject *, int, PyObject *,
            |                                             ^~~
      src/pylibinjection.c:124:11: error: too few arguments to function ‘PyCode_New’
        124 |           PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
            |           ^~~~~~~~~~
      src/pylibinjection.c:882:33: note: in expansion of macro ‘__Pyx_PyCode_New’
        882 |   __pyx_codeobj__2 = (PyObject*)__Pyx_PyCode_New(1, 0, 4, 0, 0, __pyx_empty_bytes, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_tuple_, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_kp_s_home_glaslos_workspace_pylibinj, __pyx_n_s_detect_sqli, 25, __pyx_empty_bytes); if (unlikely(!__pyx_codeobj__2)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 25; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
            |                                 ^~~~~~~~~~~~~~~~
      /usr/include/python3.11/cpython/code.h:148:28: note: declared here
        148 | PyAPI_FUNC(PyCodeObject *) PyCode_New(
            |                            ^~~~~~~~~~
      src/pylibinjection.c: In function ‘__Pyx_CreateCodeObjectForTraceback’:
      src/pylibinjection.c:1457:9: warning: passing argument 14 of ‘PyCode_New’ makes pointer from integer without a cast [-Wint-conversion]
       1457 |         py_line,      /*int firstlineno,*/
            |         ^~~~~~~
            |         |
            |         int
      src/pylibinjection.c:124:72: note: in definition of macro ‘__Pyx_PyCode_New’
        124 |           PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
            |                                                                        ^~~~~
      /usr/include/python3.11/cpython/code.h:151:33: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘int’
        151 |         PyObject *, PyObject *, PyObject *, int, PyObject *,
            |                                 ^~~~~~~~~~
      src/pylibinjection.c:1458:9: warning: passing argument 15 of ‘PyCode_New’ makes integer from pointer without a cast [-Wint-conversion]
       1458 |         __pyx_empty_bytes  /*PyObject *lnotab*/
            |         ^~~~~~~~~~~~~~~~~
            |         |
            |         PyObject * {aka struct _object *}
      src/pylibinjection.c:124:79: note: in definition of macro ‘__Pyx_PyCode_New’
        124 |           PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
            |                                                                               ^~~~
      /usr/include/python3.11/cpython/code.h:151:45: note: expected ‘int’ but argument is of type ‘PyObject *’ {aka ‘struct _object *’}
        151 |         PyObject *, PyObject *, PyObject *, int, PyObject *,
            |                                             ^~~
      src/pylibinjection.c:124:11: error: too few arguments to function ‘PyCode_New’
        124 |           PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
            |           ^~~~~~~~~~
      src/pylibinjection.c:1443:15: note: in expansion of macro ‘__Pyx_PyCode_New’
       1443 |     py_code = __Pyx_PyCode_New(
            |               ^~~~~~~~~~~~~~~~
      /usr/include/python3.11/cpython/code.h:148:28: note: declared here
        148 | PyAPI_FUNC(PyCodeObject *) PyCode_New(
            |                            ^~~~~~~~~~
      src/pylibinjection.c: In function ‘__Pyx_AddTraceback’:
      src/pylibinjection.c:1489:13: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’}
       1489 |     py_frame->f_lineno = py_line;
            |             ^~
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pylibinjection
  Running setup.py clean for pylibinjection
  Building wheel for langid (setup.py) ... done
  Created wheel for langid: filename=langid-1.1.6-py3-none-any.whl size=1942290 sha256=126beb15d3e98d963d36205c53392af7f7d519e0eeb359f91a5b4da861efe04a
  Stored in directory: /tmp/pip-ephem-wheel-cache-eohj17di/wheels/d1/98/ca/efb42df1369f894e668d2b2f01fd9899fb780ef6863b9b80d5
Successfully built pytaxonomies pymispgalaxies langid
Failed to build pylibinjection
ERROR: Could not build wheels for pylibinjection, which is required to install pyproject.toml-based projects
SteveClement commented 7 months ago

All issues resolved. We are ready for 24.04 :)

SteveClement commented 4 months ago

pylibinjection is again in the requirements and thus AIL fails on Ubuntu 24.04

adulau commented 1 week ago

Yep a replacement is clearly needed for the SQL injection detection module or we fork it and continue to maintain it.

The issue it's two software packages, the Python library and the C library.

Discussing with @Terrtia we might put this module in disabled mode until an improved version is done.

mark11111112 commented 1 week ago

their are numerous issues i managed to get it working after hours of messing around and trying many different versions of ubuntu {20.04.6 will allow it to install but even after i managed to get it installed the dashboard does not launch (i checked netstat and no port 443/7000 is open as mentioned here: https://github.com/ail-project/ail-framework/issues/204) i checked the error logs and no errors.