search

tklab-tud / ID2T

Official ID2T repository. ID2T creates labeled IT network datasets that contain user defined synthetic attacks.
MIT License
58 stars 23 forks source link

Error at Runnigng ./id2t #112

Closed mshafqat00 closed 2 years ago

mshafqat00 commented 2 years ago 
shafqat@shafqat-VirtualBox:~/ID2T-toolkit$ ./id2t 
Traceback (most recent call last):
  File "/home/shafqat/ID2T-toolkit/code/CLI.py", line 6, in <module>
    from Core.Controller import Controller
  File "/home/shafqat/ID2T-toolkit/code/Core/__init__.py", line 2, in <module>
    import matplotlib as mplt
ModuleNotFoundError: No module named 'matplotlib'

Even though I have already install matplotlib i am getting this error again and again. I am stuck at this place

pepper-jk commented 2 years ago

Did you follow the installation instructions to build the python environment? If so did you encounter any errors?

Which linux distribution are you using?

mshafqat00 commented 2 years ago

I have followed and installed all of the requirements and no! none any error occurred during installation any of them. Currently I am using Ubuntu 24.04 on VM.

pepper-jk commented 2 years ago

Could you provide the output of the following commands:

# Activate the venv
source .venv/bin/activate

# Upgrade pip if necessary
pip3 install --upgrade pip

# Install python packages
pip3 install --no-cache-dir -r resources/requirements.txt

# Deactivate the venv
deactivate

This should install all the requirements in the virtual environment, including matplotlib.

pepper-jk commented 2 years ago

It appears id2t runs fine in github actions on ubuntu-latest (ubuntu-20.04).

mshafqat00 commented 2 years ago 
shafqat@shafqat:~/ID2T-toolkit$ source .venv/bin/activate
(.venv) shafqat@shafqat:~/ID2T-toolkit$ pip3 install --upgrade pip
Collecting pip
  Downloading pip-22.0.3-py3-none-any.whl (2.1 MB)
     |████████████████████████████████| 2.1 MB 1.5 MB/s 
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.0.2
    Uninstalling pip-20.0.2:
      Successfully uninstalled pip-20.0.2
Successfully installed pip-22.0.3
(.venv) shafqat@shafqat:~/ID2T-toolkit$ pip3 install --no-cache-dir -r resources/requirements.txt 
Collecting cairocffi==0.8.1
  Downloading cairocffi-0.8.1.tar.gz (79 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.4/79.4 KB 347.8 kB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting cffi==1.11.5
  Downloading cffi-1.11.5.tar.gz (438 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 438.5/438.5 KB 343.4 kB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting coverage==4.5.1
  Downloading coverage-4.5.1.tar.gz (379 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 379.7/379.7 KB 228.8 kB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting cycler==0.10.0
  Downloading cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
Collecting kiwisolver==1.0.1
  Downloading kiwisolver-1.0.1.tar.gz (31 kB)
  Preparing metadata (setup.py) ... done
Collecting lea==2.3.5
  Downloading lea-2.3.5.tar.gz (64 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.5/64.5 KB 131.8 kB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting matplotlib==2.2.3
  Downloading matplotlib-2.2.3.tar.gz (36.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 36.8/36.8 MB 133.9 kB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [63 lines of output]
      ============================================================================
      Edit setup.cfg to change the build options

      BUILDING MATPLOTLIB
                  matplotlib: yes [2.2.3]
                      python: yes [3.8.10 (default, Nov 26 2021, 20:14:08)  [GCC
                              9.3.0]]
                    platform: yes [linux]

      REQUIRED DEPENDENCIES AND EXTENSIONS
                       numpy: yes [not found. pip may install it below.]
            install_requires: yes [handled by setuptools]
                      libagg: yes [pkg-config information for 'libagg' could not
                              be found. Using local copy.]
                    freetype: no  [The C/C++ header for freetype2 (ft2build.h)
                              could not be found.  You may need to install the
                              development package.]
                         png: no  [pkg-config information for 'libpng' could not
                              be found.]
                       qhull: yes [pkg-config information for 'libqhull' could not
                              be found. Using local copy.]

      OPTIONAL SUBPACKAGES
                 sample_data: yes [installing]
                    toolkits: yes [installing]
                       tests: no  [skipping due to configuration]
              toolkits_tests: no  [skipping due to configuration]

      OPTIONAL BACKEND EXTENSIONS
                      macosx: no  [Mac OS-X only]
                      qt5agg: no  [PySide2 not found; PyQt5 not found]
                      qt4agg: no  [PySide not found; PyQt4 not found]
                     gtk3agg: no  [Requires pygobject to be installed.]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-_s5fmj3j/matplotlib_9a1c19ed8a5b4a729141402e3ceacf05/setup.py", line 198, in <module>
          msg = pkg.install_help_msg()
        File "/tmp/pip-install-_s5fmj3j/matplotlib_9a1c19ed8a5b4a729141402e3ceacf05/setupext.py", line 592, in install_help_msg
          release = platform.linux_distribution()[0].lower()
      AttributeError: module 'platform' has no attribute 'linux_distribution'
                   gtk3cairo: no  [Requires cairocffi or pycairo to be installed.]
                      gtkagg: no  [Requires pygtk]
                       tkagg: yes [installing; run-time loading from Python Tcl /
                              Tk]
                       wxagg: no  [requires wxPython]
                         gtk: no  [Requires pygtk]
                         agg: yes [installing]
                       cairo: no  [cairocffi or pycairo not found]
                   windowing: no  [Microsoft Windows only]

      OPTIONAL LATEX DEPENDENCIES
                      dvipng: no
                 ghostscript: yes [version 9.50]
                       latex: no
                     pdftops: yes [version 0.86.1]

      OPTIONAL PACKAGE DATA
                        dlls: no  [skipping due to configuration]

      ============================================================================
                              * The following required packages can not be built:
                              * freetype, png
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

here is the output of the above code Actually I am not that professional with the codes so sometimes I just stuck on even small problems. this tool is actually very helpful for my research so please help me to resolve the issue. Btw thankyou for your time and support

pepper-jk commented 2 years ago

It appears matplotlib is missing some C/C++ dependencies on your system. Try installing them with this and repeat the process.

It is missing libagg and libqhull, which can be installed like this:

sudo apt install libagg-dev libqhull-dev

It is also missing libpng and libfreetype, but the config sets them to no, so I assume it is not using them. However, you can install them to be sure:

sudo apt install libpng-dev libfreetype-dev

Also make sure you have the build-essential installed:

sudo apt install build-essential

EDIT: sorry for the heavy edit on this comment.

mshafqat00 commented 2 years ago 
shafqat@shafqat:~/ID2T-toolkit$ ls
build.sh  code_boost  id2t        logo       resources  test_efficiency
code      env1        LICENSE.md  README.md  run_tests
shafqat@shafqat:~/ID2T-toolkit$ source env1/bin/activate
(env1) shafqat@shafqat:~/ID2T-toolkit$ pip3 install --upgrade pip
Requirement already satisfied: pip in ./env1/lib/python3.8/site-packages (22.0.3)
(env1) shafqat@shafqat:~/ID2T-toolkit$ pip3 install --no-cache-dir -r resources/requirements.txt 
Requirement already satisfied: cairocffi==1.3.0 in ./env1/lib/python3.8/site-packages (from -r resources/requirements.txt (line 1)) (1.3.0)
Requirement already satisfied: cffi==1.15.0 in ./env1/lib/python3.8/site-packages (from -r resources/requirements.txt (line 2)) (1.15.0)
Requirement already satisfied: coverage==6.3.1 in ./env1/lib/python3.8/site-packages (from -r resources/requirements.txt (line 3)) (6.3.1)
Requirement already satisfied: cycler==0.11.0 in ./env1/lib/python3.8/site-packages (from -r resources/requirements.txt (line 4)) (0.11.0)
Requirement already satisfied: kiwisolver==1.3.2 in ./env1/lib/python3.8/site-packages (from -r resources/requirements.txt (line 5)) (1.3.2)
Requirement already satisfied: lea==3.4.1 in ./env1/lib/python3.8/site-packages (from -r resources/requirements.txt (line 6)) (3.4.1)
Requirement already satisfied: matplotlib==3.5.1 in ./env1/lib/python3.8/site-packages (from -r resources/requirements.txt (line 7)) (3.5.1)
Requirement already satisfied: memory-profiler==0.60.0 in ./env1/lib/python3.8/site-packages (from -r resources/requirements.txt (line 8)) (0.60.0)
Requirement already satisfied: numpy==1.22.2 in ./env1/lib/python3.8/site-packages (from -r resources/requirements.txt (line 9)) (1.22.2)
Requirement already satisfied: psutil==5.9.0 in ./env1/lib/python3.8/site-packages (from -r resources/requirements.txt (line 10)) (5.9.0)
Requirement already satisfied: pycparser==2.21 in ./env1/lib/python3.8/site-packages (from -r resources/requirements.txt (line 11)) (2.21)
Requirement already satisfied: pyparsing==3.0.7 in ./env1/lib/python3.8/site-packages (from -r resources/requirements.txt (line 12)) (3.0.7)
Requirement already satisfied: python-dateutil==2.8.2 in ./env1/lib/python3.8/site-packages (from -r resources/requirements.txt (line 13)) (2.8.2)
Requirement already satisfied: pytz==2021.3 in ./env1/lib/python3.8/site-packages (from -r resources/requirements.txt (line 14)) (2021.3)
Requirement already satisfied: pyxdg==0.27 in ./env1/lib/python3.8/site-packages (from -r resources/requirements.txt (line 15)) (0.27)
Requirement already satisfied: scapy-python3==0.26 in ./env1/lib/python3.8/site-packages (from -r resources/requirements.txt (line 16)) (0.26)
Requirement already satisfied: scipy==1.8.0 in ./env1/lib/python3.8/site-packages (from -r resources/requirements.txt (line 17)) (1.8.0)
Requirement already satisfied: six==1.16.0 in ./env1/lib/python3.8/site-packages (from -r resources/requirements.txt (line 18)) (1.16.0)
Requirement already satisfied: packaging>=20.0 in ./env1/lib/python3.8/site-packages (from matplotlib==3.5.1->-r resources/requirements.txt (line 7)) (21.3)
Requirement already satisfied: fonttools>=4.22.0 in ./env1/lib/python3.8/site-packages (from matplotlib==3.5.1->-r resources/requirements.txt (line 7)) (4.29.1)
Requirement already satisfied: pillow>=6.2.0 in ./env1/lib/python3.8/site-packages (from matplotlib==3.5.1->-r resources/requirements.txt (line 7)) (9.0.1)

I've installed all the requirements but with updated versions of each package and all installed perfectly but this time different errors has been generated such as below mentioned: I am sure updating packages will not create any problem in running the tool? 

(env1) shafqat@shafqat:~/ID2T-toolkit$ ./id2t 
/home/shafqat/ID2T-toolkit/code/Core/Controller.py:182: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if not self.non_verbose and len(attacks_config) is not 1:
/home/shafqat/ID2T-toolkit/code/ID2TLib/Utility.py:208: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if platform is "linux":
/home/shafqat/ID2T-toolkit/code/ID2TLib/Utility.py:211: SyntaxWarning: "is" with a literal. Did you mean "=="?
  elif platform is "macos":
Matplotlib is building the font cache; this may take a moment.
Traceback (most recent call last):
  File "/home/shafqat/ID2T-toolkit/code/CLI.py", line 5, in <module>
    from Core.Controller import Controller
  File "/home/shafqat/ID2T-toolkit/code/Core/Controller.py", line 9, in <module>
    import Core.AttackController as atkCtrl
  File "/home/shafqat/ID2T-toolkit/code/Core/AttackController.py", line 8, in <module>
    import Core.LabelManager as LabelManager
  File "/home/shafqat/ID2T-toolkit/code/Core/LabelManager.py", line 8, in <module>
    import ID2TLib.TestLibrary as Lib
  File "/home/shafqat/ID2T-toolkit/code/ID2TLib/TestLibrary.py", line 5, in <module>
    import ID2TLib.Utility as Util
  File "/home/shafqat/ID2T-toolkit/code/ID2TLib/Utility.py", line 8, in <module>
    import scapy.layers.inet as inet
ModuleNotFoundError: No module named 'scapy.layers'
pepper-jk commented 2 years ago

I am sure updating packages will not create any problem in running the tool?

They might and obviously do in your case. scapy complains that it is missing the module scapy.layers. Python packages tend to break compatibility with older code. Looks like your scapy-python3 version is strange 0.26, which is either super old or a new versioning scheme. We specify scapy==2.4.2 in the requirements file.

You should install the dependencies from the unmodified requirements.txt. It specifies the packages that need to keep their old version. This way compatibility between the libraries and id2t can be ensured.

(If necessary delete the virtual environment and create it again.)

You can try to install dependencies manually and/or different versions, but this is not recommended or supported by us.

mshafqat00 commented 2 years ago 
(venv3) shafqat@shafqat:~/ID2T-toolkit$ pip3 install --no-cache-dir -r resources/re1.txt 
Collecting scipy==1.1.0
  Downloading scipy-1.1.0.tar.gz (15.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.6/15.6 MB 286.4 kB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting cffi==1.11.5
  Downloading cffi-1.11.5.tar.gz (438 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 438.5/438.5 KB 906.2 kB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Requirement already satisfied: pycparser in ./venv3/lib/python3.8/site-packages (from cffi==1.11.5->-r resources/re1.txt (line 2)) (2.18)
Building wheels for collected packages: scipy, cffi
  Building wheel for scipy (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [178 lines of output]
      /tmp/pip-install-7co21yp_/scipy_2532ce3716c9428aaba28f5a33d8a81f/setup.py:114: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
        import imp
      lapack_opt_info:
      lapack_mkl_info:
      customize UnixCCompiler
        libraries mkl_rt not found in ['/home/shafqat/ID2T-toolkit/venv3/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
        NOT AVAILABLE

      openblas_lapack_info:
      customize UnixCCompiler
      customize UnixCCompiler
        libraries openblas not found in ['/home/shafqat/ID2T-toolkit/venv3/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
        NOT AVAILABLE

      openblas_clapack_info:
      customize UnixCCompiler
      customize UnixCCompiler
        libraries openblas,lapack not found in ['/home/shafqat/ID2T-toolkit/venv3/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
        NOT AVAILABLE

      atlas_3_10_threads_info:
      Setting PTATLAS=ATLAS
      customize UnixCCompiler
        libraries tatlas,tatlas not found in /home/shafqat/ID2T-toolkit/venv3/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /home/shafqat/ID2T-toolkit/venv3/lib
      customize UnixCCompiler
        libraries tatlas,tatlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries tatlas,tatlas not found in /usr/lib64
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib64
      customize UnixCCompiler
        libraries tatlas,tatlas not found in /usr/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib
      customize UnixCCompiler
        libraries tatlas,tatlas not found in /usr/lib/x86_64-linux-gnu
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
      <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
        NOT AVAILABLE

      atlas_3_10_info:
      customize UnixCCompiler
        libraries satlas,satlas not found in /home/shafqat/ID2T-toolkit/venv3/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /home/shafqat/ID2T-toolkit/venv3/lib
      customize UnixCCompiler
        libraries satlas,satlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries satlas,satlas not found in /usr/lib64
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib64
      customize UnixCCompiler
        libraries satlas,satlas not found in /usr/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib
      customize UnixCCompiler
        libraries satlas,satlas not found in /usr/lib/x86_64-linux-gnu
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
      <class 'numpy.distutils.system_info.atlas_3_10_info'>
        NOT AVAILABLE

      atlas_threads_info:
      Setting PTATLAS=ATLAS
      customize UnixCCompiler
        libraries ptf77blas,ptcblas,atlas not found in /home/shafqat/ID2T-toolkit/venv3/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /home/shafqat/ID2T-toolkit/venv3/lib
      customize UnixCCompiler
        libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries ptf77blas,ptcblas,atlas not found in /usr/lib64
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib64
      customize UnixCCompiler
        libraries ptf77blas,ptcblas,atlas not found in /usr/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib
      customize UnixCCompiler
        libraries ptf77blas,ptcblas,atlas not found in /usr/lib/x86_64-linux-gnu
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
      <class 'numpy.distutils.system_info.atlas_threads_info'>
        NOT AVAILABLE

      atlas_info:
      customize UnixCCompiler
        libraries f77blas,cblas,atlas not found in /home/shafqat/ID2T-toolkit/venv3/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /home/shafqat/ID2T-toolkit/venv3/lib
      customize UnixCCompiler
        libraries f77blas,cblas,atlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries f77blas,cblas,atlas not found in /usr/lib64
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib64
      customize UnixCCompiler
        libraries f77blas,cblas,atlas not found in /usr/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib
      customize UnixCCompiler
        libraries f77blas,cblas,atlas not found in /usr/lib/x86_64-linux-gnu
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
      <class 'numpy.distutils.system_info.atlas_info'>
        NOT AVAILABLE

      accelerate_info:
        NOT AVAILABLE

      /home/shafqat/ID2T-toolkit/venv3/lib/python3.8/site-packages/numpy/distutils/system_info.py:625: UserWarning:
          Atlas (http://math-atlas.sourceforge.net/) libraries not found.
          Directories to search for the libraries can be specified in the
          numpy/distutils/site.cfg file (section [atlas]) or by setting
          the ATLAS environment variable.
        self.calc_info()
      lapack_info:
      customize UnixCCompiler
        libraries lapack not found in ['/home/shafqat/ID2T-toolkit/venv3/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
        NOT AVAILABLE

      /home/shafqat/ID2T-toolkit/venv3/lib/python3.8/site-packages/numpy/distutils/system_info.py:625: UserWarning:
          Lapack (http://www.netlib.org/lapack/) libraries not found.
          Directories to search for the libraries can be specified in the
          numpy/distutils/site.cfg file (section [lapack]) or by setting
          the LAPACK environment variable.
        self.calc_info()
      lapack_src_info:
        NOT AVAILABLE

      /home/shafqat/ID2T-toolkit/venv3/lib/python3.8/site-packages/numpy/distutils/system_info.py:625: UserWarning:
          Lapack (http://www.netlib.org/lapack/) sources not found.
          Directories to search for the sources can be specified in the
          numpy/distutils/site.cfg file (section [lapack_src]) or by setting
          the LAPACK_SRC environment variable.
        self.calc_info()
        NOT AVAILABLE

      Running from scipy source directory.
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-7co21yp_/scipy_2532ce3716c9428aaba28f5a33d8a81f/setup.py", line 474, in <module>
          setup_package()
        File "/tmp/pip-install-7co21yp_/scipy_2532ce3716c9428aaba28f5a33d8a81f/setup.py", line 470, in setup_package
          setup(**metadata)
        File "/home/shafqat/ID2T-toolkit/venv3/lib/python3.8/site-packages/numpy/distutils/core.py", line 135, in setup
          config = configuration()
        File "/tmp/pip-install-7co21yp_/scipy_2532ce3716c9428aaba28f5a33d8a81f/setup.py", line 388, in configuration
          config.add_subpackage('scipy')
        File "/home/shafqat/ID2T-toolkit/venv3/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1032, in add_subpackage
          config_list = self.get_subpackage(subpackage_name, subpackage_path,
        File "/home/shafqat/ID2T-toolkit/venv3/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 998, in get_subpackage
          config = self._get_configuration_from_setup_py(
        File "/home/shafqat/ID2T-toolkit/venv3/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 940, in _get_configuration_from_setup_py
          config = setup_module.configuration(*args)
        File "scipy/setup.py", line 15, in configuration
          config.add_subpackage('linalg')
        File "/home/shafqat/ID2T-toolkit/venv3/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1032, in add_subpackage
          config_list = self.get_subpackage(subpackage_name, subpackage_path,
        File "/home/shafqat/ID2T-toolkit/venv3/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 998, in get_subpackage
          config = self._get_configuration_from_setup_py(
        File "/home/shafqat/ID2T-toolkit/venv3/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 940, in _get_configuration_from_setup_py
          config = setup_module.configuration(*args)
        File "scipy/linalg/setup.py", line 19, in configuration
          raise NotFoundError('no lapack/blas resources found')
      numpy.distutils.system_info.NotFoundError: no lapack/blas resources found
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for scipy
  Running setup.py clean for scipy
  error: subprocess-exited-with-error

  × python setup.py clean did not run successfully.
  │ exit code: 1
  ╰─> [11 lines of output]
      /tmp/pip-install-7co21yp_/scipy_2532ce3716c9428aaba28f5a33d8a81f/setup.py:114: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
        import imp

      `setup.py clean` is not supported, use one of the following instead:

        - `git clean -xdf` (cleans all files)
        - `git clean -Xdf` (cleans all versioned files, doesn't touch
                            files that aren't checked into the git repo)

      Add `--force` to your command to use it anyway if you must (unsupported).

      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed cleaning build dir for scipy
  Building wheel for cffi (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [24 lines of output]
      running bdist_wheel
      running build
      running build_py
      running build_ext
      building '_cffi_backend' extension
      c/_cffi_backend.c: In function ‘b_do_dlopen’:
      c/_cffi_backend.c:4197:31: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
       4197 |         *p_printable_filename = PyText_AsUTF8(s);
            |                               ^
      c/_cffi_backend.c: In function ‘b_callback’:
      c/_cffi_backend.c:5911:5: warning: ‘ffi_prep_closure’ is deprecated: use ffi_prep_closure_loc instead [-Wdeprecated-declarations]
       5911 |     if (ffi_prep_closure(closure, &cif_descr->cif,
            |     ^~
      In file included from c/_cffi_backend.c:15:
      /usr/include/x86_64-linux-gnu/ffi.h:334:1: note: declared here
        334 | ffi_prep_closure (ffi_closure*,
            | ^~~~~~~~~~~~~~~~
      In file included from c/cffi1_module.c:20,
                       from c/_cffi_backend.c:7370:
      c/call_python.c: In function ‘_get_interpstate_dict’:
      c/call_python.c:20:30: error: dereferencing pointer to incomplete type ‘PyInterpreterState’ {aka ‘struct _is’}
         20 |     builtins = tstate->interp->builtins;
            |                              ^~
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for cffi
  Running setup.py clean for cffi
Failed to build scipy cffi
Installing collected packages: scipy, cffi
  Running setup.py install for scipy ... error
  error: subprocess-exited-with-error

  × Running setup.py install for scipy did not run successfully.
  │ exit code: 1
  ╰─> [187 lines of output]
      /tmp/pip-install-7co21yp_/scipy_2532ce3716c9428aaba28f5a33d8a81f/setup.py:114: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
        import imp

      Note: if you need reliable uninstall behavior, then install
      with pip instead of using `setup.py install`:

        - `pip install .`       (from a git repo or downloaded source
                                 release)
        - `pip install scipy`   (last SciPy release on PyPI)

      lapack_opt_info:
      lapack_mkl_info:
      customize UnixCCompiler
        libraries mkl_rt not found in ['/home/shafqat/ID2T-toolkit/venv3/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
        NOT AVAILABLE

      openblas_lapack_info:
      customize UnixCCompiler
      customize UnixCCompiler
        libraries openblas not found in ['/home/shafqat/ID2T-toolkit/venv3/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
        NOT AVAILABLE

      openblas_clapack_info:
      customize UnixCCompiler
      customize UnixCCompiler
        libraries openblas,lapack not found in ['/home/shafqat/ID2T-toolkit/venv3/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
        NOT AVAILABLE

      atlas_3_10_threads_info:
      Setting PTATLAS=ATLAS
      customize UnixCCompiler
        libraries tatlas,tatlas not found in /home/shafqat/ID2T-toolkit/venv3/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /home/shafqat/ID2T-toolkit/venv3/lib
      customize UnixCCompiler
        libraries tatlas,tatlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries tatlas,tatlas not found in /usr/lib64
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib64
      customize UnixCCompiler
        libraries tatlas,tatlas not found in /usr/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib
      customize UnixCCompiler
        libraries tatlas,tatlas not found in /usr/lib/x86_64-linux-gnu
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
      <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
        NOT AVAILABLE

      atlas_3_10_info:
      customize UnixCCompiler
        libraries satlas,satlas not found in /home/shafqat/ID2T-toolkit/venv3/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /home/shafqat/ID2T-toolkit/venv3/lib
      customize UnixCCompiler
        libraries satlas,satlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries satlas,satlas not found in /usr/lib64
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib64
      customize UnixCCompiler
        libraries satlas,satlas not found in /usr/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib
      customize UnixCCompiler
        libraries satlas,satlas not found in /usr/lib/x86_64-linux-gnu
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
      <class 'numpy.distutils.system_info.atlas_3_10_info'>
        NOT AVAILABLE

      atlas_threads_info:
      Setting PTATLAS=ATLAS
      customize UnixCCompiler
        libraries ptf77blas,ptcblas,atlas not found in /home/shafqat/ID2T-toolkit/venv3/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /home/shafqat/ID2T-toolkit/venv3/lib
      customize UnixCCompiler
        libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries ptf77blas,ptcblas,atlas not found in /usr/lib64
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib64
      customize UnixCCompiler
        libraries ptf77blas,ptcblas,atlas not found in /usr/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib
      customize UnixCCompiler
        libraries ptf77blas,ptcblas,atlas not found in /usr/lib/x86_64-linux-gnu
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
      <class 'numpy.distutils.system_info.atlas_threads_info'>
        NOT AVAILABLE

      atlas_info:
      customize UnixCCompiler
        libraries f77blas,cblas,atlas not found in /home/shafqat/ID2T-toolkit/venv3/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /home/shafqat/ID2T-toolkit/venv3/lib
      customize UnixCCompiler
        libraries f77blas,cblas,atlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/local/lib
      customize UnixCCompiler
        libraries f77blas,cblas,atlas not found in /usr/lib64
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib64
      customize UnixCCompiler
        libraries f77blas,cblas,atlas not found in /usr/lib
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib
      customize UnixCCompiler
        libraries f77blas,cblas,atlas not found in /usr/lib/x86_64-linux-gnu
      customize UnixCCompiler
        libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu
      <class 'numpy.distutils.system_info.atlas_info'>
        NOT AVAILABLE

      accelerate_info:
        NOT AVAILABLE

      /home/shafqat/ID2T-toolkit/venv3/lib/python3.8/site-packages/numpy/distutils/system_info.py:625: UserWarning:
          Atlas (http://math-atlas.sourceforge.net/) libraries not found.
          Directories to search for the libraries can be specified in the
          numpy/distutils/site.cfg file (section [atlas]) or by setting
          the ATLAS environment variable.
        self.calc_info()
      lapack_info:
      customize UnixCCompiler
        libraries lapack not found in ['/home/shafqat/ID2T-toolkit/venv3/lib', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
        NOT AVAILABLE

      /home/shafqat/ID2T-toolkit/venv3/lib/python3.8/site-packages/numpy/distutils/system_info.py:625: UserWarning:
          Lapack (http://www.netlib.org/lapack/) libraries not found.
          Directories to search for the libraries can be specified in the
          numpy/distutils/site.cfg file (section [lapack]) or by setting
          the LAPACK environment variable.
        self.calc_info()
      lapack_src_info:
        NOT AVAILABLE

      /home/shafqat/ID2T-toolkit/venv3/lib/python3.8/site-packages/numpy/distutils/system_info.py:625: UserWarning:
          Lapack (http://www.netlib.org/lapack/) sources not found.
          Directories to search for the sources can be specified in the
          numpy/distutils/site.cfg file (section [lapack_src]) or by setting
          the LAPACK_SRC environment variable.
        self.calc_info()
        NOT AVAILABLE

      Running from scipy source directory.
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-7co21yp_/scipy_2532ce3716c9428aaba28f5a33d8a81f/setup.py", line 474, in <module>
          setup_package()
        File "/tmp/pip-install-7co21yp_/scipy_2532ce3716c9428aaba28f5a33d8a81f/setup.py", line 470, in setup_package
          setup(**metadata)
        File "/home/shafqat/ID2T-toolkit/venv3/lib/python3.8/site-packages/numpy/distutils/core.py", line 135, in setup
          config = configuration()
        File "/tmp/pip-install-7co21yp_/scipy_2532ce3716c9428aaba28f5a33d8a81f/setup.py", line 388, in configuration
          config.add_subpackage('scipy')
        File "/home/shafqat/ID2T-toolkit/venv3/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1032, in add_subpackage
          config_list = self.get_subpackage(subpackage_name, subpackage_path,
        File "/home/shafqat/ID2T-toolkit/venv3/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 998, in get_subpackage
          config = self._get_configuration_from_setup_py(
        File "/home/shafqat/ID2T-toolkit/venv3/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 940, in _get_configuration_from_setup_py
          config = setup_module.configuration(*args)
        File "scipy/setup.py", line 15, in configuration
          config.add_subpackage('linalg')
        File "/home/shafqat/ID2T-toolkit/venv3/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1032, in add_subpackage
          config_list = self.get_subpackage(subpackage_name, subpackage_path,
        File "/home/shafqat/ID2T-toolkit/venv3/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 998, in get_subpackage
          config = self._get_configuration_from_setup_py(
        File "/home/shafqat/ID2T-toolkit/venv3/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 940, in _get_configuration_from_setup_py
          config = setup_module.configuration(*args)
        File "scipy/linalg/setup.py", line 19, in configuration
          raise NotFoundError('no lapack/blas resources found')
      numpy.distutils.system_info.NotFoundError: no lapack/blas resources found
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> scipy

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
(venv3) shafqat@shafqat:~/ID2T-toolkit$

This is my 3rd virtual box and I am still struggling with this so the problem I am facing in install 1-scipy==1.1.0 2-cffi==1.11.5 these two packages I have installed all the other packages excluding these two and separately created requirements.txt and included these and both are giving me errors. I have searched a lot but didn't work at all So could look at the error and help me btw sorry for a very long comment!

Note: I have installed all the other packages according to the versions that are mentioned!

pepper-jk commented 2 years ago

I'm closing this issue, since this is not a bug with ID2T but clearly user error.

Stay tuned for the final advice.

pepper-jk commented 2 years ago

As stated before, if you install packages manually, specifically other versions than we specify, we can not support you. We do not have the time or resources to debug every user's personal setup.

This is why we have a build script included, that installs all required dependencies.

I can not follow what you are doing over there, especially since you take huge steps before telling me your "next problem".

Did the solution I posted for matplotlib work? If not, why not?

Why did you start installing random package versions manually, while you state you have no experience with code?

Why would you set up new virtual boxes? To be clear virtual-environment is the .venv folder which is created during id2t build, not the whole system! If you are unfamiliar with an expression, just ask. Don't just assume and go off on your own.

All you should have done was install the missing libs for matplotlib as described above and then reinstalled the pip dependencies, which should have worked by then. Either by doing it manually as I posted here or by running ./build.sh, or even ./build.sh --full (which creates a new venv).

I will give you a last guide on how to get id2t to run, step by step, but this is the last thing I will do, afterwards you are on your own.

pepper-jk commented 2 years ago

Install ID2T on Ubuntu 20.04 VBox

  1. Update system 
    sudo apt update
sudo apt upgrade
  2. Install build-essentials and matplotlib libraries 
    sudo apt install build-essential
sudo apt install libagg-dev libqhull-dev
sudo apt install libpng-dev libfreetype-dev
  3. Clone ID2T. 
    git clone https://github.com/tklab-tud/ID2T
  4. Run build script. 
    ./build.sh --full
  5. Profit. 
    # -h is to see all command line arguments
./id2t -h

I sincerely hope this helps you. Please let me know if you have any issues with this guide.

Do not google smaller problems you encounter, do things manually, and then post a new unrelated problem!

mshafqat00 commented 2 years ago

Thank you @pepper-jk Man thanks a lot! It finally work without any error. you were right I was actually manually installing all the dependencies and requirements without understanding the basics. so for this time I installed from the scratch followed your instructions without any extra commands and it worked

Thank you again for your time!

here is the final output:

All is set. ID2T is ready.

Run efficiency tests with the command './test_efficiency'
Run unit tests with the command './run_tests'
Run ID2T with the command './id2t'
(.env1) shafqat@shafqat:~/id2t_1$ ./id2t 
usage: id2t [-h] (-i PCAP_FILE | -l) [-c CONFIG_FILE] [-e] [-r] [-s]
            [-p [PLOT]] [-q [QUERY]] [-t] [-S RNGSEED [RNGSEED ...]] [-T] [-V]
            [-o PCAP_FILE] [-ie] [-d]
            [-si STATISTICS_INTERVAL [STATISTICS_INTERVAL ...]] [-rd] [-ry]
            [-rn] [-li] [--skip] [-a ATTACK [ATTACK ...]]
pepper-jk commented 2 years ago

Glad it worked out for you. Good luck with your research.