Step 10/13 fails: can't pip install failure

[rjzak]

=> [base 10/13] RUN wget -qO /tmp/dotnet-install.sh https://dot.net/v1/dotnet-install.sh  && chmod +x /tmp/dotnet-install.sh  && /tmp/dotnet-install.sh -c Current  && rm -f /tmp/dotnet-install.sh  && chmod +x /home/user/.  14.6s 
4.741 Collecting cffi>=1.14.0 (from angr)
tainers-2.4.0 sympy-1.12 threadpoolctl-3.1.0 tomlkit-0.11.8 trove-classifiers-2023.7.6 tzdata-2023.3 unicorn-2.0.1.post1 urllib3-1.26.16 virtualenv-20.23.1 webencodings-0.5.1 z3-solver-4.10.2.0 zipp-3.16.0
47.30 Collecting dl8.5
47.38   Downloading dl8.5-0.0.15.tar.gz (38 kB)
47.41   Preparing metadata (setup.py): started
48.50   Preparing metadata (setup.py): finished with status 'error'
48.51   error: subprocess-exited-with-error
48.51   
48.51   × python setup.py egg_info did not run successfully.
48.51   │ exit code: 1
48.51   ╰─> [16 lines of output]
48.51       /usr/lib/python3/dist-packages/pythran/tables.py:4530: FutureWarning: In the future `np.bool` will be defined as the corresponding NumPy scalar.
48.51         if not hasattr(numpy, method):
48.51       /usr/lib/python3/dist-packages/pythran/tables.py:4563: FutureWarning: In the future `np.bytes` will be defined as the corresponding NumPy scalar.
48.51         obj = getattr(themodule, elem)
48.51       Traceback (most recent call last):
48.51         File "<string>", line 2, in <module>
48.51         File "<pip-setuptools-caller>", line 34, in <module>
48.51         File "/tmp/pip-install-kw0w6jiz/dl8-5_909a5b641c0b4f3bab2b9d7b9964c9ed/setup.py", line 6, in <module>
48.51           from dl85 import __version__
48.51         File "/tmp/pip-install-kw0w6jiz/dl8-5_909a5b641c0b4f3bab2b9d7b9964c9ed/dl85/__init__.py", line 3, in <module>
48.51           from .unsupervised.clustering import DL85Cluster
48.51         File "/tmp/pip-install-kw0w6jiz/dl8-5_909a5b641c0b4f3bab2b9d7b9964c9ed/dl85/unsupervised/__init__.py", line 1, in <module>
48.51           from .clustering import DL85Cluster
48.51         File "/tmp/pip-install-kw0w6jiz/dl8-5_909a5b641c0b4f3bab2b9d7b9964c9ed/dl85/unsupervised/clustering.py", line 3, in <module>
48.51           from sklearn.neighbors import DistanceMetric
48.51       ImportError: cannot import name 'DistanceMetric' from 'sklearn.neighbors' (/home/user/.local/lib/python3.11/site-packages/sklearn/neighbors/__init__.py)
48.51       [end of output]
48.51   
48.51   note: This error originates from a subprocess, and is likely not a problem with pip.
48.51 error: metadata-generation-failed
48.51 
48.51 × Encountered error while generating package metadata.
48.51 ╰─> See above for output.
48.51 
48.51 note: This is an issue with the package mentioned above, not pip.
48.51 hint: See above for details.
------
Dockerfile:109
--------------------
 108 |     RUN python3 -m pip install --user --upgrade --break-system-packages pip
 109 | >>> RUN pip3 install --user --no-warn-script-location --ignore-installed --break-system-packages \
 110 | >>>         angr capa capstone meson pandas poetry scikit-learn \
 111 | >>>  && pip3 install --user --no-warn-script-location --ignore-installed --break-system-packages \
 112 | >>>         dl8.5 pefile pyelftools thefuck tinyscript tldr weka
 113 |     # initialize Go
--------------------
ERROR: failed to solve: process "/bin/sh -c pip3 install --user --no-warn-script-location --ignore-installed --break-system-packages         angr capa capstone meson pandas poetry scikit-learn  && pip3 install --user --no-warn-script-location --ignore-installed --break-system-packages         dl8.5 pefile pyelftools thefuck tinyscript tldr weka" did not complete successfully: exit code: 1

This is with git hash b229a11f.

[rjzak]

A requirements.txt with specific versions might be a way to prevent this, probably caused by changing versions between sklearn, dl8.5, and maybe something else. I tried having python3-sklearn and python3-pandas installed via apt-get, but that didn't help.

[rjzak]

Some progress. I added gfortran to one of the apt-get install commands, and set a version of 0.24.2 for sklearn installation with pip. That seems to have worked, as it got passed the install for dl8.5.

But then came another issue at line 165 (about 15 lines from the end!):

 > [framework 10/17] RUN find /tmp/analyzers -type f -executable -exec mv {} /home/user/.opt/bin/ ;  && /home/user/.opt/tools/packing-box setup analyzer:                                                                                         
0.622 Traceback (most recent call last):                                                                                                                                                                                                          
0.622   File "/home/user/.opt/tools/packing-box", line 4, in <module>                                                                                                                                                                             
0.622     from pbox import *
0.622   File "/home/user/.local/lib/python3.11/site-packages/pbox/__init__.py", line 6, in <module>
0.622     from .common import *
0.622   File "/home/user/.local/lib/python3.11/site-packages/pbox/common/__init__.py", line 6, in <module>
0.622     from .alterations import *
0.622   File "/home/user/.local/lib/python3.11/site-packages/pbox/common/alterations/__init__.py", line 8, in <module>
0.622     from .pe import *
0.622   File "/home/user/.local/lib/python3.11/site-packages/pbox/common/alterations/pe.py", line 3, in <module>
0.622     from .parsers import parser_handler, parse_exe_info_default, SectionAbstract
0.622 ImportError: cannot import name 'parse_exe_info_default' from 'pbox.common.alterations.parsers' (/home/user/.local/lib/python3.11/site-packages/pbox/common/alterations/parsers.py)
------
Dockerfile:165
--------------------
 164 |     COPY --chown=$USER $FILES/analyzers/* /tmp/analyzers/
 165 | >>> RUN find /tmp/analyzers -type f -executable -exec mv {} $UOPT/bin/ \; \
 166 | >>>  && $PBOX setup analyzer
 167 |     # install detectors (including wrapper scripts)
--------------------
ERROR: failed to solve: process "/bin/sh -c find /tmp/analyzers -type f -executable -exec mv {} $UOPT/bin/ \\;  && $PBOX setup analyzer" did not complete successfully: exit code: 1

[dhondta]

@rjzak Thank you for mentioning this.

• The error of dl8.5 comes from the fact that its latest version on PyPi is 0.1.5 and tries to import DistanceMetric from sklearn.neighbors while the latest version of sklearn holds this class in sklearn.metrics, which is fixed in version 0.1.8 of dl8.5.
• The error of parse_exe_info_default comes from a bug I hadn't fixed yet by testing the latest enhancements I've executed these lat days.

Both errors are solved with 9b95886a86abc15fefaba3ae4577255349a1b692.