Make install script compatible with newer releases

[JRomainG]

Add compatibility with Ubuntu 20.04 (focal) and Debian 11 (bullseye).

I took the liberty to slightly refactor the install scripts' code to make it easier to handle different releases. I tested these changes both on Ubuntu 20.04 and Debian 11.

[dorpvom]

Thanks for the Pull Request. We'll have a look on it shortly.

Meanwhile, how did you test this? I found that at least Bullseye does not yet work, since the pre_install script exits prematurely. This can be easily fixed though (python-pip does not exist as package anymore, so this has to be installed separately and errors have to be handled)

[JRomainG]

@dorpvom I installed Debian bullseye on my laptop and ran the pre_install.sh and install.py scripts, and then played around with the ./fact_extractor/fact_extract.py to make sure that it worked correctly. However, I may have installed python3-pip manually while testing, which made the whole install work (since pip2 isn't used in the Debian install.py script).

I think this can be easily fixed by separating the install steps in pre_install.sh so, event if python-pip isn't installed, the script doesn't abort.

@jstucke Thanks for the review! Regarding generic_fs/install.sh, I didn't have to change anything when installing, since the package was still available. Did you run into an issue?

I had to move the pip2 dependencies away from common because pip2 is not longer available on Debian bullseye. I replaced the missing dependency with a package installed through apt and everything seems to work fine.

The change to the install_github_project is necessary to fix an issue on the latest release of Debian, but I indeed forgot to comment that. Was there a specific reason to stick to that commit?

I'll work on fixing those issues in the meantime and update my branch.

[jstucke]

I had to move the pip2 dependencies away from common because pip2 is not longer available on Debian bullseye. I replaced the missing dependency with a package installed through apt and everything seems to work fine.

Sadly, this does also not seem to work on focal because it seems the package (which gets installed in the pre_install.sh script) was removed:

E: Unable to locate package python-pip

Installing directly from https://bootstrap.pypa.io/get-pip.py (as in https://github.com/fkie-cad/FACT_core/blob/master/src/install/common.py) could solve this, though.

[JRomainG]

@jstucke Indeed, the python-pip package is missing for focal too, I just added a new commit and I'll try reinstalling on another VM from scratch.

[dorpvom]

Our CI fails on the current working tree with

15:51:02 [2020-06-05 15:51:02][install][INFO]: Installing pluginbase git+https://github.com/armbues/python-entropy git+https://github.com/fkie-cad/common_helper_unpacking_classifier.git git+https://github.com/fkie-cad/fact_helper_file.git pyqtgraph capstone cstruct python-lzo numpy scipy

15:51:30 Traceback (most recent call last):
15:51:30   File "fact_extractor/install.py", line 109, in <module>
15:51:30     sys.exit(main())
15:51:30   File "fact_extractor/install.py", line 103, in main
15:51:30     unpacker(distribution)
15:51:30   File "/home/jenkins/workspace/fact_extractor_remote/fact_extractor/install/unpacker.py", line 198, in main
15:51:30     install_dependencies(DEPENDENCIES[distribution])
15:51:30 KeyError: 'bionic'

[JRomainG]

I tested the install on Ubuntu 20.04 again, and with the latest commit everything works from scratch. I also tested the same thing on Ubuntu 18.04, the install seems to be working fine.

I also noticed some failed tests, like you mentioned:

test_plugin_jffs2.py::Test_JFFS2_Unpacker::test_extraciton_big    FAILED
test_plugin_jffs2.py::Test_JFFS2_Unpacker::test_extraction_little FAILED
test_plugin_ubi_image.py::Test_UBIFS_Unpacker::test_extraction    FAILED
test_plugin_ubifs.py::Test_UBIFS_Unpacker::test_extraction        FAILED

It seems like jefferson uses Python 2, but there also is a Python 3 version. Switching to this new package doesn't solve the issue, though.

On Debian, the additional following test fails:

test_plugin_generic_fs.py::TestGenericFsUnpacker::test_extraction_cramfs FAILED

For reference, on Ubuntu 18.04, all those tests succeed.

[JRomainG]

Latest commits fix the failing tests for Ubuntu 20.04 and Debian 11. Unfortunately, Debian cannot mount cramfs images:

unknown filesystem type 'cramfs'.

Apart from that, everything seems good to me.

[jstucke]

('sviehb/jefferson', ['sudo python3 setup.py install'])

It seems jefferson is python2 code so the installation may work but running the code won't.

$ jefferson 
  File "/usr/local/bin/jefferson", line 156
    print 'node_crc does not match!'
          ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print('node_crc does not match!')?

That's also why the jffs2 tests are failing.

Is it possible to install python2 on debian 11? On focal the python2 installation is no problem. Only the package python-pip is missing. But that's not really a problem when installing pip following the official documentation. Then installing pyliblzma is no problem and jefferson works on focal without issues.

I would also really like to get rid of those python2 dependencies but not really at the cost of removing unpacking plugins. Maybe we could at least find a clean way to disable them on incompatible systems and log a warning.

[JRomainG]

In one of the last few commits, I switch from https://github.com/sviehb/jefferson to https://github.com/svidovich/jefferson-3 for bullseye and focal, which uses Python 3. With that dependency, the jffs2 tests all succeed, both on Debian and Ubuntu.

If you would rather stick to the original dependency, there is a python 2 package for bullseye that could be installed.

[jstucke]

If it works, switching to the Python 3 version would be great. Is there even anything left that needs Python 2? The pyliblzma dependency was only in there because of jefferson and jefferson-3 does not need that package.

The only remaining ones from the original install should be ubi, patool and uefi and all of them seem to be Python 3 compatible now. It looks like we could git rid of all Python 2 dependencies (which would be really nice).

[JRomainG]

I don't think there are any Python 2 dependencies left for focal and bullseye, or at least that's what I tried to do. I'll have to double check that the apt package for patool is using Python 3 though.

[jstucke]

I don't think there are any Python 2 dependencies left for focal and bullseye

Then there is probably no reason to keep Python 2 dependencies for any other distribution (it is EOL anyway).

[JRomainG]

I just checked, the patool package for Debian does indeed use Python 3. Moreover, I am able to run fact_extractor.py without Python 2 installed, as expected.

I didn't change the dependencies for other distributions as I didn't want to break anything, especially considering I had to change quite a few apt and pip packages. However, it should be possible to removed all those Python 2 dependencies if the packages are available for older distributions as well. Should we try that out?

[jstucke]

I think we should. The only problem could be that one of the ported Python 3 packages now needs a Python version >3.5 (which would only be a problem for Xenial). In my opinion we should aim at keeping the individual dependencies of the distributions as small as possible.

[dorpvom]

I like where this is going :smile: . Maybe we can just install python > 3.5 on xenial et al. and we are golden.

I had in mind the DJI stuff from here running on py2 but they have changed that at some point as well.

Dropping py2 would be enviable.

[JRomainG]

The patool packages for Debian buster and Ubuntu bionic both rely on python2, so the current dependencies for focal and bullseye can't just be copied over. However, it seems that patool is available through pip for Python 3 and Python2, so that should solve the issue.

python-lzo is also available to install through pip3, and ubi_reader and jefferson can be installed for Python 3 using the same dependencies as for focal / bullseye.

I think those are the only dependencies left that rely on Python2, so I'll try to replace those, remove python2, reinstall from scratch on bionic, and run pytest to see if everything works as expected.

[JRomainG]

It seems like archmage, when installed through apt, relies on Python 2:

$ aptitude why python
i   archmage Depends python

Installing libchm-dev though apt and then archmage though pip3 should do the trick.

However, it seems like fact_extractor/bin/uefi-firmware-parser is still using Python 2, so that code should be migrated. Moreover, the patool script installed through pip3 uses #!/usr/bin/env python, which is not linked to Python 3 on bionic, so it fails. One solution I can see working is using virtualenv to make sure Python 3 is called.

[JRomainG]

I got patool to work without Python 2, but I have found 2 remaining issues:

• uefi-firmware-parser's --superbrute options is broken under Python 3 (see this issue),
• Freetz cannot be installed without Python 2 (see the list of dependencies)

[jstucke]

I tried to run it again with focal, this time without python 2 and the freetz installation ran through after I created a symlink from python3 to python. We only use shell scripts and ELF binaries from the freetz tools anyway so this shouldn't cause problems.

[JRomainG]

I created a PR for uefi-firmware-parser to try to fix the issue with Python 3, a temporary fix could be to checkout the current commit from master and apply a patch.

Regarding freetz, I guess that means the install script in unpacker.py needs to be slightly changed so python points to python3 for makeuser while it exists, so the install always goes through.

[jstucke]

adding the line 'sudo ln -s $(which python3) ./python', in _install_freetz and changing 'sudo su makeuser -c "umask 0022 && make -j$(nproc) tools"', to 'sudo su makeuser -c "export PATH=$(pwd):$PATH && umask 0022 && make -j$(nproc) tools"', worked for me.

On this occasion I noticed that tools/unlzma gets copied from the freetz tool set but it is only a symlink to busybox. unlzma is installed anyway so I guess this can be safely removed.

[jstucke]

The uefi unpacker tests ran successfully on focal with the patch.

[dorpvom]

There is a PEP8 Error in the srec.py Plug In

_______________________ FLAKE8-check(ignoring E501 W503) _______________________
/home/jenkins/workspace/fact_extractor_remote/fact_extractor/plugins/unpacking/srec/code/srec.py:22:13: E741 ambiguous variable name 'l'

@jstucke we may want to solve this in a different branch

[dorpvom]

Meanwhile Xenial and Bionic builds on our CI run successfully aside of this one PEP issue

[dorpvom]

I'll test the extractor in the coming days and if I don't find runtime issues I'll press merge

[JRomainG]

Thanks for the feedback! Let me know if you run into any issue