search

fkie-cad / FACT_core

Firmware Analysis and Comparison Tool
https://fkie-cad.github.io/FACT_core
GNU General Public License v3.0
1.21k stars 224 forks source link

Installation failure #455

Closed doasmike closed 4 years ago

doasmike commented 4 years ago

I've a problem with installing FACT. During installation I receive an "no space left on device" error but there is plenty of space on HD. Log:

mike@mike-VirtualBox:~/Pulpit$ ~/FACT_core/src/install.py
[2020-07-27 22:49:26][install][INFO]: FACT Installer 1.1
[2020-07-27 22:49:26][install][INFO]: Installing apt-transport-https
[sudo] hasło użytkownika mike: 
[2020-07-27 22:49:30][common][INFO]: Updating system
[2020-07-27 22:49:32][install][INFO]: Installing python3 python3-dev build-essential automake autoconf libtool git unzip
[2020-07-27 22:49:33][install][INFO]: Installing testresources
[2020-07-27 22:49:35][install][INFO]: Removing python3-pip python3-setuptools python3-wheel
[2020-07-27 22:49:36][common][INFO]: Installing python3 pip
[2020-07-27 22:49:41][install][INFO]: Installing python python-dev
[2020-07-27 22:49:42][install][INFO]: Removing python-pip
[2020-07-27 22:49:43][common][INFO]: Installing python2 pip
[2020-07-27 22:49:47][install][INFO]: Installing libmagic-dev
[2020-07-27 22:49:47][install][INFO]: Installing libfuzzy-dev
[2020-07-27 22:49:48][install][INFO]: Installing python3-tlsh
[2020-07-27 22:49:49][install][INFO]: Installing ssdeep
[2020-07-27 22:49:50][install][INFO]: Installing git+https://github.com/fkie-cad/fact_helper_file.git
[2020-07-27 22:49:53][install][INFO]: Installing psutil
[2020-07-27 22:49:55][install][INFO]: Installing pytest==3.5.1 pytest-cov pytest-flake8 pylint python-magic xmltodict yara-python==3.7.0 appdirs
[2020-07-27 22:50:08][install][INFO]: Installing lief
[2020-07-27 22:50:09][install][INFO]: Installing requests
[2020-07-27 22:50:11][install][INFO]: Installing pymongo pyyaml
[2020-07-27 22:50:14][common][WARNING]: variety spec not overwritten
[2020-07-27 22:50:14][install][INFO]: Installing hurry.filesize
[2020-07-27 22:50:15][install][INFO]: Installing git+https://github.com/fkie-cad/common_helper_files.git
[2020-07-27 22:50:18][install][INFO]: Installing git+https://github.com/fkie-cad/common_helper_mongo.git
[2020-07-27 22:50:21][install][INFO]: Installing git+https://github.com/mass-project/common_helper_encoder.git
[2020-07-27 22:50:24][install][INFO]: Installing git+https://github.com/fkie-cad/common_helper_filter.git
[2020-07-27 22:50:28][install][INFO]: Installing git+https://github.com/fkie-cad/common_helper_process.git
[2020-07-27 22:50:31][install][INFO]: Installing werkzeug==0.16.1
[2020-07-27 22:50:32][install][INFO]: Installing flask flask_restful flask_security flask_sqlalchemy flask-paginate Flask-API uwsgi bcrypt python-dateutil si-prefix email-validator
[2020-07-27 22:50:50][frontend][INFO]: Install static jstree content
[2020-07-27 22:50:51][frontend][INFO]: Install static angularJS content
[2020-07-27 22:50:51][frontend][INFO]: Install static charts.js content
[2020-07-27 22:50:51][frontend][INFO]: Installing highlight js
[2020-07-27 22:50:55][frontend][INFO]: Install static https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css content
[2020-07-27 22:50:55][frontend][INFO]: Install static https://cdnjs.cloudflare.com/ajax/libs/bootstrap-datepicker/1.8.0/css/bootstrap-datepicker.standalone.css content
[2020-07-27 22:50:55][frontend][INFO]: Install static https://cdnjs.cloudflare.com/ajax/libs/jquery/1.12.1/jquery.min.js content
[2020-07-27 22:50:55][frontend][INFO]: Install static https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js content
[2020-07-27 22:50:55][frontend][INFO]: Install static https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js content
[2020-07-27 22:50:55][frontend][INFO]: Install static https://cdnjs.cloudflare.com/ajax/libs/bootstrap-datepicker/1.8.0/js/bootstrap-datepicker.js content
[2020-07-27 22:50:55][frontend][INFO]: Install static https://raw.githubusercontent.com/moment/moment/develop/moment.js content
[2020-07-27 22:50:55][frontend][INFO]: Creating directory for authentication
[2020-07-27 22:50:55][frontend][INFO]: Initializing docker container for radare
[2020-07-27 22:50:58][frontend][INFO]: Pulling pdf report container
[2020-07-27 22:51:00][db][INFO]: Setting up mongo database
[2020-07-27 22:51:00][install][INFO]: Installing mongodb
[2020-07-27 22:51:01][db][INFO]: Initialize database
[2020-07-27 22:51:04][install][INFO]: Installing python-dev python-setuptools
[2020-07-27 22:51:04][install][INFO]: Installing libjpeg-dev
[2020-07-27 22:51:05][install][INFO]: Installing libssl-dev python3-tk
[2020-07-27 22:51:06][install][INFO]: Installing pluginbase Pillow cryptography pyopenssl matplotlib docker networkx
[2020-07-27 22:51:17][backend][INFO]: Installing yara
[2020-07-27 22:51:17][install][INFO]: Installing bison flex
[2020-07-27 22:51:17][backend][INFO]: skipping yara installation (already installed)
[2020-07-27 22:51:17][backend][INFO]: Building fact extraction container
[2020-07-27 22:51:19][install][INFO]: Installing git+https://github.com/fkie-cad/common_helper_yara.git
[2020-07-27 22:51:22][install][INFO]: Installing git+https://github.com/mass-project/common_analysis_base.git
[2020-07-27 22:51:25][backend][INFO]: Installing plugins
[2020-07-27 22:51:25][backend][INFO]: Running ../plugins/analysis/linter/install.sh
[2020-07-27 22:51:44][backend][INFO]: Running ../plugins/analysis/cwe_checker/install.sh
Traceback (most recent call last):
  File "/home/mike/FACT_core/src/install.py", line 181, in <module>
    install()
  File "/home/mike/FACT_core/src/install.py", line 169, in install
    backend(distribution)
  File "/home/mike/FACT_core/src/install/backend.py", line 45, in main
    _install_plugins(distribution)
  File "/home/mike/FACT_core/src/install/backend.py", line 109, in _install_plugins
    raise InstallationError('Error in installation of {} plugin\n{}'.format(Path(install_script).parent.name, shell_output))
helperFunctions.install.InstallationError: Error in installation of cwe_checker plugin
------------------------------------
 Installing cwe_checker Plugin 
------------------------------------
Trying to pull cwe_checker Docker image
latest: Pulling from fkiecad/cwe_checker
23884877105a: Pulling fs layer
bc38caa0f5b9: Pulling fs layer
2910811b6c42: Pulling fs layer
36505266dcc6: Pulling fs layer
07a923053093: Pulling fs layer
776b7eaf0a03: Pulling fs layer
bc700a6f34b5: Pulling fs layer
b8737cc1b28a: Pulling fs layer
b080d246d138: Pulling fs layer
aebbe06e9c29: Pulling fs layer
ec92c48ba1e3: Pulling fs layer
ede53ba17cee: Pulling fs layer
ff0563bfeb51: Pulling fs layer
36505266dcc6: Waiting
b080d246d138: Waiting
aebbe06e9c29: Waiting
ec92c48ba1e3: Waiting
ede53ba17cee: Waiting
bc700a6f34b5: Waiting
07a923053093: Waiting
b8737cc1b28a: Waiting
776b7eaf0a03: Waiting
ff0563bfeb51: Waiting
2910811b6c42: Verifying Checksum
2910811b6c42: Download complete
bc38caa0f5b9: Verifying Checksum
bc38caa0f5b9: Download complete
36505266dcc6: Verifying Checksum
36505266dcc6: Download complete
07a923053093: Download complete
23884877105a: Verifying Checksum
23884877105a: Download complete
776b7eaf0a03: Verifying Checksum
776b7eaf0a03: Download complete
b8737cc1b28a: Verifying Checksum
b8737cc1b28a: Download complete
23884877105a: Pull complete
bc38caa0f5b9: Pull complete
2910811b6c42: Pull complete
36505266dcc6: Pull complete
07a923053093: Pull complete
776b7eaf0a03: Pull complete
bc700a6f34b5: Verifying Checksum
bc700a6f34b5: Download complete
ec92c48ba1e3: Verifying Checksum
ec92c48ba1e3: Download complete
bc700a6f34b5: Pull complete
b8737cc1b28a: Pull complete
ede53ba17cee: Verifying Checksum
ede53ba17cee: Download complete
ff0563bfeb51: Verifying Checksum
b080d246d138: Verifying Checksum
b080d246d138: Download complete
b080d246d138: Pull complete
aebbe06e9c29: Verifying Checksum
aebbe06e9c29: Download complete
ff0563bfeb51: Download complete
failed to register layer: Error processing tar file(exit status 1): write /home/bap/.opam/4.07.1/.opam-switch/sources/bap-primus-taint.master/testsuite/veri/traces/x86-64/DIV.binprot: no space left on device
Docker pull failed. Installing cwe_checker from git
Checking out cwe_checker
Cloning into 'internal'...
Building docker container
Sending build context to Docker daemon  10.04MB
Step 1/6 : FROM fkiecad/cwe_checker_travis_docker_image:latest
latest: Pulling from fkiecad/cwe_checker_travis_docker_image
23884877105a: Pulling fs layer
bc38caa0f5b9: Pulling fs layer
2910811b6c42: Pulling fs layer
36505266dcc6: Pulling fs layer
07a923053093: Pulling fs layer
776b7eaf0a03: Pulling fs layer
bc700a6f34b5: Pulling fs layer
b8737cc1b28a: Pulling fs layer
b080d246d138: Pulling fs layer
aebbe06e9c29: Pulling fs layer
ec92c48ba1e3: Pulling fs layer
36505266dcc6: Waiting
07a923053093: Waiting
776b7eaf0a03: Waiting
bc700a6f34b5: Waiting
b8737cc1b28a: Waiting
b080d246d138: Waiting
aebbe06e9c29: Waiting
ec92c48ba1e3: Waiting
bc38caa0f5b9: Verifying Checksum
bc38caa0f5b9: Download complete
2910811b6c42: Verifying Checksum
2910811b6c42: Download complete
36505266dcc6: Verifying Checksum
36505266dcc6: Download complete
07a923053093: Verifying Checksum
07a923053093: Download complete
23884877105a: Verifying Checksum
23884877105a: Download complete
776b7eaf0a03: Verifying Checksum
776b7eaf0a03: Download complete
b8737cc1b28a: Verifying Checksum
b8737cc1b28a: Download complete
23884877105a: Pull complete
bc38caa0f5b9: Pull complete
2910811b6c42: Pull complete
36505266dcc6: Pull complete
07a923053093: Pull complete
776b7eaf0a03: Pull complete
bc700a6f34b5: Verifying Checksum
bc700a6f34b5: Download complete
ec92c48ba1e3: Verifying Checksum
ec92c48ba1e3: Download complete
bc700a6f34b5: Pull complete
b8737cc1b28a: Pull complete
b080d246d138: Verifying Checksum
b080d246d138: Download complete
b080d246d138: Pull complete
aebbe06e9c29: Verifying Checksum
aebbe06e9c29: Download complete
failed to register layer: Error processing tar file(exit status 1): write /home/bap/.opam/4.07.1/.opam-switch/sources/bap-primus.master/testsuite/bin/x86_64-w64-mingw32-symexec-input: no space left on device

My df -H:

udev            4,2G     0  4,2G   0% /dev
tmpfs           835M  1,5M  834M   1% /run
/dev/sda5        21G   13G  6,5G  67% /
tmpfs           4,2G  110M  4,1G   3% /dev/shm
tmpfs           5,3M  4,1k  5,3M   1% /run/lock
tmpfs           4,2G     0  4,2G   0% /sys/fs/cgroup
/dev/loop1      253M  253M     0 100% /snap/gnome-3-34-1804/24
/dev/loop0       58M   58M     0 100% /snap/core18/1705
/dev/loop2       66M   66M     0 100% /snap/gtk-common-themes/1506
/dev/loop4       29M   29M     0 100% /snap/snapd/7264
/dev/loop3       53M   53M     0 100% /snap/snap-store/433
/dev/sda1       536M  4,1k  536M   1% /boot/efi
tmpfs           835M   29k  835M   1% /run/user/1000

I'm trying to run in on VM:

mike@mike-VirtualBox
OS: Ubuntu 20.04 focal
Kernel: x86_64 Linux 5.4.0-42-generic
Uptime: 20m
Packages: 2093
Shell: bash
Resolution: 1920x975
DE: GNOME 3.36.4
WM: Mutter
WM Theme: Adwaita
GTK Theme: Yaru [GTK2/3]
Icon Theme: Yaru
Font: Ubuntu 11
Disk: 13G / 24G (55%)
CPU: Intel Core i7-4700HQ @ 4x 2,394GHz
GPU: VMware SVGA II Adapter
AM: 1941MiB / 7961MiB
jstucke commented 4 years ago

The problem could also be that you are running out of inodes. What is the output of df -i?

doasmike commented 4 years ago

The problem could also be that you are running out of inodes. What is the output of df -i?

Filesystem     Inodes    IUsed IFree    IUse%  Mounted on
udev           1012066    469 1011597    1% /dev
tmpfs          1019106    840 1018266    1% /run
/dev/sda5      1277952 363841  914111   29% /
tmpfs          1019106     58 1019048    1% /dev/shm
tmpfs          1019106      5 1019101    1% /run/lock
tmpfs          1019106     18 1019088    1% /sys/fs/cgroup
/dev/loop0       10765  10765       0  100% /snap/core18/1705
/dev/loop1       25913  25913       0  100% /snap/gnome-3-34-1804/24
/dev/loop3       24339  24339       0  100% /snap/gnome-3-34-1804/36
/dev/loop2       10756  10756       0  100% /snap/core18/1880
/dev/loop6       15827  15827       0  100% /snap/snap-store/433
/dev/loop4       62342  62342       0  100% /snap/gtk-common-themes/1506
/dev/loop7         463    463       0  100% /snap/snapd/8542
/dev/loop5         459    459       0  100% /snap/snapd/7264
/dev/loop8       15827  15827       0  100% /snap/snap-store/467
/dev/sda1            0      0       0     - /boot/efi
tmpfs          1019106     93 1019013    1% /run/user/1000
/dev/sr0             0      0       0     - /media/mike/VBox_GAs_6.0.20
jstucke commented 4 years ago

I'm not really sure what causes this. Normally, 24 GB of disk space should be enough (assuming there isn't much else on the disk). You could try enlarging the disk of the VM or moving the cwe_checker out of the plugin folder temporarily (since it is by far the largest plugin and takes up ~5 GB alone) and then starting the installation again.

dorpvom commented 4 years ago

I'd say more than 5 GB actually since the docker container is like 7.5 GB alone. The FACT your df shows 6.5 GB of free space might be misleading since the containers are probably stored in total or cleaned up. So it's fairly possible that docker pull found not enough space, deleted the temporary files and you're back to 6.5 GB of free space.

Anyway, deleting the CWE checker directory (i.e. src/plugins/analysis/cwe_checker) is a safe option to test if everything else runs fine.

We should probably fix the hardware requirements in the README @jstucke

doasmike commented 4 years ago

I'm not really sure what causes this. Normally, 24 GB of disk space should be enough (assuming there isn't much else on the disk). You could try enlarging the disk of the VM or moving the cwe_checker out of the plugin folder temporarily (since it is by far the largest plugin and takes up ~5 GB alone) and then starting the installation again.

After moving "cwe_checker":

mike@mike-VirtualBox:~/FACT_core$ ~/FACT_core/src/install.py
[2020-07-30 18:07:48][install][INFO]: FACT Installer 1.1
[2020-07-30 18:07:48][install][INFO]: Installing apt-transport-https
[sudo] hasło użytkownika mike: 
[2020-07-30 18:07:55][common][INFO]: Updating system
[2020-07-30 18:08:01][install][INFO]: Installing python3 python3-dev build-essential automake autoconf libtool git unzip
[2020-07-30 18:08:03][install][INFO]: Installing testresources
[2020-07-30 18:08:07][install][INFO]: Removing python3-pip python3-setuptools python3-wheel
[2020-07-30 18:08:08][common][INFO]: Installing python3 pip
[2020-07-30 18:08:21][install][INFO]: Installing python python-dev
[2020-07-30 18:08:23][install][INFO]: Removing python-pip
[2020-07-30 18:08:24][common][INFO]: Installing python2 pip
[2020-07-30 18:08:34][install][INFO]: Installing libmagic-dev
[2020-07-30 18:08:36][install][INFO]: Installing libfuzzy-dev
[2020-07-30 18:08:37][install][INFO]: Installing python3-tlsh
[2020-07-30 18:08:39][install][INFO]: Installing ssdeep
[2020-07-30 18:08:42][install][INFO]: Installing git+https://github.com/fkie-cad/fact_helper_file.git
[2020-07-30 18:08:48][install][INFO]: Installing psutil
[2020-07-30 18:08:52][install][INFO]: Installing pytest==3.5.1 pytest-cov pytest-flake8 pylint python-magic xmltodict yara-python==3.7.0 appdirs
[2020-07-30 18:09:26][install][INFO]: Installing lief
[2020-07-30 18:09:29][install][INFO]: Installing requests
[2020-07-30 18:09:32][install][INFO]: Installing pymongo pyyaml
[2020-07-30 18:09:40][common][WARNING]: variety spec not overwritten
[2020-07-30 18:09:40][install][INFO]: Installing hurry.filesize
[2020-07-30 18:09:44][install][INFO]: Installing git+https://github.com/fkie-cad/common_helper_files.git
[2020-07-30 18:09:49][install][INFO]: Installing git+https://github.com/fkie-cad/common_helper_mongo.git
[2020-07-30 18:09:55][install][INFO]: Installing git+https://github.com/mass-project/common_helper_encoder.git
[2020-07-30 18:10:01][install][INFO]: Installing git+https://github.com/fkie-cad/common_helper_filter.git
[2020-07-30 18:10:07][install][INFO]: Installing git+https://github.com/fkie-cad/common_helper_process.git
[2020-07-30 18:10:13][install][INFO]: Installing werkzeug==0.16.1
[2020-07-30 18:10:16][install][INFO]: Installing flask flask_restful flask_security flask_sqlalchemy flask-paginate Flask-API uwsgi bcrypt python-dateutil si-prefix email-validator
[2020-07-30 18:10:49][frontend][INFO]: Install static jstree content
[2020-07-30 18:10:50][frontend][INFO]: Install static angularJS content
[2020-07-30 18:10:50][frontend][INFO]: Install static charts.js content
[2020-07-30 18:10:50][frontend][INFO]: Installing highlight js
[2020-07-30 18:10:54][frontend][INFO]: Install static https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css content
[2020-07-30 18:10:54][frontend][INFO]: Install static https://cdnjs.cloudflare.com/ajax/libs/bootstrap-datepicker/1.8.0/css/bootstrap-datepicker.standalone.css content
[2020-07-30 18:10:54][frontend][INFO]: Install static https://cdnjs.cloudflare.com/ajax/libs/jquery/1.12.1/jquery.min.js content
[2020-07-30 18:10:54][frontend][INFO]: Install static https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js content
[2020-07-30 18:10:54][frontend][INFO]: Install static https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js content
[2020-07-30 18:10:54][frontend][INFO]: Install static https://cdnjs.cloudflare.com/ajax/libs/bootstrap-datepicker/1.8.0/js/bootstrap-datepicker.js content
[2020-07-30 18:10:54][frontend][INFO]: Install static https://raw.githubusercontent.com/moment/moment/develop/moment.js content
[2020-07-30 18:10:54][frontend][INFO]: Creating directory for authentication
[2020-07-30 18:10:54][frontend][INFO]: Initializing docker container for radare
[2020-07-30 18:11:00][frontend][INFO]: Pulling pdf report container
[2020-07-30 18:11:03][db][INFO]: Setting up mongo database
[2020-07-30 18:11:03][install][INFO]: Installing mongodb
[2020-07-30 18:11:04][db][INFO]: Initialize database
[2020-07-30 18:11:09][install][INFO]: Installing python-dev python-setuptools
[2020-07-30 18:11:11][install][INFO]: Installing libjpeg-dev
[2020-07-30 18:11:13][install][INFO]: Installing libssl-dev python3-tk
[2020-07-30 18:11:14][install][INFO]: Installing pluginbase Pillow cryptography pyopenssl matplotlib docker networkx
[2020-07-30 18:11:38][backend][INFO]: Installing yara
[2020-07-30 18:11:38][install][INFO]: Installing bison flex
[2020-07-30 18:11:40][backend][INFO]: skipping yara installation (already installed)
[2020-07-30 18:11:40][backend][INFO]: Building fact extraction container
[2020-07-30 18:11:42][install][INFO]: Installing git+https://github.com/fkie-cad/common_helper_yara.git
[2020-07-30 18:11:48][install][INFO]: Installing git+https://github.com/mass-project/common_analysis_base.git
[2020-07-30 18:11:53][backend][INFO]: Installing plugins
[2020-07-30 18:11:53][backend][INFO]: Running ../plugins/analysis/linter/install.sh
[2020-07-30 18:12:37][backend][INFO]: Running ../plugins/analysis/users_and_passwords/install.sh
[2020-07-30 18:12:57][backend][INFO]: Running ../plugins/analysis/oms/install.sh
Traceback (most recent call last):
  File "/home/mike/FACT_core/src/install.py", line 181, in <module>
    install()
  File "/home/mike/FACT_core/src/install.py", line 169, in install
    backend(distribution)
  File "/home/mike/FACT_core/src/install/backend.py", line 45, in main
    _install_plugins(distribution)
  File "/home/mike/FACT_core/src/install/backend.py", line 109, in _install_plugins
    raise InstallationError('Error in installation of {} plugin\n{}'.format(Path(install_script).parent.name, shell_output))
helperFunctions.install.InstallationError: Error in installation of oms plugin
------------------------------------
     install oms dependencies       
------------------------------------
E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6684 (apt-get)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?
jstucke commented 4 years ago

E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6684 (apt-get)

Sounds like you need to disable Unattended Upgrades

doasmike commented 4 years ago

Problem solved