search

fkie-cad / FACT_docker

Dockerfile for building the FACT container
GNU General Public License v3.0
21 stars 9 forks source link

Support upgrades without database loss #2

Closed frakman1 closed 2 years ago

frakman1 commented 2 years ago

Be aware that all state of FACT is only in the container so if you pull a new FACT version everything (most importantly your database) will be lost.

Why can't the database be on a mounted volume (-v option in docker run) so that the container can be upgraded without losing the databse? This is how most docker web-apps handle databases. For example, I can upgrade my Gitlab docker container without losing all my repos/user info.

jstucke commented 2 years ago

It should be possible to mount your own database (with -v) and mount your own main.cfg and mongo.conf (where the database path and passwords are configured). The text only refers to the default configuration where everything is contained inside the container (and nothing needs to be manually configured). Maybe it is a bit ambiguous in this regard.

jstucke commented 2 years ago

It is probably necessary to touch [YOUR_FACT_WT_MONGODB_PATH]/REINITIALIZE_DB (as it says in the readme.md) so that the new database is correctly initialized when you start the container, though.

frakman1 commented 2 years ago

What is YOUR_FACT_WT_MONGODB_PATH? (What does the WT stand for?) is it in /media/ or /media/data/ or inside one of the three folders (fact_auth_data fact_fw_data fact_wt_mongodb)? Although the touch command is mentioned in the README.md, it doesn't say what that path is or what it means. It seems you want the user to create a new file but it's not clear to me where.

Sorry if this is obvious but I have no experience with mongodb.

UPDATE: OK, I am guessing it is: media/data/fact_wt_mongodb/REINITIALIZE_DB

jstucke commented 2 years ago

OK, I am guessing it is: media/data/fact_wt_mongodb/REINITIALIZE_DB

Yes, that should be the correct path. I will update the readme to make it clearer.

frakman1 commented 2 years ago

I am currently facing permission issues between the container and host. I keep getting permission errors for the media sub-folders despite using --group-add $(getent group docker | cut -d: -f3) I got past this using the brute force chmod -R 777 for the /tmp/... folder but I'm not getting any luck with the media folder.

PermissionError: [Errno 13] Permission denied: '/media/data'

I'm currently running as:

docker run -it --name fact --group-add $(getent group docker | cut -d: -f3) -v /home/frak/git/tch/fact/media:/media -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/fact-docker-tmp:/tmp/fact-docker-tmp -p 0.0.0.0:5000:5000 frakman1/fact:latest start

with themedia folder in the current working directory. I even tried starting with a clean slate with fresh, empty folder and also did a chmod -R 777 on media but still got permission errors.

Full error message:

[2021-10-21 14:31:23][start_all_installed_fact_components][INFO]: starting db
Traceback (most recent call last):
  File "/opt/FACT_core/src/../start_fact_db", line 44, in <module>
    FactDb().main()
  File "/opt/FACT_core/src/../start_fact_db", line 35, in __init__
    self.mongo_server = MongoMgr(config=config)
  File "/opt/FACT_core/src/storage/MongoMgr.py", line 28, in __init__
    os.makedirs(self.mongo_db_file_path, exist_ok=True)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/media/data'
[2021-10-21 14:31:25][start_all_installed_fact_components][INFO]: starting frontend
[2021-10-21 14:31:25][start_all_installed_fact_components][INFO]: starting backend
^C[2021-10-21 14:31:35][start_all_installed_fact_components][INFO]: shutting down...
[2021-10-21 14:31:36][fact_base][INFO]: Received signal 10. Shutting down FACT Backend...
Traceback (most recent call last):
  File "/opt/FACT_core/src/../start_fact_frontend", line 94, in <module>
    FactFrontend().main()
  File "/opt/FACT_core/src/../start_fact_frontend", line 73, in __init__
    super().__init__()
  File "/opt/FACT_core/src/fact_base.py", line 31, in __init__
    self.work_load_stat = WorkLoadStatistic(config=self.config, component=self.COMPONENT)
  File "/opt/FACT_core/src/statistic/work_load.py", line 21, in __init__
    self.db = StatisticDbUpdater(config=self.config)
  File "/opt/FACT_core/src/storage/db_interface_statistic.py", line 15, in __init__
    super().__init__(config=config)
  File "/opt/FACT_core/src/storage/mongo_interface.py", line 25, in __init__
    self._authenticate()
  File "/opt/FACT_core/src/storage/mongo_interface.py", line 40, in _authenticate
    self.client.admin.authenticate(user, pw, mechanism='SCRAM-SHA-1')
  File "/usr/local/lib/python3.8/dist-packages/pymongo/database.py", line 1578, in authenticate
    self.client._cache_credentials(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 809, in _cache_credentials
    server = self._get_topology().select_server(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/topology.py", line 244, in select_server
    return random.choice(self.select_servers(selector,
  File "/usr/local/lib/python3.8/dist-packages/pymongo/topology.py", line 202, in select_servers
    server_descriptions = self._select_servers_loop(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/topology.py", line 218, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: localhost:27018: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 617179bd21b4b87c3ac60a38, topology_type: Single, servers: [<ServerDescription ('localhost', 27018) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:27018: [Errno 111] Connection refused')>]>
Traceback (most recent call last):
  File "/opt/FACT_core/src/../start_fact_backend", line 91, in <module>
    FactBackend().main()
  File "/opt/FACT_core/src/../start_fact_backend", line 39, in __init__
    super().__init__()
  File "/opt/FACT_core/src/fact_base.py", line 31, in __init__
    self.work_load_stat = WorkLoadStatistic(config=self.config, component=self.COMPONENT)
  File "/opt/FACT_core/src/statistic/work_load.py", line 21, in __init__
    self.db = StatisticDbUpdater(config=self.config)
  File "/opt/FACT_core/src/storage/db_interface_statistic.py", line 15, in __init__
    super().__init__(config=config)
  File "/opt/FACT_core/src/storage/mongo_interface.py", line 25, in __init__
    self._authenticate()
  File "/opt/FACT_core/src/storage/mongo_interface.py", line 40, in _authenticate
    self.client.admin.authenticate(user, pw, mechanism='SCRAM-SHA-1')
  File "/usr/local/lib/python3.8/dist-packages/pymongo/database.py", line 1578, in authenticate
    self.client._cache_credentials(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 809, in _cache_credentials
    server = self._get_topology().select_server(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/topology.py", line 244, in select_server
    return random.choice(self.select_servers(selector,
  File "/usr/local/lib/python3.8/dist-packages/pymongo/topology.py", line 202, in select_servers
    server_descriptions = self._select_servers_loop(
  File "/usr/local/lib/python3.8/dist-packages/pymongo/topology.py", line 218, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: localhost:27018: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 617179be6bb05944cc064016, topology_type: Single, servers: [<ServerDescription ('localhost', 27018) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:27018: [Errno 111] Connection refused')>]>
frakman1 commented 2 years ago

When I added -e PUID=1000 -e PGID=1000 -e UMASK=0000 to docker run to use my username/group as obtained from /etc/passwd, I got a different error:

$ docker run -it --name fact --group-add $(getent group docker | cut -d: -f3) -e PUID=1000 -e PGID=1000 -e UMASK=0000  -v /home/frak/git/tch/fact/media:/media -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/fact-docker-tmp:/tmp/fact-docker-tmp -p 0.0.0.0:5000:5000 frakman1/fact:latest start
[2021-10-21 14:42:20][start_all_installed_fact_components][INFO]: starting db
[2021-10-21 14:42:20][MongoMgr][INFO]: Starting local mongo database
[2021-10-21 14:42:20][MongoMgr][INFO]: Starting DB: mongod --auth --config /opt/FACT_core/src/config/mongod.conf --fork --logpath /var/log/fact/mongo.log
[2021-10-21 14:42:21][process][WARNING]: Error: Authentication not successful: Authentication failed., full error: {'ok': 0.0, 'errmsg': 'Authentication failed.', 'code': 18, 'codeName': 'AuthenticationFailed'}
[2021-10-21 14:42:21][process][WARNING]: Error: Authentication not successful: Authentication failed., full error: {'ok': 0.0, 'errmsg': 'Authentication failed.', 'code': 18, 'codeName': 'AuthenticationFailed'}
[2021-10-21 14:42:21][process][CRITICAL]: SHUTTING DOWN SYSTEM
[2021-10-21 14:42:21][process][CRITICAL]: SHUTTING DOWN SYSTEM
[2021-10-21 14:42:22][start_all_installed_fact_components][INFO]: starting frontend
[2021-10-21 14:42:22][start_all_installed_fact_components][INFO]: starting backend
[2021-10-21 14:42:22][process][WARNING]: Error: Authentication not successful: Authentication failed., full error: {'ok': 0.0, 'errmsg': 'Authentication failed.', 'code': 18, 'codeName': 'AuthenticationFailed'}
[2021-10-21 14:42:22][process][CRITICAL]: SHUTTING DOWN SYSTEM
[2021-10-21 14:42:22][process][WARNING]: Error: Authentication not successful: Authentication failed., full error: {'ok': 0.0, 'errmsg': 'Authentication failed.', 'code': 18, 'codeName': 'AuthenticationFailed'}
[2021-10-21 14:42:22][process][CRITICAL]: SHUTTING DOWN SYSTEM
[2021-10-21 14:42:24][start_all_installed_fact_components][CRITICAL]: Backend did not start. Shutting down...

I saw this in /var/log/fact/mongo.log:

2021-10-21T14:42:21.568+0000 I INDEX    [LogicalSessionCacheRefresh]     building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2021-10-21T14:42:21.571+0000 I INDEX    [LogicalSessionCacheRefresh] build index done.  scanned 0 total records. 0 secs
2021-10-21T14:42:22.668+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:36336 #3 (1 connection now open)
2021-10-21T14:42:22.669+0000 I NETWORK  [conn3] received client metadata from 127.0.0.1:36336 conn3: { driver: { name: "PyMongo", version: "3.12.1" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-84-generic" }, platform: "CPython 3.8.10.final.0" }
2021-10-21T14:42:22.671+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:36338 #4 (2 connections now open)
2021-10-21T14:42:22.671+0000 I NETWORK  [conn4] received client metadata from 127.0.0.1:36338 conn4: { driver: { name: "PyMongo", version: "3.12.1" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-84-generic" }, platform: "CPython 3.8.10.final.0" }
2021-10-21T14:42:22.672+0000 I ACCESS   [conn4] SCRAM-SHA-1 authentication failed for fact_admin on admin from client 127.0.0.1:36338 ; UserNotFound: Could not find user fact_admin@admin
2021-10-21T14:42:22.678+0000 I NETWORK  [conn4] end connection 127.0.0.1:36338 (1 connection now open)
2021-10-21T14:42:22.678+0000 I NETWORK  [conn3] end connection 127.0.0.1:36336 (0 connections now open)
2021-10-21T14:42:22.913+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:36340 #5 (1 connection now open)
2021-10-21T14:42:22.914+0000 I NETWORK  [conn5] received client metadata from 127.0.0.1:36340 conn5: { driver: { name: "PyMongo", version: "3.12.1" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-84-generic" }, platform: "CPython 3.8.10.final.0" }
2021-10-21T14:42:22.915+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:36342 #6 (2 connections now open)
2021-10-21T14:42:22.916+0000 I NETWORK  [conn6] received client metadata from 127.0.0.1:36342 conn6: { driver: { name: "PyMongo", version: "3.12.1" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.4.0-84-generic" }, platform: "CPython 3.8.10.final.0" }
2021-10-21T14:42:22.917+0000 I ACCESS   [conn6] SCRAM-SHA-1 authentication failed for fact_admin on admin from client 127.0.0.1:36342 ; UserNotFound: Could not find user fact_admin@admin
2021-10-21T14:42:22.923+0000 I NETWORK  [conn6] end connection 127.0.0.1:36342 (1 connection now open)
2021-10-21T14:42:22.923+0000 I NETWORK  [conn5] end connection 127.0.0.1:36340 (0 connections now open)
frakman1 commented 2 years ago

I also tried mapping to the host's /media folder with no luck. It would be great if there was an official, working docker image.

I tried a publicly available MIPS based-squashfs openwrt image file but also saw that there was no cwe_checker output and no binwalk tab output (entropy image missing) so I doubt it's something to do with my ARM based binaries.

I opened a separate ticket for the binwalk issue here

frakman1 commented 2 years ago

OK, the following comment in another issue resolved the database and general permissions issue : https://github.com/fkie-cad/FACT_docker/issues/3#issuecomment-948706930

Thank you for that!

frakman1 commented 2 years ago

Have you considered creating a Discussions forum for this repository? It would be a good way for people to ask questions about getting FACT to work rather than create issue tickets which is not really the intent of the Issues feature.

Discussions are a bult-in feature of Github and is available under a repository's Settings page:

image