search

eliasgranderubio / dagda

a tool to perform static analysis of known vulnerabilities, trojans, viruses, malware & other malicious threats in docker images/containers and to monitor the docker daemon and running docker containers for detecting anomalous activities
Apache License 2.0
1.16k stars 163 forks source link

unable to start dagda #61

Closed ntart23 closed 5 years ago

ntart23 commented 5 years ago

Short description

I was trying to do start dagda with 'sudo python3 dagda start', but nothing happened in ubuntu. almost like #29:failed to start dagda, but #29 not finished and closed

Reproduction steps

  1. install docker CE
  2. install pip3
  3. install sysdig (manual)
  4. install requirements.txt
  5. install mongodb
  6. install os kernel
  7. sudo python3 dagda.py start then dagda printed, 'serving on localhost:5000' and nothing happened(like #29), over 1 hour. In this situation, i typed 'netstat -an | grep 5000' on the other terminal, and it showed 127.0.0.1 LISTEN. I also tried after 'serving on local host:5000' without control+c , export DAGDA_HOST='127.0.0.1' export DAGDA_PORT=5000 python3 dagda.py vuln --init, then nothing happened and i typed 'sudo python3 dagda.py vuln --init_status' on the other terminal, then <_get_dagda_base_url:174> . . error came out. (also tried when nothing happened after start, typed all commands that export host,port, vuln init, and status on the other terminal, but same error came out)

and these are output of sudo ps -ef . . root 8313 8188 0 09:01 pts/0 00:00:00 sudo python3 dagda.py start root 8314 8313 0 09:01 pts/0 00:00:01 python3 dagda.py start root 8319 8314 0 09:01 pts/0 00:00:00 /usr/bin/python3 -c from mult joon 8320 7696 0 09:01 tty2 00:00:00 /usr/lib/deja-dup/deja-dup-mo root 8324 8314 0 09:01 pts/0 00:00:00 python3 dagda.py start root 8325 8314 0 09:01 pts/0 00:00:00 python3 dagda.py start root 8326 8314 0 09:01 pts/0 00:00:00 python3 dagda.py start root 10255 2 0 09:02 ? 00:00:00 [kworker/0:1-eve] root 10287 680 0 09:02 ? 00:00:00 containerd-shim -namespace mo root 10316 10287 0 09:02 ? 00:00:03 falco -pc -o json_output=true . . i can't find what's wrong in my environment settings , help me

On which platforms did you notice this:

Please complete the following information:

eliasgranderubio commented 5 years ago

Hi @ntart23 ,

Have you review the Dagda documentation about its CLI usage?

On the other hand, be careful with the sudo because you look like set the environment variable with a non-sudo user and then you use the CLI with a sudo user. Please, review this issue for more information because it was some similar to your description.

Regards.

ntart23 commented 5 years ago

Thanks for quickly response . I referenced your answer and reinstalled ubuntu , docker, dagda with root(without sudo) and solved that problem.

but now i have a new problem. after start dagda, i typed [export DAGDA_HOST='127.0.0.1' and export DAGDA_PORT=5000 and python3 dagda.py vuln --init ] in other terminal. but when i typed init_status , it is printed status:initializing over 4hours.

and i referenced #32 , so i typed start dagda -d , but forever initializing and not became updated too.

root@ubuntu:/home/joon/Downloads/dagda-master/dagda# python3 dagda.py vuln --init { "msg": "Accepted the init db request" } root@ubuntu:/home/joon/Downloads/dagda-master/dagda# python3 dagda.py vuln --init_status { "status": "Initializing", "timestamp": "2019-05-28 02:53:58.304500" } root@ubuntu:/home/joon/Downloads/dagda-master/dagda# python3 dagda.py vuln --init_status { "status": "Initializing", "timestamp": "2019-05-28 02:53:58.304500" }

Serving on http://localhost:5000 <2019-05-28 11:53:58,363> <2019-05-28 11:53:58,364> <Updating CVE collection ...> <2019-05-28 11:53:58,365> <_threaded_cve_gathering:148> <... Including CVEs - 2002> <2019-05-28 11:53:58,368> <_threaded_cve_gathering:148> <... Including CVEs - 2003> <2019-05-28 11:53:58,370> <_threaded_cve_gathering:148> <... Including CVEs - 2004> <2019-05-28 11:53:58,371> <_threaded_cve_gathering:148> <... Including CVEs - 2005> <2019-05-28 11:53:58,373> <_threaded_cve_gathering:148> <... Including CVEs - 2006> <2019-05-28 11:53:58,381> <_threaded_cve_gathering:148> <... Including CVEs - 2007> <2019-05-28 11:53:58,383> <_threaded_cve_gathering:148> <... Including CVEs - 2008> <2019-05-28 11:53:58,386> <_threaded_cve_gathering:148> <... Including CVEs - 2009> <2019-05-28 11:53:58,388> <_threaded_cve_gathering:148> <... Including CVEs - 2010> <2019-05-28 11:53:58,394> <_threaded_cve_gathering:148> <... Including CVEs - 2011> <2019-05-28 11:53:58,397> <_threaded_cve_gathering:148> <... Including CVEs - 2012> <2019-05-28 11:53:58,400> <_threaded_cve_gathering:148> <... Including CVEs - 2013> <2019-05-28 11:53:58,403> <_threaded_cve_gathering:148> <... Including CVEs - 2014> <2019-05-28 11:53:58,405> <_threaded_cve_gathering:148> <... Including CVEs - 2015> <2019-05-28 11:53:58,407> <_threaded_cve_gathering:148> <... Including CVEs - 2016> <2019-05-28 11:53:58,408> <_threaded_cve_gathering:148> <... Including CVEs - 2017> <2019-05-28 11:53:58,410> <_threaded_cve_gathering:148> <... Including CVEs - 2018> <2019-05-28 11:53:58,412> <_threaded_cve_gathering:148> <... Including CVEs - 2019>

root@ubuntu:/home/joon/Downloads/dagda-master/dagda# netstat -an | grep 5000 tcp 0 0 127.0.0.1:5000 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:36200 127.0.0.1:5000 TIME_WAIT
tcp 0 0 127.0.0.1:36198 127.0.0.1:5000 TIME_WAIT

in this situation, i excuted python3 dagda.py vuln --product openldap --product_version 2.2.20, then these error came out { "err": 500, "msg": "Internal Server Error" } in start dagda terminal <2019-05-28 16:03:59,356> <Exception on /v1/vuln/products/openldap/2.2.20 [GET]> Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2292, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1815, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python3.6/dist-packages/flask_cors/extension.py", line 161, in wrapped_function return cors_after_request(app.make_response(f(*args, kwargs))) File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1718, in handle_user_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 35, in reraise raise value File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1813, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1799, in dispatch_request return self.view_functions[rule.endpoint](req.view_args) File "/home/joon/Downloads/dagda-master/dagda/api/service/vuln.py", line 57, in get_vulns_by_product_and_version vulns = InternalServer.get_mongodb_driver().get_vulnerabilities(product, version) File "/home/joon/Downloads/dagda-master/dagda/driver/mongodb_driver.py", line 325, in get_vulnerabilities for bid in bid_cursor: File "/usr/local/lib/python3.6/dist-packages/pymongo/cursor.py", line 1189, in next if len(self.data) or self._refresh(): File "/usr/local/lib/python3.6/dist-packages/pymongo/cursor.py", line 1104, in _refresh self.send_message(q) File "/usr/local/lib/python3.6/dist-packages/pymongo/cursor.py", line 982, in __send_message helpers._check_command_response(first) File "/usr/local/lib/python3.6/dist-packages/pymongo/helpers.py", line 155, in _check_command_response raise OperationFailure(msg % errmsg, code, response) pymongo.errors.OperationFailure: text index required for $text query (no such collection 'vuln_database.bid')

everytime i try, i erased and reinstalled mongodb to clean every collections, and my ubuntu RAM is 4GB. i can't find out what's wrong in my environment settings again. help me please !

3lcarry commented 5 years ago

Hi, I'm running the same issue, additional after some time the vuln --init_status give me the following error: { "status": "Unexpected exception of type ValueError occurred: ('substring not found',)", "timestamp": "2019-06-09 11:05:03.552788" }

I've tried with mongo 3.6 and 4.

Additional on the server I've seen the following error:

<2019-06-09 13:04:12,478> <2019-06-09 13:04:12,479> <Updating Exploit DB collection ...> <2019-06-09 13:04:18,904> <2019-06-09 13:04:18,904> <Updating RHSA & RHBA collections ...> <2019-06-09 13:05:03,544> <_init_or_update_db:155> <Unexpected exception of type ValueError occurred: ('substring not found',)> Traceback (most recent call last): File "/home/carrillo/tfm/dagda/dagda/api/dagda_server.py", line 150, in _init_or_update_db db_composer.compose_vuln_db() File "/home/carrillo/tfm/dagda/dagda/vulnDB/db_composer.py", line 100, in compose_vuln_db rhsa_list, rhba_list, rhsa_info_list, rhba_info_list = get_rhsa_and_rhba_lists_from_file(bz2_file) File "/home/carrillo/tfm/dagda/dagda/vulnDB/ext_source_util.py", line 271, in get_rhsa_and_rhba_lists_from_file rhsa_id = rhsa_id[:rhsa_id.index("-", 5)] ValueError: substring not found

eliasgranderubio commented 5 years ago

Hi,

@3lcarry , my last commit should fix the ValueError you had when the init db process was run.

@ntart23 , if you are using the same machine for the MongoDB and the Dagda Server, you need at least 6GB RAM because MongoDB uses a lot of RAM for the indexes creation and the init db process uses RAM in the same way.

Regards.

3lcarry commented 5 years ago

Thank you @eliasgranderubio, now is working fine

@ntart23 I was having the same error but after initial sync disappeared (but I have 8GB of RAM)