search

doomedraven / Tools

Combination of different utilities, have fun!
MIT License
202 stars 96 forks source link

cuckoo.py and scripts don't run after installing based on scripts and blog tutorials #46

Closed benkittner closed 4 years ago

benkittner commented 4 years ago

Expected Behavior

After configuring and running cape2.sh and configuring files in config, cape starts up without warnings

Current Behavior

Running Cuckoo.py results in only the splash screen and a warning after some delay.

WARNING: You have enabled webgui but mongo ins't working, see mongodb manual for correct instalation and configuration

Cuckoo closes after this, returning control of bash to the user. Not sure if CAPE is meant to remain on the terminal, but I have other errors related to trying to start web\manage.py

Failure Information (for bugs)

Screenshot from 2020-07-09 16-16-06

Screenshot from 2020-07-09 16-05-17

Steps to Reproduce

Please provide detailed steps for reproducing the issue.

  1. install bare ubuntu 20.04 lts
  2. run kvm-qemu script, configure and install vms
  3. Run capev2.sh
  4. set config in files
  5. start rooter: sudo python3 utils/rooter.py /tmp/cuckoo-rooter -g cape &
  6. run cuckoo.py: sudo python3 cuckoo.py -h
  7. run python3 web/manage.py migrate

Context

Question Answer
OS version Ubuntu 20.04,

Failure Logs

cape2.sh.log kvm-qemu.sh.log

doomedraven commented 4 years ago

few small things, never run it as root, about mongo checj systemctl status mongodb what is wrong about migration, try to run it from inside of the /opt/CAPEv2/web folder not from /opt/CAPEv2/

benkittner commented 4 years ago

I get an error when running without root,:

Traceback (most recent call last):
  File "cuckoo.py", line 100, in <module>
    cuckoo_init(quiet=args.quiet, debug=args.debug, artwork=args.artwork, test=args.test)
  File "cuckoo.py", line 55, in cuckoo_init
    init_logging()
  File "/opt/CAPEv2/lib/cuckoo/core/startup.py", line 148, in init_logging
    fh = logging.handlers.TimedRotatingFileHandler(os.path.join(CUCKOO_ROOT, "log", "cuckoo.log"), when="midnight", backupCount=days)
  File "/usr/lib/python3.8/logging/handlers.py", line 200, in __init__
    BaseRotatingHandler.__init__(self, filename, 'a', encoding, delay)
  File "/usr/lib/python3.8/logging/handlers.py", line 55, in __init__
    logging.FileHandler.__init__(self, filename, mode, encoding, delay)
  File "/usr/lib/python3.8/logging/__init__.py", line 1143, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/lib/python3.8/logging/__init__.py", line 1172, in _open
    return open(self.baseFilename, self.mode, encoding=self.encoding)
PermissionError: [Errno 13] Permission denied: '/opt/CAPEv2/log/cuckoo.log'

I'm following the startup tutorial at https://medium.com/@soji256/build-a-cape-sandbox-to-analyze-emotet-3d507599dda6 which is why I'm expecting something like the following when running cuckoo.py:

2019-05-25 18:37:23,512 [modules.machinery.virtualbox] DEBUG: Getting status for Win7SP1x86 2019-05-25 18:37:23,640 [modules.machinery.virtualbox] DEBUG: Machine Win7SP1x86 status saved 2019-05-25 18:37:23,660 [lib.cuckoo.core.scheduler] INFO: Loaded 1 machine/s 2019-05-25 18:37:23,678 [lib.cuckoo.core.scheduler] INFO: Waiting for analysis tasks.

running systemctl yields:

boxy@boxy-Precision-7510:/opt/CAPEv2$ systemctl status mongod
● mongod.service - MongoDB Database Server
     Loaded: loaded (/lib/systemd/system/mongod.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: https://docs.mongodb.org/manual

Jul 09 16:22:49 boxy-Precision-7510 systemd[1]: /lib/systemd/system/mongod.service:11: PIDFile= references a path below legacy directory /var/run/, updating /var/run/mongodb/mongod.pid → /run/mongodb/mo>

boxy@boxy-Precision-7510:/opt/CAPEv2$ systemctl status mongodb
● mongodb.service - High-performance, schema-free document-oriented database
     Loaded: loaded (/etc/systemd/system/mongodb.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2020-07-08 16:46:36 UTC; 23h ago
   Main PID: 38463 (code=exited, status=100)

Jul 08 16:46:36 boxy-Precision-7510 systemd[1]: mongodb.service: Scheduled restart job, restart counter is at 5.
Jul 08 16:46:36 boxy-Precision-7510 systemd[1]: Stopped High-performance, schema-free document-oriented database.
Jul 08 16:46:36 boxy-Precision-7510 systemd[1]: mongodb.service: Start request repeated too quickly.
Jul 08 16:46:36 boxy-Precision-7510 systemd[1]: mongodb.service: Failed with result 'exit-code'.
Jul 08 16:46:36 boxy-Precision-7510 systemd[1]: Failed to start High-performance, schema-free document-oriented database.
doomedraven commented 4 years ago

well permission error is simple to solve, there is https://github.com/doomedraven/Tools/blob/master/Sandbox/cape2.sh#L839 to handle that, and default value is cape, so yu just was need run with correct username

well about systemd that is 3rd part dependency, this stuff chagnes so frequently that i can't to monitor without community help, i will check that to change

doomedraven commented 4 years ago

i suggest you to change in /lib/systemd/system/mongod.service the /var/run/ for PIDFile to /tmp/

doomedraven commented 4 years ago

i have changed that in script

doomedraven commented 4 years ago

also all those blogpost normaly super outdated

benkittner commented 4 years ago

Yeah, I figured the blog posts aren't fully up to date capev1 and without scripts, but they were the only documentation for cape I could find. They're still good for documenting what needs to be changed in config I think.

Trying your edit I was able to start mongod but mongodb is still out with the same error, except the status for pid is now 48 instead of 100.

As far as changing my user to 'cape' I don't seem to have the correct password for su, I've tried 'SuperPuperSecret' and my root password to no effect.

doomedraven commented 4 years ago

https://capev2.readthedocs.io/en/latest/ SuperPuperSecret isn't root pass, is db passwd so root pass is your pass after edit moongodb service you need to reload deamon and restart it

benkittner commented 4 years ago

Thanks for the docs, for some reason my root password is resulting in authentication failure when using su, my primary user is the admin (as this is more of a test / demonstration run than anything else), but I'm not sure if that has any bearing.

I tried reloading and restarting both services after saving and mongod still isn't being responsive:

boxy@boxy-Precision-7510:/lib/systemd/system$ sudo systemctl daemon-reload
boxy@boxy-Precision-7510:/lib/systemd/system$ sudo systemctl restart mongod.service 
boxy@boxy-Precision-7510:/lib/systemd/system$ sudo systemctl restart mongodb.service 
boxy@boxy-Precision-7510:/lib/systemd/system$ sudo systemctl status mongod.service 
● mongod.service - MongoDB Database Server
     Loaded: loaded (/lib/systemd/system/mongod.service; disabled; vendor preset: enabled)
     Active: active (running) since Thu 2020-07-09 17:32:55 UTC; 11s ago
       Docs: https://docs.mongodb.org/manual
   Main PID: 66879 (mongod)
     Memory: 170.6M
     CGroup: /system.slice/mongod.service
             └─66879 /usr/bin/mongod --config /etc/mongod.conf

Jul 09 17:32:55 boxy-Precision-7510 systemd[1]: Started MongoDB Database Server.
boxy@boxy-Precision-7510:/lib/systemd/system$ sudo systemctl restart mongodb.service 
boxy@boxy-Precision-7510:/lib/systemd/system$ sudo systemctl status mongodb.service 
● mongodb.service - High-performance, schema-free document-oriented database
     Loaded: loaded (/etc/systemd/system/mongodb.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2020-07-09 17:33:12 UTC; 6s ago
    Process: 66996 ExecStart=/usr/bin/numactl --interleave=all /usr/bin/mongod (code=exited, status=48)
   Main PID: 66996 (code=exited, status=48)

Jul 09 17:33:12 boxy-Precision-7510 systemd[1]: mongodb.service: Scheduled restart job, restart counter is at 5.
Jul 09 17:33:12 boxy-Precision-7510 systemd[1]: Stopped High-performance, schema-free document-oriented database.
Jul 09 17:33:12 boxy-Precision-7510 systemd[1]: mongodb.service: Start request repeated too quickly.
Jul 09 17:33:12 boxy-Precision-7510 systemd[1]: mongodb.service: Failed with result 'exit-code'.
Jul 09 17:33:12 boxy-Precision-7510 systemd[1]: Failed to start High-performance, schema-free document-oriented database.
doomedraven commented 4 years ago

What about sudo su? Instead of su? As su is debian more if i remver correctly

Try to start mongo by hand from that command in service

El jue., 9 jul. 2020 19:40, Solumnant notifications@github.com escribió:

Thanks for the docs, for some reason my root password is resulting in authentication failure when using su, my primary user is the admin (as this is more of a test / demonstration run than anything else), but I'm not sure if that has any bearing.

I tried reloading and restarting both services after saving and mongod still isn't being responsive:

boxy@boxy-Precision-7510:/lib/systemd/system$ sudo systemctl daemon-reload boxy@boxy-Precision-7510:/lib/systemd/system$ sudo systemctl restart mongod.service boxy@boxy-Precision-7510:/lib/systemd/system$ sudo systemctl restart mongodb.service boxy@boxy-Precision-7510:/lib/systemd/system$ sudo systemctl status mongod.service ● mongod.service - MongoDB Database Server

 Loaded: loaded (/lib/systemd/system/mongod.service; disabled; vendor preset: enabled)

 Active: *active (running)* since Thu 2020-07-09 17:32:55 UTC; 11s ago

   Docs: https://docs.mongodb.org/manual

Main PID: 66879 (mongod)

 Memory: 170.6M

 CGroup: /system.slice/mongod.service

         └─66879 /usr/bin/mongod --config /etc/mongod.conf

Jul 09 17:32:55 boxy-Precision-7510 systemd[1]: Started MongoDB Database Server. boxy@boxy-Precision-7510:/lib/systemd/system$ sudo systemctl restart mongodb.service boxy@boxy-Precision-7510:/lib/systemd/system$ sudo systemctl status mongodb.service ● mongodb.service - High-performance, schema-free document-oriented database

 Loaded: loaded (/etc/systemd/system/mongodb.service; enabled; vendor preset: enabled)

 Active: *failed* (Result: exit-code) since Thu 2020-07-09 17:33:12 UTC; 6s ago

Process: 66996 ExecStart=/usr/bin/numactl --interleave=all /usr/bin/mongod *(code=exited, status=48)*

Main PID: 66996 (code=exited, status=48)

Jul 09 17:33:12 boxy-Precision-7510 systemd[1]: mongodb.service: Scheduled restart job, restart counter is at 5.

Jul 09 17:33:12 boxy-Precision-7510 systemd[1]: Stopped High-performance, schema-free document-oriented database.

Jul 09 17:33:12 boxy-Precision-7510 systemd[1]: mongodb.service: Start request repeated too quickly.

Jul 09 17:33:12 boxy-Precision-7510 systemd[1]: mongodb.service: Failed with result 'exit-code'.

Jul 09 17:33:12 boxy-Precision-7510 systemd[1]: Failed to start High-performance, schema-free document-oriented database.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/doomedraven/Tools/issues/46#issuecomment-656260276, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOFH3YFUWUOIDDLGBS7TODR2X6HPANCNFSM4OVYLUZQ .

benkittner commented 4 years ago 
boxy@boxy-Precision-7510:/lib/systemd/system$ sudo service  mongod restart 
boxy@boxy-Precision-7510:/lib/systemd/system$ sudo service  mongodb restart 
boxy@boxy-Precision-7510:/lib/systemd/system$ sudo systemctl status mongodb.service 
● mongodb.service - High-performance, schema-free document-oriented database
     Loaded: loaded (/etc/systemd/system/mongodb.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2020-07-09 18:00:32 UTC; 12s ago
    Process: 68484 ExecStart=/usr/bin/numactl --interleave=all /usr/bin/mongod (code=exited, status=48)
   Main PID: 68484 (code=exited, status=48)

Jul 09 18:00:32 boxy-Precision-7510 systemd[1]: mongodb.service: Scheduled restart job, restart counter is at 5.
Jul 09 18:00:32 boxy-Precision-7510 systemd[1]: Stopped High-performance, schema-free document-oriented database.
Jul 09 18:00:32 boxy-Precision-7510 systemd[1]: mongodb.service: Start request repeated too quickly.
Jul 09 18:00:32 boxy-Precision-7510 systemd[1]: mongodb.service: Failed with result 'exit-code'.
Jul 09 18:00:32 boxy-Precision-7510 systemd[1]: Failed to start High-performance, schema-free document-oriented database.
boxy@boxy-Precision-7510:/lib/systemd/system$
doomedraven commented 4 years ago

/usr/bin/numactl --interleave=all /usr/bin/mongod

benkittner commented 4 years ago 
sudo /usr/bin/numactl --interleave=all /usr/bin/mongod
[sudo] password for boxy: 
2020-07-09T18:16:52.926+0000 I  CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2020-07-09T18:16:52.929+0000 W  ASIO     [main] No TransportLayer configured during NetworkInterface startup
2020-07-09T18:16:52.929+0000 I  CONTROL  [initandlisten] MongoDB starting : pid=68774 port=27017 dbpath=/data/db 64-bit host=boxy-Precision-7510
2020-07-09T18:16:52.929+0000 I  CONTROL  [initandlisten] db version v4.2.8
2020-07-09T18:16:52.929+0000 I  CONTROL  [initandlisten] git version: 43d25964249164d76d5e04dd6cf38f6111e21f5f
2020-07-09T18:16:52.929+0000 I  CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020
2020-07-09T18:16:52.929+0000 I  CONTROL  [initandlisten] allocator: tcmalloc
2020-07-09T18:16:52.929+0000 I  CONTROL  [initandlisten] modules: none
2020-07-09T18:16:52.929+0000 I  CONTROL  [initandlisten] build environment:
2020-07-09T18:16:52.930+0000 I  CONTROL  [initandlisten]     distmod: ubuntu1804
2020-07-09T18:16:52.930+0000 I  CONTROL  [initandlisten]     distarch: x86_64
2020-07-09T18:16:52.930+0000 I  CONTROL  [initandlisten]     target_arch: x86_64
2020-07-09T18:16:52.930+0000 I  CONTROL  [initandlisten] options: {}
2020-07-09T18:16:52.930+0000 E  STORAGE  [initandlisten] Failed to set up listener: SocketException: Address already in use
2020-07-09T18:16:52.930+0000 I  CONTROL  [initandlisten] now exiting
2020-07-09T18:16:52.930+0000 I  CONTROL  [initandlisten] shutting down with code:48

oxy@boxy-Precision-7510:/lib/systemd/system$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04 LTS
Release:    20.04
Codename:   focal
benkittner commented 4 years ago

mongod starts fine, it's mongodb that isn't cooperating.

I'm not sure the difference between them, or if there is one at all.

benkittner commented 4 years ago

Also, still getting permission denied when running under user cape:

cape@boxy-Precision-7510:/opt/CAPEv2$ python3 cuckoo.py -d

   _______ _     _ _______ _     _  _____   _____
   |       |     | |       |____/  |     | |     |
   |_____  |_____| |_____  |    \_ |_____| |_____|

 Cuckoo Sandbox 2.1-CAPE
 www.cuckoosandbox.org
 Copyright (c) 2010-2015

 CAPE: Config and Payload Extraction
 github.com/kevoreilly/CAPEv2

Traceback (most recent call last):
  File "cuckoo.py", line 100, in <module>
    cuckoo_init(quiet=args.quiet, debug=args.debug, artwork=args.artwork, test=args.test)
  File "cuckoo.py", line 55, in cuckoo_init
    init_logging()
  File "/opt/CAPEv2/lib/cuckoo/core/startup.py", line 148, in init_logging
    fh = logging.handlers.TimedRotatingFileHandler(os.path.join(CUCKOO_ROOT, "log", "cuckoo.log"), when="midnight", backupCount=days)
  File "/usr/lib/python3.8/logging/handlers.py", line 200, in __init__
    BaseRotatingHandler.__init__(self, filename, 'a', encoding, delay)
  File "/usr/lib/python3.8/logging/handlers.py", line 55, in __init__
    logging.FileHandler.__init__(self, filename, mode, encoding, delay)
  File "/usr/lib/python3.8/logging/__init__.py", line 1143, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/lib/python3.8/logging/__init__.py", line 1172, in _open
    return open(self.baseFilename, self.mode, encoding=self.encoding)
PermissionError: [Errno 13] Permission denied: '/opt/CAPEv2/log/cuckoo.log'
benkittner commented 4 years ago

Also, thank you for going above and beyond in support of this product, and especially in assisting with issues.

doomedraven commented 4 years ago

ya bcz you started it as root before, you need to use chown as i posted before im removing mongod version here https://github.com/doomedraven/Tools/blob/master/Sandbox/cape2.sh#L510

you are welcome

doomedraven commented 4 years ago

as sudo chown cape:cape -R "/opt/CAPEv2/"

benkittner commented 4 years ago

Thank you, it's running now. It complains about:

invalid syntax (recon_fingerprint.py, line 37)

but it says it's waiting for analysis tasks so it looks fine. I'm going to look into that complaint, but otherwise I think the issue is closed.

benkittner commented 4 years ago

Oh, I see, you're removing mongod in favor of mongodb.

doomedraven commented 4 years ago

ya that are signatures and they made by community so they might have some fails, i will check later what is wrong with it

benkittner commented 4 years ago

Thank you for the readthedocs page. As a suggestion, you could advertise it on the main readme for the capev2 project so people are aware it exists.

doomedraven commented 4 years ago

Will add it later thanks

El vie., 10 jul. 2020 19:05, Solumnant notifications@github.com escribió:

Thank you for the readthedocs page. As a suggestion, you could advertise it on the main readme for the capev2 project so people are aware it exists.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/doomedraven/Tools/issues/46#issuecomment-656783236, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOFH34TBFK3B2GB26ABFLDR25C6FANCNFSM4OVYLUZQ .

benkittner commented 4 years ago

Having pulled the most recent scripts and installing them on a new machine, I still have the

"you have enabled webgui but mongo isn't working"

warning.

The same manual fix you mentioned earlier still works.

doomedraven commented 4 years ago

as i told few times, this is automated stuff that should work out of the box, if that doesn't work you need to investigate why it doesn't work, as in theory i should just put that all in documentation and you would need to exec line by line, is your work to at least read the log output to detect if something doesn't work fine ;)

benkittner commented 4 years ago

Yeah, I figured it out by reading the logs, and I've fixed my own install, I just thought I'd give you a bug report for your script. I'll be back with the install logs to help identify what went wrong.

doomedraven commented 4 years ago

yep the bugs are welcome, but we fixed another day it with you ;)

doomedraven commented 4 years ago

the problem is that if you just run script or doing clean install? if just reexec, that won't fix if something doesn't work

benkittner commented 4 years ago

100% clean install, wiped the partition clean since I realized I had duplicate services trying to listen on the same port and I felt it would be better to start from scratch with my new understanding.

I reinstalled 20.04 from media, overwriting the partition I was working with before, and ran the two scripts for the first time on this new install. Sorry for the confusion.

On Mon, Jul 13, 2020 at 4:24 PM doomedraven notifications@github.com wrote:

the problem is that if you just run script or doing clean install? if just reexec, that won't fix if something doesn't work

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/doomedraven/Tools/issues/46#issuecomment-657772493, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEEEQJWVY4J5MOOU3XR5XSDR3NUQDANCNFSM4OVYLUZQ .

doomedraven commented 4 years ago

em that is really weird, let me know what is wrong if you find the reason

benkittner commented 4 years ago

I have the log, not sure why it happened though cape2.sh.log

doomedraven commented 4 years ago

i don't see any problem