Api & account settings not available ?

[juju4]

Hello,

I'm trying 1.2 alpha and it find very light and visual, especially compare to alternatives :) My current problem, it seems the api links are not available

$ curl http://127.0.0.1:8080/api/query?value=malcom.io This page does not exist

checking webserver.py $ curl http://127.0.0.1:8080/public/api?value=malcom.io [] $ curl http://127.0.0.1:8080/evil?value=bevrifuli.geohats.com {"nodes": [], "edges": []} $ curl http://127.0.0.1:8080/api/analytics Internal Server Error = ValueError: View function did not return a response

and grepping account inside source just returns one line $ grep -rin account * web/static/jquery/jquery-ui.js:13781: // into account and update option properly. = No /account/settings to generate API key

unreleased stuff? or I missed a doc/code ?

Thanks Tom

[tomchop]

Thanks for reporting the issue Julien. I just checked on my local Malcom instance and it seems I forgot to mention that there should be a trailing slash (i.e. $ curl http://127.0.0.1:8080/api/query/?value=malcom.io) when querying API URLs. I just changed the Wiki to reflect this.

Why did you grep for account? What were you looking for? Bear in mind that if you're using Malcom as a local instance with no authentication, you can just curl away whichever API you need. Otherwise, if you're using authentication, make sure to include the API key that was generated by the Malcom install (you can find that in Account settings)

[juju4]

same. made a git pull to check but already latest $ curl http://127.0.0.1:8080/api/query/?value=malcom.io This page does not exist

For authentication, how do you enable it? I suppose Readme.md install path is for local auth. Enabling auth is external like Apache htaccess or internal (no flags or code, it seems) ?

My setup is a Vagrantfile and script which does what's in Readme.md without virtualenv. feeds fetching and web ui seems working well. Outside of git clone, there is no install or setup.py, no? not seen any key generation and did a quick check in mongo db but didn't find it

[tomchop]

Nope, there's not install or setup.py. I think the problem is you're on the master branch. Authentication and API are available on the dev branch. Try git checkout dev and let me know if it worked for you.

[juju4]

$ git checkout dev Branch dev set up to track remote branch dev from origin. Switched to a new branch 'dev' $ ./malcom.py ===== Malcom 1.2a - Malware Communications Analyzer =====

Traceback (most recent call last): File "./malcom.py", line 58, in setup.load_config(args) File "/home/vagrant/malcom/Malcom/config/malconf.py", line 13, in load_config self.parse_command_line(args) File "/home/vagrant/malcom/Malcom/config/malconf.py", line 24, in parse_command_line self['AUTH'] = args.auth AttributeError: 'Namespace' object has no attribute 'auth'

If I do a config file inspired from malcom.py $ ./malcom.py -c malcom.cfg ===== Malcom 1.2a - Malware Communications Analyzer =====

Traceback (most recent call last): File "./malcom.py", line 58, in setup.load_config(args) File "/home/vagrant/malcom/Malcom/config/malconf.py", line 13, in load_config self.parse_command_line(args) File "/home/vagrant/malcom/Malcom/config/malconf.py", line 19, in parse_command_line self.parse_config_file(args.config) File "/home/vagrant/malcom/Malcom/config/malconf.py", line 33, in parse_config_file config.read(filename) File "/usr/lib/python2.7/ConfigParser.py", line 305, in read self._read(fp, filename) File "/usr/lib/python2.7/ConfigParser.py", line 512, in _read raise MissingSectionHeaderError(fpname, lineno, line) ConfigParser.MissingSectionHeaderError: File contains no section headers. file: malcom.cfg, line: 1 "malconf['DEBUG'] = True\n"

= added a section "[section]" top

===== Malcom 1.2a - Malware Communications Analyzer =====

Detected interfaces: eth1: 192.168.x.y eth0: 10.0.v.w Traceback (most recent call last): File "./malcom.py", line 75, in if setup['SNIFFER']: KeyError: 'SNIFFER'

same or other errors if I have no entry, {} or with multiple sections which could correspond config/malcom.py

[tomchop]

If you showed me your configuration file it might be easier to see where it's going wrong.

Try with something like this (the one I'm currently using)

[web]
activated = true
listen_interface = 0.0.0.0
listen_port = 8080
auth = false 

[feeds]
activated = false 
feeds_dir = /home/malcom/malcom/Malcom/feeds # must exist and contain feeds
scheduler = false

[analytics]
activated = false 
max_workers = 12

[sniffer]
activated = true
network = true
pcap = true
sniffer_dir = /home/malcom/malcom/Malcom/sniffer # must exist
tls_proxy_port = 9000
yara_path = /home/malcom/yararules

# databases can be configured here

[db_local]
name = Local
type = db
host = localhost
port = 27017

[feeds]
# Insert feeds here

[tomchop]

Any updates on this?

[juju4]

Sorry for the delay Tom. Restart my vagrant with latest git clone/1.3a and when I start it with a copy of example conf, got

$ ./malcom.py 2>&1 | tee malcom.log &
===== Malcom 1.3a - Malware Communications Analyzer =====

Detected interfaces:
eth1:   192.168.0.140
eth0:   10.0.2.15
WARNING: No route found for IPv6 destination :: (no default route?)
[+] Starting sniffer...
[+] Successfully loaded sniffer directory: /home/vagrant/malcom/Malcom/sniffer
[DEBUG] [2014-12-27 23:37:37.043151] - Rebuilding indexes...
[DEBUG] [2014-12-27 23:37:37.044987] - Done rebuilding indexes...
[DEBUG] [2014-12-27 23:37:37.045025] - [+] Subscribing to sniffer-commands
[DEBUG] [2014-12-27 23:37:37.045311] - [+] Sniffer Messenger started
[DEBUG] [2014-12-27 23:37:37.045323] - Compiling YARA rules from /home/vagrant/malcom/Malcom/yara
[DEBUG] [2014-12-27 23:37:37.045347] - Loaded 0 YARA rule files in /home/vagrant/malcom/Malcom/yara/
[DEBUG] [2014-12-27 23:37:37.056585] - Rebuilding indexes...
[DEBUG] [2014-12-27 23:37:37.058181] - Done rebuilding indexes...
Traceback (most recent call last):
  File "./malcom.py", line 112, in <module>
    from Malcom.web.webserver import MalcomWeb
  File "/home/vagrant/malcom/Malcom/web/webserver.py", line 178, in <module>
    @lm.request_loader
AttributeError: 'LoginManager' object has no attribute 'request_loader'

$ ./malcom.py -c malcom.conf 2>&1 | tee malcom.log &
===== Malcom 1.3a - Malware Communications Analyzer =====

Detected interfaces:
eth1:   192.168.0.140
eth0:   10.0.2.15
WARNING: No route found for IPv6 destination :: (no default route?)
[+] Starting sniffer...
[+] Successfully loaded sniffer directory: /home/vagrant/malcom/Malcom/sniffer
[+] Starting TLS proxy on port 9000
[+] Starting analytics engine...
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/home/vagrant/malcom/Malcom/networking/tlsproxy/tlsproxy.py", line 152, in run
    reactor.listenSSL(self.port, self.factory, ssl.DefaultOpenSSLContextFactory('Malcom/networking/tlsproxy/keys/server.key', 'Malcom/networking/tlsproxy/keys/server.crt'), interface="0.0.0.0")
  File "/usr/lib/python2.7/dist-packages/twisted/internet/ssl.py", line 70, in __init__
    self.cacheContext()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/ssl.py", line 79, in cacheContext
    ctx.use_certificate_file(self.certificateFileName)
Error: [('system library', 'fopen', 'No such file or directory'), ('BIO routines', 'FILE_CTRL', 'system lib'), ('SSL routines', 'SSL_CTX_use_certificate_file', 'system lib')]

[DEBUG] [2014-12-27 23:40:07.646780] - Rebuilding indexes...
[DEBUG] [2014-12-27 23:40:07.648222] - Done rebuilding indexes...
[DEBUG] [2014-12-27 23:40:07.648267] - [+] Subscribing to sniffer-commands
[DEBUG] [2014-12-27 23:40:07.648568] - [+] Sniffer Messenger started
[DEBUG] [2014-12-27 23:40:07.648621] - Compiling YARA rules from /home/vagrant/malcom/Malcom/yara
[DEBUG] [2014-12-27 23:40:07.648651] - Loaded 0 YARA rule files in /home/vagrant/malcom/Malcom/yara/
[DEBUG] [2014-12-27 23:40:07.657569] - Rebuilding indexes...
[DEBUG] [2014-12-27 23:40:07.658960] - Done rebuilding indexes...
[DEBUG] [2014-12-27 23:40:07.677056] - Rebuilding indexes...
[DEBUG] [2014-12-27 23:40:07.678455] - Done rebuilding indexes...
Traceback (most recent call last):
  File "./malcom.py", line 112, in <module>
    from Malcom.web.webserver import MalcomWeb
  File "/home/vagrant/malcom/Malcom/web/webserver.py", line 178, in <module>
    @lm.request_loader
AttributeError: 'LoginManager' object has no attribute 'request_loader'

If I disable the sniffer, I'm back to first case This is blocking for the web interface, not for analytics or feeds it seems In all case, web interface doesn't answer.

I did not find the command for the feeds scheduler. starting with -f/--feeds get the same message with "[!] Feed scheduler must be started manually." and nothing more happening (not like 1.1 or 1.2 downloading feeds message). got FeedEngine heartbeat but that's all

Also (just) try test_feeds.py

$ python test_feeds.py
[...]
tendonina.co.cc <SpyEye CC (tendonina.co.cc)>
dontstop2346210.com <SpyEye CC (dontstop2346210.com)>
instantsoft.info <SpyEye CC (instantsoft.info)>
Test on SpyEyeTracker succeeeded (40 elements fetched)
Testing feed CybercrimeTracker
Traceback (most recent call last):
  File "test_feeds.py", line 42, in <module>
    fe.feeds[feed].update()
  File "/home/vagrant/malcom/Malcom/feeds/cybercrimetracker.py", line 20, in update
    self.update_xml('item', ["title", "link", "pubDate", "description"])
  File "/home/vagrant/malcom/Malcom/feeds/feed.py", line 53, in update_xml
    tree = etree.parse(feed)
  File "lxml.etree.pyx", line 3239, in lxml.etree.parse (src/lxml/lxml.etree.c:69955)
  File "parser.pxi", line 1769, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:102257)
  File "parser.pxi", line 1789, in lxml.etree._parseFilelikeDocument (src/lxml/lxml.etree.c:102516)
  File "parser.pxi", line 1684, in lxml.etree._parseDocFromFilelike (src/lxml/lxml.etree.c:101442)
  File "parser.pxi", line 1134, in lxml.etree._BaseParser._parseDocFromFilelike (src/lxml/lxml.etree.c:97069)
  File "parser.pxi", line 582, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:91275)
  File "parser.pxi", line 683, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:92461)
  File "parser.pxi", line 622, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:91757)
lxml.etree.XMLSyntaxError: Opening and ending tag mismatch: img line 30 and center, line 31, column 10

[tomchop]

Getting back to you on this, sorry for the massive delay. Have you tried running the latest version of Malcom? There has been quite a lot of changes and bugfixes. Let me know.

[juju4]

Hello Tom, No problem. I'm currently upgrading my main system (testing Qubes) so probably not during the week, will try this week-end hopefully. (and congrats for new release stuff :)

[juju4]

Still too much to do testing on Qubes. back lubuntu/vagrant

Quick try, but still have the same error with the web module when launching malcom(AttributeError: 'LoginManager' object has no attribute 'request_loader'). other modules one by one or all start without error. test_feeds.py starts and stopped on an 'ordinal not in range'.

I copied default malcom.conf and git clone current.

Also gave a try to docker $ sudo docker pull tomchop/malcom Pulling repository tomchop/malcom fac211dad424: Download complete [... ends OK] $ sudo docker run -P -d --name malcom tomchop/malcom 3773510edb6c67c32589575b1c23c411edfa703c9f59860e8ce18f8674af4afc $ sudo docker port malcom Usage: docker port CONTAINER PRIVATE_PORT Lookup the public-facing port which is NAT-ed to PRIVATE_PORT $ dpkg -l |grep docker.io ii docker.io 1.0.1~dfsg1-0ubuntu1~ubuntu0.14.04.1 amd64 Linux container runtime (I know, it bit old..)

with a later one (1.5), 'docker port malcom' outputs nothing ...

[tomchop]

Are you using the latest Docker version? They recommend not to use the Ubuntu-managed package: https://docs.docker.com/installation/ubuntulinux/#installing-docker-on-ubuntu... that's what I followed for my installs. The tomchop/malcom Docker image should work correctly, but you can also try with tomchop/malcom-automatic (this one follows the latest commit on GitHub).

At any rate, the error you're getting when launching Malcom manually is still quite strange. I'll re-do the install procedure tonight. If you have time, please check that the requirements in requirements.txt are all installed (you can try doing one more pip install -r requirements.txt just to be sure).

Thanks for your help!

[tomchop]

Well, I just tested the install procedure from scratch on Ubuntu 14.04 and it works just fine. Are you sure you're installing all the packages from requirements.txt? Sounds like compatibility issues with flask-login. Let me know if I can help you further.

[juju4]

That's it ... if I go with upstream 'pip install -r requirements.txt', it works well. But if I stay with mostly ubuntu14.04 packages, it bugged (probably flask-login, but also had pip bugging me and had to update it too).

Seems there is really lot of nice change in the interface. great work!