mhn install fails on linux ubuntu (14.4/16.4)

[0x7fff9]

Hello, sorry to open an issue.

When installing MHN on UBUNTU 14.4 or 16.4 I always end up with this stack trace:

Traceback (most recent call last):
  File "initdatabase.py", line 5, in <module>
    create_clean_db()
  File "/opt/mhn/server/mhn/__init__.py", line 177, in create_clean_db
    fetch_sources()
  File "/opt/mhn/env/local/lib/python2.7/site-packages/celery/local.py", line 167, in <lambda>
    __call__ = lambda x, *a, **kw: x._get_current_object()(*a, **kw)
  File "/opt/mhn/server/mhn/tasks/__init__.py", line 13, in __call__
    return TaskBase.__call__(self, *args, **kwargs)
  File "/opt/mhn/env/local/lib/python2.7/site-packages/celery/app/task.py", line 420, in __call__
    return self.run(*args, **kwargs)
  File "/opt/mhn/server/mhn/tasks/rules.py", line 77, in fetch_sources
    Rule.bulk_import(rules)
  File "/opt/mhn/server/mhn/api/models.py", line 185, in bulk_import
    rule.insert_refs(ru['references'])
  File "/opt/mhn/server/mhn/api/models.py", line 132, in insert_refs
    db.session.commit()
  File "/opt/mhn/env/local/lib/python2.7/site-packages/sqlalchemy/orm/scoping.py", line 157, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/opt/mhn/env/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 874, in commit
    self.transaction.commit()
  File "/opt/mhn/env/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 461, in commit
    self._prepare_impl()
  File "/opt/mhn/env/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 441, in _prepare_impl
    self.session.flush()
  File "/opt/mhn/env/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2139, in flush
    self._flush(objects)
  File "/opt/mhn/env/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2259, in _flush
    transaction.rollback(_capture_exception=True)
  File "/opt/mhn/env/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/opt/mhn/env/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2223, in _flush
    flush_context.execute()
  File "/opt/mhn/env/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute
    rec.execute(self)
  File "/opt/mhn/env/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute
    uow
  File "/opt/mhn/env/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 181, in save_obj
    mapper, table, insert)
  File "/opt/mhn/env/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 835, in _emit_insert_statements
    execute(statement, params)
  File "/opt/mhn/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 945, in execute
    return meth(self, multiparams, params)
  File "/opt/mhn/env/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/opt/mhn/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/opt/mhn/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
    context)
  File "/opt/mhn/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1394, in _handle_dbapi_exception
    exc_info
  File "/opt/mhn/env/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/opt/mhn/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "/opt/mhn/env/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (pysqlite2.dbapi2.ProgrammingError) You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. [SQL: u'INSERT INTO rule_references (text, rule_id) VALUES (?, ?)'] [parameters: ('url,blogs.forcepoint.com/security-labs/trojanized-adobe-installer-used-install-dragonok\xe2\x80\x99s-new-custom-backdoor', 7547)]

I look at mongo and it seems fine

gg@ubuntu:/opt/mhn$ mongo
MongoDB shell version: 3.2.12
connecting to: test
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
    http://docs.mongodb.org/
Questions? Try the support group
    http://groups.google.com/group/mongodb-user
Server has startup warnings: 
2017-04-01T12:47:58.053+0200 I CONTROL  [initandlisten] 
2017-04-01T12:47:58.053+0200 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2017-04-01T12:47:58.053+0200 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2017-04-01T12:47:58.053+0200 I CONTROL  [initandlisten] 
2017-04-01T12:47:58.053+0200 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2017-04-01T12:47:58.053+0200 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2017-04-01T12:47:58.053+0200 I CONTROL  [initandlisten] 
> show dbs
hpfeeds    0.000GB
local      0.000GB
mnemosyne  0.000GB

This is on 16.04 and happens right after Imported 7500 rules so far...

anyone experiencing it?

thanks!

cheers, Guido.

[tvickory]

I am spinning them up on Digital Ocean servers and I get the same error. No idea how to fix

[0x7fff9]

weird. I had all working ok on UBUNTU 16 for tests and now new builds can't go.

I will try to change the mongo installation script as I believe the issue is with Mongo and will post back the results.

testing replace: echo "deb http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list

with echo "deb [ arch=amd64,arm64 ] http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.4.list

cheers.

[tvickory]

It seems like the database init gets to about 7000 and then it fails on the same file every time. I will post the file that causes it to fail when I get back to my desk.

On Apr 1, 2017, at 9:47 AM, Guido Galego notifications@github.com wrote:

weird. I had all working ok on UBUNTU 16 for tests and now new builds can't go.

I will try to change the mongo installation script as I believe the issue is with Mongo and will post back the results.

testing replace: echo "deb http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list

with echo "deb [ arch=amd64,arm64 ] http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.4.list

cheers.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[tvickory]

it gets to here: 'url, blogs.forcepoint.com/security-labs/trojanized-adobe-installer-used-install-dragonok\xe2\x80\x99s-new-custom-backdoor' then fails with a unicode error. Commenting out init_database.py in the install_mhn_server.sh file is the only way to get the install to finish. After that everything seems to work fine except for the deployment of suricata and snort as when it goes to get check the prerendered list it 404s. Tried restarted all the mhn services, no luck

On Sat, Apr 1, 2017 at 9:58 AM, Trey Vickory tvickory@gmail.com wrote:

It seems like the database init gets to about 7000 and then it fails on the same file every time. I will post the file that causes it to fail when I get back to my desk.

On Apr 1, 2017, at 9:47 AM, Guido Galego notifications@github.com wrote:

weird. I had all working ok on UBUNTU 16 for tests and now new builds can't go.

I will try to change the mongo installation script as I believe the issue is with Mongo and will post back the results.

testing replace: echo "deb http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list

with echo "deb [ arch=amd64,arm64 ] http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.4 multiverse" | sudo tee /etc/apt/sources.list.d/ mongodb-org-3.4.list

cheers.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/threatstream/mhn/issues/383#issuecomment-290921115, or mute the thread https://github.com/notifications/unsubscribe-auth/AIORHXnxbyjgARBtkCqsRj-hUevjjFh9ks5rrlVjgaJpZM4MwdB8 .

[0x7fff9]

the real url for that is https://blogs.forcepoint.com/security-labs/trojanized-adobe-installer-used-install-dragonok%E2%80%99s-new-custom-backdoor why is it looking for that, any idea?

thanks for the comment tip! indeed it passes for the rest of the install after the Imported 7500 rules so far...

Let's wait and see if there is a fix for this, a lot of users should be experiencing this on new deployments.

cheers.

[tvickory]

Not sure. I think it is adding it as a rule to the database. When the install finishes are you able to see the prendered rules list on the web instance. Mine 404s

On Apr 1, 2017, at 11:19 AM, Guido Galego notifications@github.com wrote:

the real url for that is https://blogs.forcepoint.com/security-labs/trojanized-adobe-installer-used-install-dragonok%E2%80%99s-new-custom-backdoor why is it looking for that, any idea?

thanks for the comment tip! indeed it passes for the rest of the install after the Imported 7500 rules so far...

Let's wait and see if there is a fix for this, a lot of users should be experiencing this on new deployments.

cheers.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[0x7fff9]

yes but I think that's because once we comment out the init_database.py from the install script the rules are no longer downloaded. Also this must impact snort as these rules relate to it. it's a shame that can't make this work perfectly, the last build I had was OK, so I really believe there is a problem with the last commit where the mongo install was changed.

I'll try to use the 3.4 and tell you how it went.

[0x7fff9]

just tested. Mongo version is not related.

[tvickory]

I'm going through the inti_database.py file to see if I can't force cast everything to correct unicode format before it tries to insert into the database. I'll let you know if I come up with a work around.

On Sat, Apr 1, 2017 at 11:58 AM, Guido Galego notifications@github.com wrote:

just tested. Mongo version is not related.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/threatstream/mhn/issues/383#issuecomment-290929007, or mute the thread https://github.com/notifications/unsubscribe-auth/AIORHW0LdJXFi3mmSfl5u96t6vjXfLFxks5rrnQpgaJpZM4MwdB8 .

[0x7fff9]

ok, that's a good idea. I have forced to unicode mode on the locale so that it isn't picking weird stuff like ascii, but yeah nothing. also just tried on centos 6.7, exactly the same error.

[juju4]

Also have this error appeared in my latest ansible role travis role for some unrelated change... https://travis-ci.org/juju4/ansible-mhn/builds/217658632

Not sure why have utf8 here but that can happen, so let's decode properly inside code. I managed to get it work by changing

ref.text = r

into

ref.text = r.decode('utf-8')

inside /var/_mhn/mhn/server/mhn/api/models.py (or whatever path you use) but instead got at the end of initdatabase

Imported 20500 rules so far...
Imported 21000 rules so far...
Finished Importing 21000 rules.  Committing data
Traceback (most recent call last):
  File "initdatabase.py", line 5, in <module>
    create_clean_db()
  File "/var/_mhn/mhn/server/mhn/__init__.py", line 177, in create_clean_db
    fetch_sources()
  File "/var/_mhn/mhn/env/local/lib/python2.7/site-packages/celery/local.py", line 167, in <lambda>
    __call__ = lambda x, *a, **kw: x._get_current_object()(*a, **kw)
  File "/var/_mhn/mhn/server/mhn/tasks/__init__.py", line 13, in __call__
    return TaskBase.__call__(self, *args, **kwargs)
  File "/var/_mhn/mhn/env/local/lib/python2.7/site-packages/celery/app/task.py", line 420, in __call__
    return self.run(*args, **kwargs)
  File "/var/_mhn/mhn/server/mhn/tasks/rules.py", line 78, in fetch_sources
    render_rules()
  File "/var/_mhn/mhn/env/local/lib/python2.7/site-packages/celery/local.py", line 167, in <lambda>
    __call__ = lambda x, *a, **kw: x._get_current_object()(*a, **kw)
  File "/var/_mhn/mhn/server/mhn/tasks/__init__.py", line 13, in __call__
    return TaskBase.__call__(self, *args, **kwargs)
  File "/var/_mhn/mhn/env/local/lib/python2.7/site-packages/celery/app/task.py", line 420, in __call__
    return self.run(*args, **kwargs)
  File "/var/_mhn/mhn/server/mhn/tasks/rules.py", line 21, in render_rules
    sbuffer = Rule.renderall()
  File "/var/_mhn/mhn/server/mhn/api/models.py", line 169, in renderall
    return '\n\n'.join([ru.render() for ru in rules])
  File "/var/_mhn/mhn/server/mhn/api/models.py", line 153, in render
    reference += 'reference:{}; '.format(r.text)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 87: ordinal not in range(128)

Adding urllib.quote_plus()

Imported 7500 rules so far...
Traceback (most recent call last):
  File "initdatabase.py", line 5, in <module>
    create_clean_db()
  File "/var/_mhn/mhn/server/mhn/__init__.py", line 177, in create_clean_db
    fetch_sources()
  File "/var/_mhn/mhn/env/local/lib/python2.7/site-packages/celery/local.py", line 167, in <lambda>
    __call__ = lambda x, *a, **kw: x._get_current_object()(*a, **kw)
  File "/var/_mhn/mhn/server/mhn/tasks/__init__.py", line 13, in __call__
    return TaskBase.__call__(self, *args, **kwargs)
  File "/var/_mhn/mhn/env/local/lib/python2.7/site-packages/celery/app/task.py", line 420, in __call__
    return self.run(*args, **kwargs)
  File "/var/_mhn/mhn/server/mhn/tasks/rules.py", line 77, in fetch_sources
    Rule.bulk_import(rules)
  File "/var/_mhn/mhn/server/mhn/api/models.py", line 187, in bulk_import
    rule.insert_refs(ru['references'])
  File "/var/_mhn/mhn/server/mhn/api/models.py", line 131, in insert_refs
    ref.text = urllib.quote_plus(r.decode('utf-8'))
  File "/usr/lib/python2.7/urllib.py", line 1295, in quote_plus
    return quote(s, safe)
  File "/usr/lib/python2.7/urllib.py", line 1288, in quote
    return ''.join(map(quoter, s))
KeyError: u'\u2019'

what next?

[0x7fff9]

Great FIND from you!! HATS OFF!!!

to complete add what you found ref.text = r.decode('utf-8') and also on the script begin add:

import sys
reload(sys)
sys.setdefaultencoding('utf-8')

result:

gg@dev-ubnt-14-04:/opt/mhn$ sudo supervisorctl status
geoloc                           RUNNING   pid 31888, uptime 0:12:11
honeymap                         RUNNING   pid 31889, uptime 0:12:11
hpfeeds-broker                   RUNNING   pid 12441, uptime 0:13:35
mhn-celery-beat                  RUNNING   pid 23035, uptime 0:02:21
mhn-celery-worker                RUNNING   pid 23115, uptime 0:02:05
mhn-collector                    RUNNING   pid 23037, uptime 0:02:21
mhn-uwsgi                        RUNNING   pid 23039, uptime 0:02:21
mnemosyne                        RUNNING   pid 30788, uptime 0:12:31

gg@dev-ubnt-14-04:/opt/mhn$ ls server/mhn/static/
css  dionaea.conf  hpfeeds.py  ihandlers.py  img  js  mhn.rules  registration.txt

Godd stuff. THANKS!!!

[0x7fff9]

considering this is a fix... if you run ELK, you'll notice that:

{:timestamp=>"2017-04-03T02:34:40.644000+0200", :message=>"You are using a deprecated config setting \"index_type\" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Please use the 'document_type' setting instead. It has the same effect, but is more appropriately named. If you have any questions about this, please visit the #logstash channel on freenode irc.", :name=>"index_type", :plugin=><LogStash::Outputs::ElasticSearch host=>"127.0.0.1", port=>9200, protocol=>"http", index=>"mhn-%{+YYYYMMddHH00}", index_type=>"event", template_name=>"mhn_event", template=>"/opt/logstash/mhn-template.json", template_overwrite=>"true", manage_template=>"true">, :level=>:warn}
{:timestamp=>"2017-04-03T02:34:40.890000+0200", :message=>"No SINCEDB_DIR or HOME environment variable set, I don't know where to keep track of the files I'm watching. Either set HOME or SINCEDB_DIR in your environment, or set sincedb_path in in your Logstash config for the file input with path '[\"/var/log/mhn/mhn-json.log\"]'", :level=>:error}
The error reported is: 

the deprecated error about index_type is no problem. just add sincedb_path => "/opt/logstash/sincedb" on /opt/logstash/mhn.conf to fix it.

EG. default:

input {
  file {
    path => "/var/log/mhn/mhn-json.log"
    start_position => "end"
  }

after fix:

input {
  file {
    path => "/var/log/mhn/mhn-json.log"
    start_position => "end"
    sincedb_path => "/opt/logstash/sincedb"
  }

[RyanLindfield]

After editing the r.text entry in models.py as described above, I get through the rules import (20990 rules ), it moves into "rendering rules" but then fails, last line of traceback with error: UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 87: ordinal not in range(128)

[0x7fff9]

to bypass u'\u2019' encoding issues you'd have to add

import sys
reload(sys)
sys.setdefaultencoding('utf-8')

on models.py beginning.