Please, update the install procedure

vletoux commented 7 years ago

I was in the First 2016 in Seoul during the demo of the tool and I was quite amazed by its potential. Now I found some time to install it on a test VM. But if I can use Linux, I'm not a guru in this field.

I just tried to install it on a "test" VM and I spend the day without having to complete the install procedure. I couldn't test it ;-( I've tried CentOS7 (command line is not easy for copy pasting commands so I stopped), Ubuntu 14 & Ubuntu 16. I'm stuck in the 3rd step with a nice "ImportError: No module named django.core.management" while pip install says "Requirement already satisfied" This issue is not about having my problem solved, but about to have newbies like me able to run your software for tests without (too much) trouble.

My suggestion: 1) Before running the quick_install, add some command to check everything is ok.

python 3 vs python 2 (python -V)
propose work arounds: setup mysql if not present (apt-get install mysql) 2) please test the procedure on a updated Ubuntu 14 or Ubuntu 16
I had to modify requirements.txt to change version number in one compoment
install_dependancies installs things only with centos: check phantomjs, mariadb => the script need to be updated
the script install_dependancies works only with ubuntu or centos distribution. There is no "else" for other installation

Thanks you ...

vletoux commented 7 years ago

The symbolic link with libpcre.so.3 to the libpcre installed is one of the think which may be added in an install FAQ

vletoux commented 7 years ago

Are there yara rules / processes missing ? I made a basic test by changing the front page of a site with a nice "hacked by fsjll" and malspider didn't detect it. I check the repository for the "hacked" string and did not find anything

eleshar commented 7 years ago

@vletoux I've also been struggling/still am to get it working on Ubuntu 14.

What component's version did you change in requirements.txt?

The error I keep getting is:

Traceback (most recent call last):
  File "../malspider_django/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 345, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 185, in fetch_command
    settings.INSTALLED_APPS
  File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 55, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 43, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 120, in __init__
    raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.
Traceback (most recent call last):
  File "../malspider_django/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 345, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 195, in fetch_command
    klass = load_command_class(app_name, subcommand)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 39, in load_command_class
    module = import_module('%s.management.commands.%s' % (app_name, name))
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/migrate.py", line 16, in <module>
    from django.db.migrations.autodetector import MigrationAutodetector
  File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/autodetector.py", line 13, in <module>
    from django.db.migrations.questioner import MigrationQuestioner
  File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/questioner.py", line 12, in <module>
    from .loader import MigrationLoader
  File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/loader.py", line 10, in <module>
    from django.db.migrations.recorder import MigrationRecorder
  File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/recorder.py", line 12, in <module>
    class MigrationRecorder(object):
  File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/recorder.py", line 26, in MigrationRecorder
    class Migration(models.Model):
  File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/recorder.py", line 27, in Migration
    app = models.CharField(max_length=255)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 1072, in __init__
    super(CharField, self).__init__(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 166, in __init__
    self.db_tablespace = db_tablespace or settings.DEFAULT_INDEX_TABLESPACE
  File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 55, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 43, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 120, in __init__
    raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.
Failed to apply django migrations!

vletoux commented 7 years ago

The message "The SECRET_KEY setting must not be empty." means that the quick_install step number 3 didn't worked. I dismiss my ubuntu tentative and started from scratch on CentOS 7. I got it working now.

jasheppa5 commented 7 years ago

Hey guys,

Thank you for the feedback. Malspider was originally tested on minimal installs of CentOS 6, CentOS 7, and Ubuntu 14. The installation script worked flawlessly, but based on feedback i've received in the last several months our installation process is outdated. I'll test this week on Ubuntu 16.

The "SECRET_KEY" error..... SOme part of the quick_install didn't execute correctly. The installation process generates a secret key and updates the settings.py file with the generated key. If it doesn't work, you can generate one from the command line and paste in settings.py:

1) From the command line (in the root project directory) type "python malspider_django/manage.py gen_key" 2) Copy the generated key to the "SECRET_KEY" variable found in /malspider_django/malspider_django/settings.py 3) continue installation,etc

Are there yara rules / processes missing ? Malspider comes pre-built with a few sample yara rules, but you'll need to create a rule to detect defacement via the page title. To add a rule:

Navigate to “/yara”. You’ll see three folders (html, url, js) and three yara files that correspond to those folders (html_rules.yar, url_rules.yar, js_rules.yar). The folders contain individual rules and the corresponding yara files use include statements to organize the rules.

Rules in the html directory have access to the entire html page. Add a basic rule that looks for "hacked" in the page title.

I hope this helps!

James

On Sat, Feb 25, 2017 at 2:32 PM, vletoux notifications@github.com wrote:

The message "The SECRET_KEY setting must not be empty." means that the quick_install step number 3 didn't worked. I dismiss my ubuntu tentative and started from scratch on CentOS 7. I got it working now.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ciscocsirt/malspider/issues/22#issuecomment-282506719, or mute the thread https://github.com/notifications/unsubscribe-auth/AR0QEMOlk4u_lID7WxUp3hUYxcHl6anAks5rgIHcgaJpZM4ML_we .

eleshar commented 7 years ago

@jasheppa5 Thank you for the reply. I've since attempting Ubuntu downloaded CentOS 7 minimal, but still running into the same issue. I'm going to try CentOS 6 next.

vletoux commented 7 years ago

Thank you for getting back to me. I added a yara rule checking for the words "hacked by" as you suggested. I think this rule should be added by default. And my own rule improved based on field experience given the fact that a quick search on github showed a lot of defacement pages. I saw "h4k3d by" or "defaced by" or "defacement" which doesn't match the rule I made. Then, when triggered, I didn't see a category in the first column of the alerts (first column empty, unless the others)

vletoux commented 7 years ago

@eleshar CentOS 7 is working for me. Installing mysql was just a nightmare (adding rpm manually, finding temp password in the install log) I added some devel package before running the install.

ghost commented 7 years ago

I've tried to install this on ubuntu 14 (Server and Desktop) numerous times. I've had issues with installing yara (I think there's some confusion between yara and yara-python). Once yara installed properly, on ./quick_start yara was saying invalid regular expression "$". Putting quotes around the regex in malspider/yara/html/meta_refresh.yar seemed to fix that.

Now there are no errors on launch, however the crawlers spit out an error downloading the initial domain. "selenium.common.exceptions.TimeoutException: Message: Error Message => 'https://www.example.com' didn't load within the 'Page Load Timeout''

ciscocsirt / malspider

Please, update the install procedure #22