search

certsocietegenerale / FIR

Fast Incident Response
GNU General Public License v3.0
1.72k stars 507 forks source link

Errors on Ubuntu 22.04 LTS #313

Closed detvan closed 2 months ago

detvan commented 3 months ago

Hello,

i would like to install FIR in production on Ubuntu 22.04 LTS. I am following your guide https://github.com/certsocietegenerale/FIR/wiki/Installation-on-a-production-environment, but i get some errors

fir@fir:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:    22.04
Codename:   jammy
fir@fir:~$ sudo apt-get update
fir@fir:~$ sudo apt-get upgrade

Since Python2 packages are obsolete, I installed Python3 packages

fir@fir:~$ sudo apt-get install mysql-server libmysqlclient-dev gettext python3-dev python3-pip python3-lxml git libxml2-dev libxslt1-dev libz-dev nginx

fir@fir:~$ sudo pip3 install virtualenv
fir@fir:~$ virtualenv env-FIR
fir@fir:~$ source env-FIR/bin/activate
(env-FIR) fir@fir:~$ git clone https://github.com/certsocietegenerale/FIR.git
(env-FIR) fir@fir:~$ cd FIR
(env-FIR) fir@fir:~/FIR$ sudo pip3 install -r requirements.txt
Collecting dj-database-url
  Downloading dj_database_url-2.2.0-py3-none-any.whl (7.8 kB)
Collecting dj-email-url
  Downloading dj_email_url-1.0.6-py2.py3-none-any.whl (6.3 kB)
Collecting environs
  Downloading environs-11.0.0-py3-none-any.whl (12 kB)
Collecting future
  Downloading future-1.0.0-py3-none-any.whl (491 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 491.3/491.3 KB 10.4 MB/s eta 0:00:00
Collecting mysqlclient
  Downloading mysqlclient-2.2.4.tar.gz (90 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 90.4/90.4 KB 8.2 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [30 lines of output]
      /bin/sh: 1: pkg-config: not found
      /bin/sh: 1: pkg-config: not found
      /bin/sh: 1: pkg-config: not found
      Trying pkg-config --exists mysqlclient
      Command 'pkg-config --exists mysqlclient' returned non-zero exit status 127.
      Trying pkg-config --exists mariadb
      Command 'pkg-config --exists mariadb' returned non-zero exit status 127.
      Trying pkg-config --exists libmariadb
      Command 'pkg-config --exists libmariadb' returned non-zero exit status 127.
      Traceback (most recent call last):
        File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 162, in get_requires_for_build_wheel
          return self._get_build_requires(
        File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 143, in _get_build_requires
          self.run_setup()
        File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 158, in run_setup
          exec(compile(code, __file__, 'exec'), locals())
        File "setup.py", line 155, in <module>
          ext_options = get_config_posix(get_options())
        File "setup.py", line 49, in get_config_posix
          pkg_name = find_package_name()
        File "setup.py", line 28, in find_package_name
          raise Exception(
      Exception: Can not find valid pkg-config name.
      Specify MYSQLCLIENT_CFLAGS and MYSQLCLIENT_LDFLAGS env vars manually
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

It looks there are missing some packages, so i installed them with apt

(env-FIR) fir@fir:~/FIR$ sudo apt-get install pkg-config

After another try, it looks perfect, all requirements are successfully installed

(env-FIR) fir@fir:~/FIR$ sudo pip3 install -r requirements.txt

But following command finished with error message

(env-FIR) fir@fir:~/FIR$ sudo pip3 install mysql-python
Collecting mysql-python
  Downloading MySQL-python-1.2.5.zip (108 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 108.9/108.9 KB 3.1 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [8 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-okln5k_u/mysql-python_7c223a66d3b14c86a7349dc02e29d399/setup.py", line 13, in <module>
          from setup_posix import get_config
        File "/tmp/pip-install-okln5k_u/mysql-python_7c223a66d3b14c86a7349dc02e29d399/setup_posix.py", line 2, in <module>
          from ConfigParser import SafeConfigParser
      ModuleNotFoundError: No module named 'ConfigParser'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

It looks ConfigParser is missing too

(env-FIR) fir@fir:~/FIR$ sudo pip3 install ConfigParser

But still error messages persist

(env-FIR) fir@fir:~/FIR$ sudo pip3 install mysql-python
Collecting mysql-python
  Using cached MySQL-python-1.2.5.zip (108 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [8 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-ocqbicwv/mysql-python_82612445b7fd4cf39139aa612e7b85fe/setup.py", line 13, in <module>
          from setup_posix import get_config
        File "/tmp/pip-install-ocqbicwv/mysql-python_82612445b7fd4cf39139aa612e7b85fe/setup_posix.py", line 2, in <module>
          from ConfigParser import SafeConfigParser
      ModuleNotFoundError: No module named 'ConfigParser'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Please could you try to kindly help me to install FIR on production.

Thank toy very much in advance

Augustin-FL commented 3 months ago

Hello,

$ sudo pip install mysql-python should not be required anymore.

Could you please skip this command, continue the installation and tell me if it work?

detvan commented 3 months ago

thank you very much @Augustin-FL for your help and support

I continue with following command:

(env-FIR) fir@fir:~/FIR$ cp fir/config/production.py.sample fir/config/production.py
(env-FIR) fir@fir:~/FIR$ ./manage.py migrate --settings fir.config.production
Traceback (most recent call last):
  File "/home/fir/FIR/./manage.py", line 8, in <module>
    from django.core.management import execute_from_command_line
ModuleNotFoundError: No module named 'django'

So i installed django module

(env-FIR) fir@fir:~/FIR$ pip3 install django

Then i tried to create tables

(env-FIR) fir@fir:~/FIR$ ./manage.py migrate --settings fir.config.production
Traceback (most recent call last):
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/base.py", line 104, in wrapper
    saved_locale = translation.get_language()
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/utils/translation/__init__.py", line 211, in get_language
    return _trans.get_language()
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/utils/translation/__init__.py", line 66, in __getattr__
    if settings.USE_I18N:
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/conf/__init__.py", line 89, in __getattr__
    self._setup(name)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/conf/__init__.py", line 76, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/conf/__init__.py", line 190, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/fir/FIR/fir/config/production.py", line 3, in <module>
    from fir.config.base import *
  File "/home/fir/FIR/fir/config/base.py", line 8, in <module>
    import bleach
ModuleNotFoundError: No module named 'bleach'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/fir/FIR/./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/base.py", line 426, in run_from_argv
    connections.close_all()
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/utils/connection.py", line 84, in close_all
    for conn in self.all(initialized_only=True):
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/utils/connection.py", line 76, in all
    return [
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/utils/connection.py", line 73, in __iter__
    return iter(self.settings)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/utils/functional.py", line 47, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/utils/connection.py", line 45, in settings
    self._settings = self.configure_settings(self._settings)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/db/utils.py", line 148, in configure_settings
    databases = super().configure_settings(databases)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/utils/connection.py", line 50, in configure_settings
    settings = getattr(django_settings, self.settings_name)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/conf/__init__.py", line 89, in __getattr__
    self._setup(name)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/conf/__init__.py", line 76, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/conf/__init__.py", line 190, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/fir/FIR/fir/config/production.py", line 3, in <module>
    from fir.config.base import *
  File "/home/fir/FIR/fir/config/base.py", line 8, in <module>
    import bleach
ModuleNotFoundError: No module named 'bleach'

it looks some module is missing

(env-FIR) fir@fir:~/FIR$ pip3 install bleach

Following command showing error message about missing modules environs

(env-FIR) fir@fir:~/FIR$ ./manage.py migrate --settings fir.config.production

So i installed it

(env-FIR) fir@fir:~/FIR$ pip3 install environs
(env-FIR) fir@fir:~/FIR$ ./manage.py migrate --settings fir.config.production
Traceback (most recent call last):
  File "/home/fir/FIR/./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/__init__.py", line 416, in execute
    django.setup()
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/apps/registry.py", line 91, in populate
    app_config = AppConfig.create(entry)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/apps/config.py", line 193, in create
    import_module(entry)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'rest_framework'

But rest_framework, is not possible to install

(env-FIR) fir@fir:~/FIR$ pip3 install rest_framework
ERROR: Could not find a version that satisfies the requirement rest_framework (from versions: none)
ERROR: No matching distribution found for rest_framework
mrseeker commented 3 months ago

Quick 2 minute google search: pip3 install djangorestframework

detvan commented 3 months ago

Quick 2 minute google search: pip3 install djangorestframework

Thanks @mrseeker, it works. I had to install another missing packages.

(env-FIR) fir@fir:~/FIR$ pip3 install djangorestframework
(env-FIR) fir@fir:~/FIR$ pip3 install django-treebeard

But still getting error

(env-FIR) fir@fir:~/FIR$ ./manage.py migrate --settings fir.config.production
Traceback (most recent call last):
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/db/backends/mysql/base.py", line 16, in <module>
    import MySQLdb as Database
ModuleNotFoundError: No module named 'MySQLdb'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/fir/FIR/./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/__init__.py", line 416, in execute
    django.setup()
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/apps/registry.py", line 116, in populate
    app_config.import_models()
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/apps/config.py", line 269, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/contrib/auth/models.py", line 3, in <module>
    from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/contrib/auth/base_user.py", line 59, in <module>
    class AbstractBaseUser(models.Model):
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/db/models/base.py", line 143, in __new__
    new_class.add_to_class("_meta", Options(meta, app_label))
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/db/models/base.py", line 371, in add_to_class
    value.contribute_to_class(cls, name)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/db/models/options.py", line 243, in contribute_to_class
    self.db_table, connection.ops.max_name_length()
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/utils/connection.py", line 15, in __getattr__
    return getattr(self._connections[self._alias], item)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/utils/connection.py", line 62, in __getitem__
    conn = self.create_connection(alias)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/db/utils.py", line 193, in create_connection
    backend = load_backend(db["ENGINE"])
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/db/utils.py", line 113, in load_backend
    return import_module("%s.base" % backend_name)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/db/backends/mysql/base.py", line 18, in <module>
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module.
Did you install mysqlclient?

I tried install mysqlclient again however according to previous conversation it is not needed. but today it have a luck :), mysqlclinet is installed.

(env-FIR) fir@fir:~/FIR$ pip3 install mysqlclient
Collecting mysqlclient
  Downloading mysqlclient-2.2.4.tar.gz (90 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 90.4/90.4 kB 3.0 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: mysqlclient
  Building wheel for mysqlclient (pyproject.toml) ... done
  Created wheel for mysqlclient: filename=mysqlclient-2.2.4-cp310-cp310-linux_x86_64.whl size=124739 sha256=1769abd3e6d59a5f2842a93a8096602ca779d4fc0923bf238a130478d0d3435c
  Stored in directory: /home/fir/.cache/pip/wheels/ac/96/ac/2a4d8cb58a4d95de1dffc3f8b0ea42e0e5b63ab97640edbda3
Successfully built mysqlclient
Installing collected packages: mysqlclient
Successfully installed mysqlclient-2.2.4

Another modules are needed

(env-FIR) fir@fir:~/FIR$ pip3 install django-querysetsequence
(env-FIR) fir@fir:~/FIR$ pip3 install markdown2
(env-FIR) fir@fir:~/FIR$ pip3 install pymongo

Edit config file

(env-FIR) fir@fir:~/FIR$ vim fir/config/production.py

Creating superuser account

(env-FIR) fir@fir:~/FIR$ ./manage.py migrate --settings fir.config.production

Username (leave blank to use 'fir'): 
Email address: fir@test.com
Password: 
Password (again): 
Superuser created successfully.

When i want to Import initial data - process failed.

$ ./manage.py loaddata ncidents/fixtures/01_seed_data.json --settings fir.config.production
Traceback (most recent call last):
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/apps/registry.py", line 158, in get_app_config
    return self.app_configs[app_label]
KeyError: 'fir_notifications'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/serializers/python.py", line 191, in _get_model
    return apps.get_model(model_identifier)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/apps/registry.py", line 208, in get_model
    app_config = self.get_app_config(app_label)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/apps/registry.py", line 165, in get_app_config
    raise LookupError(message)
LookupError: No installed app with label 'fir_notifications'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/fir/FIR/./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/commands/loaddata.py", line 102, in handle
    self.loaddata(fixture_labels)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/commands/loaddata.py", line 163, in loaddata
    self.load_label(fixture_label)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/commands/loaddata.py", line 251, in load_label
    for obj in objects:
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/serializers/json.py", line 70, in Deserializer
    yield from PythonDeserializer(objects, **options)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/serializers/python.py", line 114, in Deserializer
    Model = _get_model(d["model"])
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/serializers/python.py", line 193, in _get_model
    raise base.DeserializationError(
django.core.serializers.base.DeserializationError: Problem installing fixture '/home/fir/FIR/incidents/fixtures/01_seed_data.json': Invalid model identifier: 'fir_notifications.notificationpreference'
(env-FIR) fir@fir:~/FIR$ ./manage.py loaddata incidents/fixtures/02_dev_users.json --settings fir.config.production
Traceback (most recent call last):
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/serializers/base.py", line 354, in deserialize_m2m_values
    values.append(m2m_convert(pk))
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/serializers/base.py", line 335, in m2m_convert
    model._default_manager.db_manager(using)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/contrib/auth/models.py", line 95, in get_by_natural_key
    return self.get(name=name)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/db/models/query.py", line 649, in get
    raise self.model.DoesNotExist(
django.contrib.auth.models.Group.DoesNotExist: Group matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/serializers/base.py", line 281, in save_deferred_fields
    values = deserialize_m2m_values(
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/serializers/base.py", line 360, in deserialize_m2m_values
    raise M2MDeserializationError(e, pk)
django.core.serializers.base.M2MDeserializationError: (DoesNotExist('Group matching query does not exist.'), ['Incident handlers'])

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/fir/FIR/./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/commands/loaddata.py", line 102, in handle
    self.loaddata(fixture_labels)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/management/commands/loaddata.py", line 165, in loaddata
    obj.save_deferred_fields(using=self.using)
  File "/home/fir/env-FIR/lib/python3.10/site-packages/django/core/serializers/base.py", line 285, in save_deferred_fields
    raise DeserializationError.WithData(
django.core.serializers.base.DeserializationError: Group matching query does not exist.: (auth.user:pk=2) field_value was '['Incident handlers']'
detvan commented 2 months ago

Hello all,

i fixed it. I would like to share step by step guide. It works on Ubuntu 22.04.

root@fir:~# usermod -a -G sudo fir

root@fir:~# exit

fir@fir:~$ sudo apt update
fir@fir:~$ sudo apt upgrade

fir@fir:~$ sudo apt install mysql-server libmysqlclient-dev gettext python3-dev python3-pip python3-lxml git libxml2-dev libxslt1-dev libz-dev nginx
fir@fir:~$ sudo apt install vim
fir@fir:~$ sudo apt install pkg-config
fir@fir:~$ sudo apt install redis-server

fir@fir:~$ sudo pip3 install virtualenv

fir@fir:~$ virtualenv env-FIR
fir@fir:~$ source env-FIR/bin/activate

(env-FIR) fir@fir:~$ sudo mkdir /opt/fir
(env-FIR) fir@fir:~$ sudo chown fir:fir /opt/fir
(env-FIR) fir@fir:~$ cd /opt/fir/

(env-FIR) fir@fir:/opt/fir$ git clone https://github.com/certsocietegenerale/FIR.git
(env-FIR) fir@fir:/opt/fir$ cd FIR

(env-FIR) fir@fir:/opt/fir/FIR$ pip3 install --upgrade pip
(env-FIR) fir@fir:/opt/fir/FIR$ pip3 install -r requirements.txt

(env-FIR) fir@fir:/opt/fir/FIR$ cp fir/config/production.py.sample fir/config/production.py
(env-FIR) fir@fir:/opt/fir/FIR$ cp fir/config/installed_apps.txt.sample fir/config/installed_apps.txt

(env-FIR) fir@fir:/opt/fir/FIR$ pip3 install xmpppy
(env-FIR) fir@fir:/opt/fir/FIR$ pip3 install celery
(env-FIR) fir@fir:/opt/fir/FIR$ pip3 install redis
(env-FIR) fir@fir:/opt/fir/FIR$ pip3 install abuse_finder
(env-FIR) fir@fir:/opt/fir/FIR$ pip3 install uwsgi

(env-FIR) fir@fir:/opt/fir/FIR$ sudo mysql
mysql> CREATE DATABASE fir;
mysql> CREATE USER 'fir'@'localhost' IDENTIFIED BY 'fir';
mysql> GRANT USAGE ON *.* TO 'fir'@'localhost';
mysql> GRANT ALL PRIVILEGES ON `fir`.* TO 'fir'@'localhost';
mysql> exit;

(env-FIR) fir@fir:/opt/fir/FIR$ vim fir/config/production.py
ALLOWED_HOSTS = ['FIR.DOMAIN.COM','192.168.1.1','127.0.0.1']

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'fir',
        'USER': 'fir',
        'PASSWORD': 'fir',
        'HOST': '127.0.0.1',
        'PORT': '3306',
    }
}

(env-FIR) fir@fir:/opt/fir/FIR$ vim fir_celery/celeryconf.py
from django.conf import settings
settings.REDIS_HOST = '127.0.0.1'
settings.REDIS_PORT = int(6379)
settings.REDIS_DB = int(0)

(env-FIR) fir@fir:/opt/fir/FIR$ vim fir/settings.py

(env-FIR) fir@fir:/opt/fir/FIR$ ./manage.py migrate --settings fir.config.production

(env-FIR) fir@fir:/opt/fir/FIR$ ./manage.py createsuperuser --settings fir.config.production

(env-FIR) fir@fir:/opt/fir/FIR$ ./manage.py loaddata incidents/fixtures/01_seed_data.json --settings fir.config.production

(env-FIR) fir@fir:/opt/fir/FIR$ ./manage.py collectstatic --settings fir.config.production

(env-FIR) fir@fir:/opt/fir/FIR$ sudo chsh www-data
Changing the login shell for www-data
Enter the new value, or press ENTER for the default
    Login Shell [/usr/sbin/nologin]: /bin/sh

(env-FIR) fir@fir:/opt/fir/FIR$ mkdir run
(env-FIR) fir@fir:/opt/fir/FIR$ sudo chown www-data run

(env-FIR) fir@fir:/opt/fir/FIR$ sudo vim /etc/systemd/system/fir_uwsgi.service
[Unit]
Description=uWSGI instance for FIR
#After=syslog.target

[Service]
User=www-data
Group=www-data
WorkingDirectory=/opt/fir/FIR/
ExecStart=/home/fir/env-FIR/bin/uwsgi --socket /opt/fir/FIR/run/fir.sock --chdir /opt/fir/FIR/ --module fir.wsgi
#Restart=always
KillSignal=SIGQUIT
Type=Debug
NotifyAccess=All

[Install]
WantedBy=multi-user.target

(env-FIR) fir@fir:/opt/fir/FIR$ sudo chmod -R 755 /home/fir

(env-FIR) fir@fir:/opt/fir/FIR$ sudo systemctl daemon-reload
(env-FIR) fir@fir:/opt/fir/FIR$ sudo systemctl start fir_uwsgi
(env-FIR) fir@fir:/opt/fir/FIR$ sudo systemctl restart fir_uwsgi
(env-FIR) fir@fir:/opt/fir/FIR$ sudo systemctl status fir_uwsgi

(env-FIR) fir@fir:/opt/fir/FIR$ wget https://raw.githubusercontent.com/nginx/nginx/master/conf/uwsgi_params -P run

(env-FIR) fir@fir:/opt/fir/FIR$ ls -la run
total 12
drwxrwxr-x  2 www-data fir      4096 Jul 22 20:14 .
drwxrwxr-x 23 fir      fir      4096 Jul 22 19:47 ..
srwxr-xr-x  1 www-data www-data    0 Jul 22 20:12 fir.sock
-rw-rw-r--  1 fir      fir       664 Jul 22 20:14 uwsgi_params

(env-FIR) fir@fir:/opt/fir/FIR$ sudo mv /etc/nginx/sites-enabled/default /home/fir/

(env-FIR) fir@fir:/opt/fir/FIR$ sudo vim /etc/nginx/sites-available/fir
upstream fir {
        server unix:///opt/fir/FIR/run/fir.sock;
}

server {
        server_name FIR.DOMAIN.COM
                    192.168.1.1
                    ;

        location / {
                uwsgi_pass fir;
                include /opt/fir/FIR/run/uwsgi_params;
        }

        location /static/ {
                alias /opt/fir/FIR/static/;
        }
}

(env-FIR) fir@fir:/opt/fir/FIR$ sudo ln -s /etc/nginx/sites-available/fir /etc/nginx/sites-enabled/fir

(env-FIR) fir@fir:/opt/fir/FIR$ sudo systemctl restart nginx
Augustin-FL commented 2 months ago

hello @detvan,

Sorry for the delay, and thanks for the guide!

As you rightfully pointed out, one of the issues seems to be requirements.txt which does not include dependencies for fir_notifications. That causes problems when importing fixtures.

I will submit a PR for that. I believe the right way would be to create a dedicated fixture file for notifications

Augustin-FL commented 2 months ago

I will close this issue, as you was able to install FIR. Feel free to reopen it if needed