Cloning a database from one instance to another via backup export

[CyberPoly1000]

Problem statement I have a testing VM on my local machine on which I installed CISO assistant. I started to put my organization details (like the assets, threats, one risk assessment, ..) in this testing VM. In the meantime I launched a prod VM in my IT environment with CISO assistant in it. I would like to take the config I have on my testing VM via the backup export (to have all my assets, threats, risk assessment, ...) and put it on my production VM. In other words, taking the data from one instance to another, totally separated from one another. I tried with the current export/import capabilities, but this does not work.

Is it considered a bug, or is it normal that this does not work ? I mark this issue as feature request not knowing this.

Expected behavior

1. Export database on instance 1
2. Import the JSON exported in step 1 in my CISO Assistant on my instance 2
3. Retrieve the same state in instance 2 than in instance 1

[nas-tabchiche]

Hi @CyberPoly1000,

Thanks for reaching out. To help you effectively, I need a bit more information:

1. Version of CISO Assistant: Could you confirm the version numbers of CISO Assistant on both your testing and production VMs? This helps in understanding if there are any version-specific limitations or bugs.
2. Details of the Issue: When you attempt to import the exported JSON file into your production VM, what exactly happens? Are there any error messages, or does the data simply not appear as expected? Detailed error messages or descriptions of the behavior can help identify the root cause.
3. Logs: If possible, please check the application logs for any warnings or errors during the export or import process and share them. Logs often provide insights that are crucial for troubleshooting.

Best regards,

Nassim

[MathRig]

Hi @nas-tabchiche

I have the same issue. I did a backup from a v1.1.0 version instance and tried to restore it in a V1.1.1 instance and there is no reaction in the GUI. In the backend container logs I have this :

2024-04-29T15:37:18.688412Z [info ] request_started [django_structlog.middlewares.request] ciso_assistant_url=https://localhost:8443 ip=172.18.0.4 request=GET /api/iam/current-user/ request_id=337d28dc-7c6c-4599-b1d1-f0fe40951572 user_agent=node user_id=None 2024-04-29T15:37:18.818650Z [info ] request_finished [django_structlog.middlewares.request] ciso_assistant_url=https://localhost:8443 code=200 ip=172.18.0.4 request=GET /api/iam/current-user/ request_id=337d28dc-7c6c-4599-b1d1-f0fe40951572 user_id=e927cfcc-24af-47a9-83e1-9a472933a8e8 2024-04-29T15:37:18.825006Z [info ] request_started [django_structlog.middlewares.request] ciso_assistant_url=https://localhost:8443 ip=172.18.0.4 request=POST /api/serdes/load-backup/ request_id=fddd044c-637c-49ca-962b-c0c35100d0de user_agent=node user_id=None 2024-04-29T15:37:18.825505Z [info ] request_finished [django_structlog.middlewares.request] ciso_assistant_url=https://localhost:8443 code=302 ip=172.18.0.4 request=POST /api/serdes/load-backup/ request_id=fddd044c-637c-49ca-962b-c0c35100d0de user_id=None 2024-04-29T15:37:18.826810Z [info ] request_started [django_structlog.middlewares.request] ciso_assistant_url=https://localhost:8443 ip=172.18.0.4 request=GET /accounts/login/?next=/api/serdes/load-backup/ request_id=10a620ec-0f59-40f8-9dc8-662927da8798 user_agent=node user_id=None 2024-04-29T15:37:18.828244Z [warning ] request_finished [django_structlog.middlewares.request] ciso_assistant_url=https://localhost:8443 code=404 ip=172.18.0.4 request=GET /accounts/login/?next=/api/serdes/load-backup/ request_id=10a620ec-0f59-40f8-9dc8-662927da8798 user_id=None Not Found: /accounts/login/ Not Found: /accounts/login/
2024-04-29T15:37:18.834729Z [info ] request_started [django_structlog.middlewares.request] ciso_assistant_url=https://localhost:8443 ip=172.18.0.4 request=GET /api/iam/current-user/ request_id=0ce4e764-6062-4967-81e3-3c39f2945c4b user_agent=node user_id=None 2024-04-29T15:37:18.933753Z [info ] request_finished [django_structlog.middlewares.request] ciso_assistant_url=https://localhost:8443 code=200 ip=172.18.0.4 request=GET /api/iam/current-user/ request_id=0ce4e764-6062-4967-81e3-3c39f2945c4b user_id=e927cfcc-24af-47a9-83e1-9a472933a8e8

I hope this helps. Best regards, Mathieu.

[ab-smith]

I can reproduce the export regression as well on 1.1.0

[eric-intuitem]

There is an issue with iam objects. We will fix that quickly.

[eric-intuitem]

Backup has an error with IAM object. A temporary workaround is to replace "is_published": null -> "is_published": false.

[ab-smith]

@CyberPoly1000 with the recent fix, it should be good now, make sure that you're using the latest version and try again.

side note: if you're using an old version, the export could require some extra fixes to accommodate the missing migrations, but unlikely. Reach out on Discord and we can take a look together

[MathRig]

Hello @eric-intuitem @ab-smith ,

Thank you for the feedback. Does that mean we need to change all "is_published": null to is_published": false in the backup json file or just for the iam.xxx objects ? On my side all iam.xxx objects have the is_published at false.

[eric-intuitem]

@MathRig v1.1.0 has a problem with the backup feature, are you sure your backup was done with this version?

[eric-intuitem]

More generally the backup/restore feature is not designed to go through migrations. You should restore on an instance with the same version, and upgrade your version after. This will ensure migrations are done properly. In the case of the trial or hosted instance, our support team can help you make the operation.

[MathRig]

Ok, Thanks, I will reinstall the previous version and restore from the file and see if there is an issue or not.

[MathRig]

Ok, I tried to restore with the V1.1.0 and v1.0.9 as I was almost sure that I was in v1.1.0 but the backup file has this at the first line : [{"meta": [{"media_version": "v1.0.9"}]}, Nevertheless, with both version, it's the same behavior and the logs of the backend are the same as the one provided.

[CyberPoly1000]

Hi guys, thanks for all your answers. I migrated both my test and prod env to the last version of CISO Assistant 1.2.2, build number 4e9024b. From my test env, I exported my DB containing all my useful data. I exported it in my prod env. I got disconnected after import but when trying to reconnect, it states "Unable to log in with provided credentials." (even though they are the right ones).

I looked at the logs of the backend and it returned this :

2024-05-02T13:17:18.640457Z [error    ] request_failed                 [django_structlog.middlewares.request] ciso_assistant_url=https://<my_prod>:8443 code=500 ip=<my_ip> request=POST /api/serdes/load-backup/ request_id=d4b7d160-e518-462f-8518-65316363***f user_id=2d88404f-edcc-4fac-bb15-7c0b631c0***
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/db/models/options.py", line 681, in get_field
    return self.fields_map[field_name]
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'level'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/core/serializers/json.py", line 70, in Deserializer
    yield from PythonDeserializer(objects, **options)
  File "/usr/local/lib/python3.11/site-packages/django/core/serializers/python.py", line 141, in Deserializer
    field = Model._meta.get_field(field_name)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/options.py", line 683, in get_field
    raise FieldDoesNotExist(
django.core.exceptions.FieldDoesNotExist: RequirementNode has no field named 'level'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/decorators/csrf.py", line 65, in _view_wrapper
    return view_func(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 104, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/serdes/views.py", line 54, in post
    management.call_command(
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 194, in call_command
    return command.execute(*args, **defaults)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/loaddata.py", line 102, in handle
    self.loaddata(fixture_labels)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/loaddata.py", line 163, in loaddata
    self.load_label(fixture_label)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/loaddata.py", line 251, in load_label
    for obj in objects:
  File "/usr/local/lib/python3.11/site-packages/django/core/serializers/json.py", line 74, in Deserializer
    raise DeserializationError() from exc
django.core.serializers.base.DeserializationError: Problem installing fixture '-':
Internal Server Error: /api/serdes/load-backup/
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/db/models/options.py", line 681, in get_field
    return self.fields_map[field_name]
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'level'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/core/serializers/json.py", line 70, in Deserializer
    yield from PythonDeserializer(objects, **options)
  File "/usr/local/lib/python3.11/site-packages/django/core/serializers/python.py", line 141, in Deserializer
    field = Model._meta.get_field(field_name)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/options.py", line 683, in get_field
    raise FieldDoesNotExist(
django.core.exceptions.FieldDoesNotExist: RequirementNode has no field named 'level'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/decorators/csrf.py", line 65, in _view_wrapper
    return view_func(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 104, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/serdes/views.py", line 54, in post
    management.call_command(
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 194, in call_command
    return command.execute(*args, **defaults)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/loaddata.py", line 102, in handle
    self.loaddata(fixture_labels)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/loaddata.py", line 163, in loaddata
    self.load_label(fixture_label)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/loaddata.py", line 251, in load_label
    for obj in objects:
  File "/usr/local/lib/python3.11/site-packages/django/core/serializers/json.py", line 74, in Deserializer
    raise DeserializationError() from exc
django.core.serializers.base.DeserializationError: Problem installing fixture '-':
Internal Server Error: /api/serdes/load-backup/
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/db/models/options.py", line 681, in get_field
    return self.fields_map[field_name]
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'level'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/core/serializers/json.py", line 70, in Deserializer
    yield from PythonDeserializer(objects, **options)
  File "/usr/local/lib/python3.11/site-packages/django/core/serializers/python.py", line 141, in Deserializer
    field = Model._meta.get_field(field_name)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/options.py", line 683, in get_field
    raise FieldDoesNotExist(
django.core.exceptions.FieldDoesNotExist: RequirementNode has no field named 'level'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/decorators/csrf.py", line 65, in _view_wrapper
    return view_func(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 104, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/serdes/views.py", line 54, in post
    management.call_command(
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 194, in call_command
    return command.execute(*args, **defaults)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/loaddata.py", line 102, in handle
    self.loaddata(fixture_labels)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/loaddata.py", line 163, in loaddata
    self.load_label(fixture_label)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/loaddata.py", line 251, in load_label
    for obj in objects:
  File "/usr/local/lib/python3.11/site-packages/django/core/serializers/json.py", line 74, in Deserializer
    raise DeserializationError() from exc
django.core.serializers.base.DeserializationError: Problem installing fixture '-':
2024-05-02T13:17:19.246645Z [info     ] request_started                [django_structlog.middlewares.request] ciso_assistant_url=https://<my_prod>:8443 ip=<my_ip> request=GET /api/iam/current-user/ request_id=8a51ebf7-ef0e-477f-ab75-d1e84728e*** user_agent=node user_id=None
2024-05-02T13:17:19.251967Z [warning  ] request_finished               [django_structlog.middlewares.request] ciso_assistant_url=https://<my_prod>:8443 code=401 ip=<my_ip> request=GET /api/iam/current-user/ request_id=8a51ebf7-ef0e-477f-ab75-d1e84728e*** user_id=None
Unauthorized: /api/iam/current-user/
Unauthorized: /api/iam/current-user/
2024-05-02T13:17:30.166677Z [info     ] request_started                [django_structlog.middlewares.request] ciso_assistant_url=https://<my_prod>:8443 ip=<my_ip> request=POST /api/iam/login/ request_id=423be8d1-0802-47f3-a0b6-858fca973*** user_agent=node user_id=None
2024-05-02T13:17:30.373534Z [warning  ] request_finished               [django_structlog.middlewares.request] ciso_assistant_url=https://<my_prod>:8443 code=400 ip=<my_ip> request=POST /api/iam/login/ request_id=423be8d1-0802-47f3-a0b6-858fca973*** user_id=None
Bad Request: /api/iam/login/

I guess there was a problem during backup restore. Could you help me out with this tracelog ?

[ab-smith]

Looks like a migration failed somewhere to get this fixture. Let's sync up with the team through Discord to see how we can debug this live together

[ab-smith]

and to be sure @CyberPoly1000, I'm assuming test is source and prod is target, are you test and prod empty and you're trying to experiment the export/import or you have data in test that you're trying to preserve?

[ab-smith]

@MathRig all good now? @CyberPoly1000 do you mind a short debug session with @eric-intuitem through Discord?

thank you both!

[MathRig]

@ab-smith yes all good now thanks to @eric-intuitem . Just for the others, I had to follow this process :

• Git checkout v1.0.9 locally
• modify the docker-compose.yml to change the image version from "latest" to "v1.0.9"
• use the docker-compose-build.sh to build it and deploy.

Then after that I was in v1.0.9 and the import worked.

[CyberPoly1000]

Hi guys, Following the debug session with @eric-intuitem, turns out it works. It was probably a compatibility issue from the JSON I provided. When performing the task from A to Z, with the same versions on both sides, the export is completed succesfully.

Thanks a lot for your help, I close the issue.

[ab-smith]

Fixed in version 1.2.1