Closed edouardsaucisse closed 1 week ago
ab-smith
commented
3 weeks ago Hello, thank you for the feedback. Just to get it right, your setup is windows host with an ubuntu VM? which emulator or is it WSL?
and you have attempted the docker-compose.sh script from within the VM or Windows command line?
We need indeed to improve the instructions for Windows-based installation.
edouardsaucisse
commented
3 weeks ago Sorry, I realize my context was not clear :/ I use a dedicated server as a lab before production. It runs a proxmox hypervisor. The hypervisor runs an ubuntu 22.04 VM, which embeds some docker virtual environments. Not the most efficient architecture, i know, but i feel it's close enough to production conditions and it allows me to tests stuff as I want.
The windows host is my desktop PC, which I use to manage proxmox (webUI) and connect to my virtual machines (inc. ubuntu 22.04 "docker" machine).
I hope it gets easier to understand and contextualize this way, but feel free to ask any precisions you want.
edit : I tested the docker-compose.sh script, but it didn't work too (same results, but i can get a copy/paste if required).
Mohamed-Hacene
commented
3 weeks ago Hi @edouardsaucisse, thanks for your contribution, it helps us a lot to better understand the different problems associated with different configurations.
I'll help you resolve these issues step by step.
Firstly, I strongly advise you to update your docker version if possible. docker-compose is deprecated for a year now (see Migrate to Compose V2), although I don't think it's causing all these problems either, it's better to be up to date so that we can be sure it's not docker-related.
I tried yesterday to install CISO Assistant on an Ubuntu server with the same version as you, with a remote access from my host machine but I didn't have the errors you encountered. So, let's check that everything is fine with your configuration before going any further with the code.
docker:8443 while the application is launched on localhost:8443. You need to modify the file docker-compose.yml to replace localhost occurrences by docker and add an argument in caddy service. I refer you to this tutorial: https://github.com/intuitem/ciso-assistant-community/issues/226#issuecomment-2051397358 which explains how to modify docker-compose.yml in your use case.edouardsaucisse
commented
3 weeks ago Thanks for your answer !
I'll try these step on my current ubuntu VM. If it fails, i'll try it out on a fresh and dedicated VM, to get rid of any environment related issue.
Just for information : if I host my CISOassistant instance on a remote server, addressing it by localhost:8443 will be pointless ^^ That's why I address it by docker:8443 : my ubuntu server is aliased "docker" in my DNS server (so I can reach it from anywhere on my network by simply calling http(s)://docker:whateverportorserviceIwant)
Mohamed-Hacene
commented
3 weeks ago Perfect !
Yes I understand, I know it comes from your DNS and it's the right thing to do.
Without the FQDN, the SNI is not transmitted during the connection and caddy does not accept https without the SNI which causes an SSL error. What I meant was that yours is the other way around, where you've set the DNS correctly but the FQDN information you're using is missing from the docker-compose.yml, which also causes an SSL error.
Sorry if my comment sounded presumptuous, that wasn't the point at all :)
ab-smith
commented
2 weeks ago @edouardsaucisse all good?
edouardsaucisse
commented
2 weeks ago Sorry for the delay. Just tested a few minutes ago : still doesn't work.
I installed a fresh ubuntu 22.04 server environnement, with docker.io and docker compose (i know this not the recommended version, but as i changed the docker compose commands to docker-compose, i guess it has a very limited impact). I cloned the repo, modified the docker-compose.sh and docker-compose.yml, ran it and installed CA smoothly.
Installation is fine, it asks for email and password, and all the setup seems ok.
But when i try to join the webpage, still the same issue (http send to https listener).
Don't understand why.
root@dockertemp:~/ciso-assistant-community# cat docker-compose.sh
#! /usr/bin/env bash
if [ -f db/ciso-assistant.sqlite3 ] ; then
echo "the database seems already created"
echo "you should launch docker compose up -d"
echo "for clean start, you can remove the database file, run docker compose down and then docker compose rm and start again"
else
docker rmi ghcr.io/intuitem/ciso-assistant-community/backend:latest ghcr.io/intuitem/ciso-assistant-community/frontend:latest 2> /dev/null
docker-compose up -d
docker-compose exec backend python manage.py migrate
echo "initialize your superuser account..."
docker-compose exec backend python manage.py createsuperuser
echo "connect to ciso assistant on https://localhost:8443"
echo "for successive runs you can now use docker compose up"
firoot@dockertemp:~/ciso-assistant-community# cat docker-compose.yml
version: "3.9"
services:
backend:
container_name: backend
image: ghcr.io/intuitem/ciso-assistant-community/backend:latest
restart: always
environment:
- ALLOWED_HOSTS=backend
- CISO_ASSISTANT_URL=https://dockertemp:8443
- DJANGO_DEBUG=True
volumes:
- ./db:/code/db
frontend:
container_name: frontend
environment:
- PUBLIC_BACKEND_API_URL=http://backend:8000/api
- PROTOCOL_HEADER=x-forwarded-proto
- HOST_HEADER=x-forwarded-host
image: ghcr.io/intuitem/ciso-assistant-community/frontend:latest
depends_on:
- backend
caddy:
container_name: caddy
image: caddy:2.7.6
restart: unless-stopped
ports:
- 8443:8443
command:
- caddy
- reverse-proxy
- --from
- https://dockertemp:8443
- --to
- frontend:3000
volumes:
- ./db:/datai'm gonna try it on an up-to-date docker installation (v25) but more for science than to use it in production. my distros are stable and standardized one, i won't introduced unsupported packages in production just to test and/or support CA.
edouardsaucisse
commented
2 weeks ago Installed ubuntu-server 24.04 installed docker v25 with the procedure described here https://docs.docker.com/engine/install/ubuntu/
I can't get a fresher and cleaner environment than that.
I ran
git clone https://github.com/intuitem/ciso-assistant-community.git
cd ciso-assistant-community/
./docker-compose.shI got :
WARN[0000] /root/ciso-assistant-community/docker-compose.yml: `version` is obsolete
[+] Running 29/29
✔ caddy Pulled 14.4s
✔ 619be1103602 Pull complete 10.3s
✔ e88463a31d6d Pull complete 10.5s
✔ de93e10dd670 Pull complete 10.6s
✔ c9ae96b49a2d Pull complete 12.4s
✔ backend Pulled 69.7s
✔ 1468e7ff95fc Pull complete 14.6s
✔ 2cf9c2b42f41 Pull complete 18.0s
✔ c4c40c3e3cdf Pull complete 23.8s
✔ c05cc1123d7e Pull complete 33.7s
✔ b6f29ccdcc55 Pull complete 34.3s
✔ 0cc3b3583e25 Pull complete 37.8s
✔ 8db593f9de8d Pull complete 38.5s
✔ 743c3596a80e Pull complete 39.5s
✔ 0db4cd6a9320 Pull complete 40.0s
✔ 3b4569b59a91 Pull complete 46.1s
✔ 96b7c0ed6c92 Pull complete 47.1s
✔ 4f4fb700ef54 Pull complete 47.8s
✔ 27ed3d675a7f Pull complete 67.8s
✔ frontend Pulled 22.1s
✔ 4abcf2066143 Pull complete 9.2s
✔ 62af34686b6b Pull complete 15.7s
✔ d3f3e43082a8 Pull complete 15.8s
✔ af0a15da4cb8 Pull complete 15.8s
✔ 8c7e1915ff31 Pull complete 15.9s
✔ b6054f745481 Pull complete 16.2s
✔ 442b3ed9a931 Pull complete 16.2s
✔ 496859abd58e Pull complete 21.0s
✔ ecdaabefbbb9 Pull complete 21.0s
[+] Running 4/4
✔ Network ciso-assistant-community_default Created 0.8s
✔ Container backend Started 5.1s
✔ Container caddy Started 5.2s
✔ Container frontend Started 2.9s
WARN[0000] /root/ciso-assistant-community/docker-compose.yml: `version` is obsolete
2024-05-02T16:38:18.410901Z [info ] BASE_DIR: /code [ciso_assistant.settings] ciso_assistant_url=https://localhost:8443
2024-05-02T16:38:18.412816Z [info ] VERSION: v1.2.2 [ciso_assistant.settings] ciso_assistant_url=https://localhost:8443
2024-05-02T16:38:18.413540Z [info ] BUILD: 4e9024b [ciso_assistant.settings] ciso_assistant_url=https://localhost:8443
2024-05-02T16:38:18.415255Z [info ] DEBUG mode: True [ciso_assistant.settings] ciso_assistant_url=https://localhost:8443
2024-05-02T16:38:18.415870Z [info ] CISO_ASSISTANT_URL: https://localhost:8443 [ciso_assistant.settings] ciso_assistant_url=https://localhost:8443
2024-05-02T16:38:18.416817Z [info ] ALLOWED_HOSTS: ['backend'] [ciso_assistant.settings] ciso_assistant_url=https://localhost:8443
2024-05-02T16:38:18.417775Z [info ] SQLITE_FILE: /code/db/ciso-assistant.sqlite3 [ciso_assistant.settings] ciso_assistant_url=https://localhost:8443
2024-05-02T16:38:18.418675Z [info ] DATABASE ENGINE: django.db.backends.sqlite3 [ciso_assistant.settings] ciso_assistant_url=https://localhost:8443
Operations to perform:
Apply all migrations: auth, cal, contenttypes, core, iam, knox, sessions
Running migrations:
Applying contenttypes.0001_initial... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0001_initial... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying auth.0012_alter_user_first_name_max_length... OK
Applying cal.0001_initial... OK
Applying iam.0001_initial... OK
Applying core.0001_initial... OK
Applying core.0002_initial... OK
Applying core.0003_alter_riskscenario_strength_of_knowledge... OK
Applying core.0004_complianceassessment_is_published_and_more... OK
Applying core.0005_alter_project_lc_status_alter_securitymeasure_effort... OK
Applying core.0006_remove_securitymeasure_security_function_and_more... OK
Applying core.0007_alter_requirementlevel_framework_and_more... OK
Applying core.0008_alter_complianceassessment_status_and_more... OK
Applying core.0009_framework_max_score_framework_min_score_and_more... OK
Applying core.0010_rename_score_definition_framework_scores_definition_and_more... OK
Applying core.0011_auto_20240501_1342... OK
Applying iam.0002_purge_validator... OK
Applying knox.0001_initial... OK
Applying knox.0002_auto_20150916_1425... OK
Applying knox.0003_auto_20150916_1526... OK
Applying knox.0004_authtoken_expires... OK
Applying knox.0005_authtoken_token_key... OK
Applying knox.0006_auto_20160818_0932... OK
Applying knox.0007_auto_20190111_0542... OK
Applying knox.0008_remove_authtoken_salt... OK
Applying sessions.0001_initial... OK
startup handler: initialize database
initialize your superuser account...
WARN[0000] /root/ciso-assistant-community/docker-compose.yml: `version` is obsolete
2024-05-02T16:38:29.009341Z [info ] BASE_DIR: /code [ciso_assistant.settings] ciso_assistant_url=https://localhost:8443
2024-05-02T16:38:29.009887Z [info ] VERSION: v1.2.2 [ciso_assistant.settings] ciso_assistant_url=https://localhost:8443
2024-05-02T16:38:29.010312Z [info ] BUILD: 4e9024b [ciso_assistant.settings] ciso_assistant_url=https://localhost:8443
2024-05-02T16:38:29.011291Z [info ] DEBUG mode: True [ciso_assistant.settings] ciso_assistant_url=https://localhost:8443
2024-05-02T16:38:29.011488Z [info ] CISO_ASSISTANT_URL: https://localhost:8443 [ciso_assistant.settings] ciso_assistant_url=https://localhost:8443
2024-05-02T16:38:29.011916Z [info ] ALLOWED_HOSTS: ['backend'] [ciso_assistant.settings] ciso_assistant_url=https://localhost:8443
2024-05-02T16:38:29.012817Z [info ] SQLITE_FILE: /code/db/ciso-assistant.sqlite3 [ciso_assistant.settings] ciso_assistant_url=https://localhost:8443
2024-05-02T16:38:29.013431Z [info ] DATABASE ENGINE: django.db.backends.sqlite3 [ciso_assistant.settings] ciso_assistant_url=https://localhost:8443
Email: blablabla@bliblibli.blou
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 134, in debug_sql
yield
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 122, in execute
return super().execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 79, in execute
return self._execute_with_wrappers(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
return executor(sql, params, many, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 105, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/sqlite3/base.py", line 329, in execute
return super().execute(query, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc2' in position 7: surrogates not allowed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/code/manage.py", line 23, in <module>
main()
File "/code/manage.py", line 19, in main
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 413, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.11/site-packages/django/contrib/auth/management/commands/createsuperuser.py", line 89, in execute
return super().execute(*args, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 459, in execute
output = self.handle(*args, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/contrib/auth/management/commands/createsuperuser.py", line 131, in handle
error_msg = self._validate_username(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/contrib/auth/management/commands/createsuperuser.py", line 304, in _validate_username
self.UserModel._default_manager.db_manager(database).get_by_natural_key(
File "/usr/local/lib/python3.11/site-packages/django/contrib/auth/base_user.py", line 56, in get_by_natural_key
return self.get(**{self.model.USERNAME_FIELD: username})
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 645, in get
num = len(clone)
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 382, in __len__
self._fetch_all()
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1928, in _fetch_all
self._result_cache = list(self._iterable_class(self))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 91, in __iter__
results = compiler.execute_sql(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 121, in execute
with self.debug_sql(sql, params, use_last_executed_query=True):
File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 139, in debug_sql
sql = self.db.ops.last_executed_query(self.cursor, sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/sqlite3/operations.py", line 176, in last_executed_query
params = self._quote_params_for_last_executed_query(params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/sqlite3/operations.py", line 165, in _quote_params_for_last_executed_query
return cursor.execute(sql, params).fetchone()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc2' in position 7: surrogates not allowed
connect to ciso assistant on https://localhost:8443
for successive runs you can now use docker compose upI cleaned up the environment and try with docker-compose-build.sh (and environmental modifications in docker-compose-build.yml) installation and deployment are obviously longer, but at least it worked.
Yet, still unable to access the webUI :
For the record :
Honestly, I think i'll stop my tests now.
I understand that you focus on the paid edition, but it's a shame that community edition doesn't work at least a bit more out of the box.
Anyway, thanks a lot for the time you allowed me, I really appreciate ! ❤️
ab-smith
commented
2 weeks ago Hey, thanks for the feedback and sorry about the frustration :)
No, our focus is on all editions, and everyone is important. We have done multiple tests in different settings, confirmed by thousands of users using the self-managed community edition, but we might have missed some cases, given that on-premises configurations are virtually infinite. Yours is particularly interesting, especially the extra intermediate step of the hypervisor.
I'm currently setting up an environment that is equivalent to yours. If I can reproduce the issue, we will fix it; otherwise, we can schedule a quick call through Discord to debug your specific case at your convenience :)
edouardsaucisse
commented
2 weeks ago would be great ! Gonna join discord soon then :)
ab-smith
commented
2 weeks ago So good news, we are pretty confident that your error is originating from caddy with your lab settings for which you missed -i to generate the self-signed certificate for your https://dockertmp:8443
We've tested on a lab like yours as well as a virtualization config with vagrant.
and you'll get the ssl warning that you can accept (but not the blockage), and voilà :
so, if you're starting fresh:
docker-compose.yml./docker-compose.shand you should be good now 😊
for the migration random errors, could be related to a missing locale on your distro, we can talk it over later on.
Given that proxmox is not the standard pattern for most users, we will probably add another flavor of docker-compose if you confirm that this fixes it. Keep in mind that the -i should not be used in production as it could present a security risk.
Let me know.
ab-smith
commented
1 week ago Added more explicit instructions here:
https://intuitem.gitbook.io/ciso-assistant/deployment/remote-virtualization
LudovicARHIMAN
commented
1 week ago So good news, we are pretty confident that your error is originating from caddy with your lab settings for which you missed
-ito generate the self-signed certificate for yourhttps://dockertmp:8443We've tested on a lab like yours as well as a virtualization config with vagrant.
and you'll get the ssl warning that you can accept (but not the blockage), and voilà :
so, if you're starting fresh:
* make sure to clean the repo, * do the **three** changes on the `docker-compose.yml` * run `./docker-compose.sh`and you should be good now 😊
for the migration random errors, could be related to a missing locale on your distro, we can talk it over later on.
Given that proxmox is not the standard pattern for most users, we will probably add another flavor of docker-compose if you confirm that this fixes it. Keep in mind that the
-ishould not be used in production as it could present a security risk.Let me know.
This fix works for me, I can reach the page without any SSL problems.
ab-smith
commented
1 week ago @edouardsaucisse, all good?
edouardsaucisse
commented
1 week ago Yep, looks good 👍 I tried it last night, seems to be fine. I'm gonna deep test it this week, i'll give you feedback if i face any issue :)
thanks a lot for your help and your time, i really appreciate !
I followed the instruction to deploy a local instance of CISO Assistant, relying on docker. I had some critical issues :
To Reproduce I follow the steps described in README.md, on the local installation section. As I am in a ubuntu 22.04 with docker v24, I modified the docker-compose.sh script to use docker-compose command instead of docker compose. I do not expect it to have huge impacts on the following bugs.
Expected behavior I expected the installation to go smooth as described
Screenshots Screenshot of docker containers running :
Screenshot of http error
Screenshot of ssl error
Environment (please complete the following information): SERVER
WORKSTATION
Additional context when running docker-compose.sh with docker-compose commands instead of docker compose, i get these messages, detailling some python issues (dont consider the "Deleted" message at the beginning, they're just cleaning files from a previous try) :
I guess it would deserve a split in several issues, but it's my first "contribution" on gh, so... forgive me if I'm doing wrong.