search

tcosolutions / betterscan

Code Scanning/SAST/Static Analysis/Linting using many tools/Scanners with One Report (Code, IaC) - Betterscan
https://betterscan.io
GNU Affero General Public License v3.0
818 stars 90 forks source link

Docker Worker/Server error #197

Closed RicardoSette closed 2 months ago

RicardoSette commented 2 months ago

I used betterscan/docker/docker-compose.yml, but in betterscan/docker/docker/worker/Dockerfile and betterscan/docker/docker/server/Dockerfile this error is present in docker logs when trying to run:

worker

2024-09-10 14:31:46 Traceback (most recent call last):
2024-09-10 14:31:46   File "/srv/betterscan/manage.py", line 22, in <module>
2024-09-10 14:31:46     from quantifiedcode.settings import settings
2024-09-10 14:31:46   File "/srv/betterscan/quantifiedcode/settings/__init__.py", line 22, in <module>
2024-09-10 14:31:46     from .base import * #noqa
2024-09-10 14:31:46   File "/srv/betterscan/quantifiedcode/settings/base.py", line 33, in <module>
2024-09-10 14:31:46     from quantifiedcode.helpers.settings import Settings, load_settings
2024-09-10 14:31:46   File "/srv/betterscan/quantifiedcode/helpers/settings.py", line 37, in <module>
2024-09-10 14:31:46     from blitzdb.backends.sql import Backend as SqlBackend
2024-09-10 14:31:46   File "/usr/local/lib/python3.8/site-packages/blitzdb/backends/sql/__init__.py", line 22, in <module>
2024-09-10 14:31:46     from .backend import Backend
2024-09-10 14:31:46 ImportError: cannot import name 'Backend' from 'blitzdb.backends.sql.backend' (/usr/local/lib/python3.8/site-packages/blitzdb/backends/sql/backend.py)

server

2024-09-10 14:31:57 sh: 1: Syntax error: ";" unexpected
2024-09-10 14:32:23 sh: 1: Syntax error: ";" unexpected
2024-09-10 14:33:15 sh: 1: Syntax error: ";" unexpected

Information about the computer used and the repository was main.

% uname -a
Darwin MacBook-Air-de-Administrador.local 23.5.0 Darwin Kernel Version 23.5.0: Wed May  1 20:19:05 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T8112 arm64
% git clone https://github.com/tcosolutions/betterscan.git

The only change was port 5000:5000 to 5001:5000 in the betterscan/docker/docker-compose.yml file. Docker postgres and rabbitmq running ok.

I tried betterscan/dockerhub/docker-compose.yml, but I get the error 'no matching manifest for linux/arm64/v8 in the manifest list entries' when I try to run it.

How can I help the project to work on a MacAir M2?

marcinguy commented 2 months ago

@RicardoSette I am actually debugging it now

marcinguy commented 2 months ago

Pushing latest changes now. Will see

RicardoSette commented 2 months ago

@marcinguy now docker sunsolution/betterscan-worker:latest execute and don't manifest error for linux/arm64/v8

I cleaned all docker caches ("docker buildx prune -f"), but we still have errors when running the server or worker of both docker-compose.yml:

Errors with docker/docker-compose.yml: docker-server

2024-09-12 13:06:37 sh: 1: Syntax error: ";" unexpected
2024-09-12 13:06:37 sh: 1: Syntax error: ";" unexpected
2024-09-12 13:06:38 sh: 1: Syntax error: ";" unexpected
2024-09-12 13:06:38 sh: 1: Syntax error: ";" unexpected
2024-09-12 13:06:39 sh: 1: Syntax error: ";" unexpected
2024-09-12 13:06:41 sh: 1: Syntax error: ";" unexpected
2024-09-12 13:06:44 sh: 1: Syntax error: ";" unexpected
2024-09-12 13:06:51 sh: 1: Syntax error: ";" unexpected
2024-09-12 13:07:04 sh: 1: Syntax error: ";" unexpected

docker-worker_1

2024-09-12 13:07:31 Traceback (most recent call last):
2024-09-12 13:07:31   File "/srv/betterscan/manage.py", line 22, in <module>
2024-09-12 13:07:31     from quantifiedcode.settings import settings
2024-09-12 13:07:31   File "/srv/betterscan/quantifiedcode/settings/__init__.py", line 22, in <module>
2024-09-12 13:07:31     from .base import * #noqa
2024-09-12 13:07:31   File "/srv/betterscan/quantifiedcode/settings/base.py", line 33, in <module>
2024-09-12 13:07:31     from quantifiedcode.helpers.settings import Settings, load_settings
2024-09-12 13:07:31   File "/srv/betterscan/quantifiedcode/helpers/settings.py", line 37, in <module>
2024-09-12 13:07:31     from blitzdb.backends.sql import Backend as SqlBackend
2024-09-12 13:07:31   File "/usr/local/lib/python3.8/site-packages/blitzdb/backends/sql/__init__.py", line 22, in <module>
2024-09-12 13:07:31     from .backend import Backend
2024-09-12 13:07:31 ImportError: cannot import name 'Backend' from 'blitzdb.backends.sql.backend' (/usr/local/lib/python3.8/site-packages/blitzdb/backends/sql/backend.py)

Errors with dockerhub/docker-compose.yml: sunsolution/betterscan-worker:latest (dockerhub-server-1)

2024-09-12 12:56:29 Traceback (most recent call last):
2024-09-12 12:56:29   File "/srv/betterscan/manage.py", line 22, in <module>
2024-09-12 12:56:29     from quantifiedcode.settings import settings
2024-09-12 12:56:29   File "/srv/betterscan/quantifiedcode/settings/__init__.py", line 22, in <module>
2024-09-12 12:56:29     from .base import * #noqa
2024-09-12 12:56:29   File "/srv/betterscan/quantifiedcode/settings/base.py", line 33, in <module>
2024-09-12 12:56:29     from quantifiedcode.helpers.settings import Settings, load_settings
2024-09-12 12:56:29   File "/srv/betterscan/quantifiedcode/helpers/settings.py", line 37, in <module>
2024-09-12 12:56:29     from blitzdb.backends.sql import Backend as SqlBackend
2024-09-12 12:56:29   File "/usr/local/lib/python3.8/site-packages/blitzdb/backends/sql/__init__.py", line 22, in <module>
2024-09-12 12:56:29     from .backend import Backend
2024-09-12 12:56:29 ImportError: cannot import name 'Backend' from 'blitzdb.backends.sql.backend' (/usr/local/lib/python3.8/site-packages/blitzdb/backends/sql/backend.py)

sunsolution/betterscan-worker:latest (dockerhub-worker_1-1)

2024-09-12 12:55:49 I cannot find a settings file, please specify one via the QC_SETTINGS environment variable.
2024-09-12 12:55:49 Traceback (most recent call last):
2024-09-12 12:55:49   File "/srv/betterscan/manage.py", line 22, in <module>
2024-09-12 12:55:49     from quantifiedcode.settings import settings
2024-09-12 12:55:49   File "/srv/betterscan/quantifiedcode/settings/__init__.py", line 22, in <module>
2024-09-12 12:55:49     from .base import * #noqa
2024-09-12 12:55:49   File "/srv/betterscan/quantifiedcode/settings/base.py", line 62, in <module>
2024-09-12 12:55:49     settings = load_settings(settings_filenames)
2024-09-12 12:55:49   File "/srv/betterscan/quantifiedcode/helpers/settings.py", line 249, in load_settings
2024-09-12 12:55:49     with open(filename, 'r') as yaml_file:
2024-09-12 12:55:49 FileNotFoundError: [Errno 2] No such file or directory: '/srv/betterscan/settings.yml'
2024-09-12 12:55:49 Loading settings from /srv/betterscan/quantifiedcode/settings/default.yml
2024-09-12 12:55:49 Loading settings from /srv/betterscan/settings.yml

If you need any more information or testing, let me know.

marcinguy commented 2 months ago

Updated code and config files

marcinguy commented 2 months ago

Pushing now the images

RicardoSette commented 2 months ago

I updated git and cleaned all docker caches ("docker buildx prune -f"), but we still have errors when running the server or worker of both docker-compose.yml:

After update git to execute docker/docker-compose.yml:

docker compose  -f "docker/docker-compose.yml" up -d --build postgres rabbitmq3 server worker_1 

[+] Building 199.4s (18/18) FINISHED                                                                                 
 => [server internal] load build definition from Dockerfile                                                     0.0s
 => => transferring dockerfile: 1.50kB                                                                          0.0s
 => [server internal] load .dockerignore                                                                        0.0s
 => => transferring context: 2B                                                                                 0.0s
 => [worker_1 internal] load metadata for docker.io/library/python:3.8-slim                                     2.8s
 => [server auth] library/python:pull token for registry-1.docker.io                                            0.0s
 => CACHED [worker_1 1/7] FROM docker.io/library/python:3.8-slim@sha256:4dd2165f119c97c32c1d30b62bbffcd4bbb0b3  2.6s
 => => resolve docker.io/library/python:3.8-slim@sha256:4dd2165f119c97c32c1d30b62bbffcd4bbb0b354d6c5522c024406  0.0s
 => => sha256:35379892d330fb362e0a98aae19c1e3d89027b6163bbae6a14df19de441febb0 5.27kB / 5.27kB                  0.0s
 => => sha256:92c3b3500be621c72c7ac6432a9d8f731f145f4a1535361ffd3a304e55f7ccda 29.16MB / 29.16MB                1.0s
 => => sha256:89790d4ca55c29720fc29c489ba4403f3bb6baa1a6d8b5d0e96c3d40521408c0 3.33MB / 3.33MB                  0.6s
 => => sha256:7321812fcf9f48f3ad00a6d7a09f5b23a9463fb49cae4cc86710c25a7ea27138 14.51MB / 14.51MB                1.2s
 => => sha256:4dd2165f119c97c32c1d30b62bbffcd4bbb0b354d6c5522c024406b5b874ac40 10.41kB / 10.41kB                0.0s
 => => sha256:cfd932084cf5dbcc28ca3577027b2255e3aee75086206abf737ef1e176804553 1.75kB / 1.75kB                  0.0s
 => => sha256:f885f51457a017b2d0b45c60398897e4b212821e6353fb286e7ae5c4a34b895b 251B / 251B                      0.8s
 => => extracting sha256:92c3b3500be621c72c7ac6432a9d8f731f145f4a1535361ffd3a304e55f7ccda                       0.9s
 => => extracting sha256:89790d4ca55c29720fc29c489ba4403f3bb6baa1a6d8b5d0e96c3d40521408c0                       0.1s
 => => extracting sha256:7321812fcf9f48f3ad00a6d7a09f5b23a9463fb49cae4cc86710c25a7ea27138                       0.4s
 => => extracting sha256:f885f51457a017b2d0b45c60398897e4b212821e6353fb286e7ae5c4a34b895b                       0.0s
 => [server 2/4] RUN apt-get update && apt-get install -y libssl-dev git python3-pip npm curl ruby-sass rsync  43.8s
 => [server 3/4] RUN pip install --upgrade pip && pip install checkmate3                                        7.0s
 => [server 4/4] RUN mkdir -p /srv &&     git clone https://github.com/tcosolutions/betterscan /srv/bettersca  48.7s
 => [server] exporting to image                                                                                 2.9s
 => => exporting layers                                                                                         2.9s
 => => writing image sha256:c3cad6c8258a6068e3cadb2a0da852f7f8af066758521438dacb76709d70df4f                    0.0s
 => => naming to docker.io/library/docker-server                                                                0.0s
 => [worker_1 internal] load build definition from Dockerfile                                                   0.0s
 => => transferring dockerfile: 2.31kB                                                                          0.0s
 => [worker_1 internal] load .dockerignore                                                                      0.0s
 => => transferring context: 2B                                                                                 0.0s
 => [worker_1 2/7] RUN apt-get update && apt-get install -y --no-install-recommends     ca-certificates     c  43.0s
 => [worker_1 3/7] RUN GO_JSON=$(curl -s https://go.dev/dl/?mode=json) &&     GO_LATEST=$(echo "$GO_JSON" | jq  4.8s
 => [worker_1 4/7] RUN gem install brakeman                                                                     3.3s
 => [worker_1 5/7] RUN pip install --upgrade pip && pip install checkmate2                                      6.8s
 => [worker_1 6/7] RUN git clone https://github.com/tcosolutions/betterscan.git /srv/betterscan &&     git clo  4.7s
 => [worker_1 7/7] RUN cp /srv/betterscan/analyzers/find_unicode_control2.py /usr/local/bin/ &&     go instal  26.3s
 => [worker_1] exporting to image                                                                               2.5s
 => => exporting layers                                                                                         2.5s
 => => writing image sha256:63d3d93ab8890a0a95fc4c24b68a4a6a37007b60bbd62e5838c84cc3ce4c01bc                    0.0s
 => => naming to docker.io/library/docker-worker_1                                                              0.0s
[+] Running 5/5
 ✔ Network docker_default       Created                                                                         0.0s 
 ✔ Container rabbitmq           Started                                                                         0.5s 
 ✔ Container docker-postgres-1  Started                                                                         0.5s 
 ✔ Container docker-server-1    Started                                                                         0.6s 
 ✔ Container docker-worker_1-1  Started

Errors with docker/docker-compose.yml: docker-server

2024-09-13 08:18:50 sh: 1: Syntax error: ";" unexpected
2024-09-13 08:18:50 sh: 1: Syntax error: ";" unexpected
2024-09-13 08:18:51 sh: 1: Syntax error: ";" unexpected
2024-09-13 08:18:51 sh: 1: Syntax error: ";" unexpected
2024-09-13 08:18:52 sh: 1: Syntax error: ";" unexpected
2024-09-13 08:18:54 sh: 1: Syntax error: ";" unexpected
2024-09-13 08:18:58 sh: 1: Syntax error: ";" unexpected

docker-worker_1

2024-09-13 08:19:19 Traceback (most recent call last):
2024-09-13 08:19:19   File "/srv/betterscan/manage.py", line 22, in <module>
2024-09-13 08:19:19     from quantifiedcode.settings import settings
2024-09-13 08:19:19   File "/srv/betterscan/quantifiedcode/settings/__init__.py", line 22, in <module>
2024-09-13 08:19:19     from .base import * #noqa
2024-09-13 08:19:19   File "/srv/betterscan/quantifiedcode/settings/base.py", line 33, in <module>
2024-09-13 08:19:19     from quantifiedcode.helpers.settings import Settings, load_settings
2024-09-13 08:19:19   File "/srv/betterscan/quantifiedcode/helpers/settings.py", line 37, in <module>
2024-09-13 08:19:19     from blitzdb.backends.sql import SqlBackend
2024-09-13 08:19:19 ImportError: cannot import name 'SqlBackend' from 'blitzdb.backends.sql' (/usr/local/lib/python3.8/site-packages/blitzdb/backends/sql/__init__.py)

After update git to execute dockerhub/docker-compose.yml:

docker compose  -f "dockerhub/docker-compose.yml" up -d --build postgres rabbitmq3 server worker_1 

WARN[0000] The "OPENAI_GPT_API" variable is not set. Defaulting to a blank string. 
WARN[0000] The "OPENAI_GPT_API" variable is not set. Defaulting to a blank string. 
[+] Running 15/15
 ✔ worker_1 6 layers [⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                           37.6s 
   ✔ 5efb61beb273 Pull complete                                                                                24.3s 
   ✔ 25bdd178bf7b Pull complete                                                                                28.3s 
   ✔ 8579f7dcf9cd Pull complete                                                                                29.4s 
   ✔ bf42f3aa2e7c Pull complete                                                                                30.6s 
   ✔ 0a20d2e17b2b Pull complete                                                                                31.3s 
   ✔ 7e57ff9da3c6 Pull complete                                                                                34.8s 
 ✔ server 7 layers [⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                            35.5s 
   ✔ 92c3b3500be6 Already exists                                                                                0.0s 
   ✔ 89790d4ca55c Already exists                                                                                0.0s 
   ✔ 7321812fcf9f Already exists                                                                                0.0s 
   ✔ f885f51457a0 Already exists                                                                                0.0s 
   ✔ 1b7f5bf0c8b7 Pull complete                                                                                21.8s 
   ✔ 911d23180ecc Pull complete                                                                                22.9s 
   ✔ 6712bb7e635f Pull complete                                                                                32.8s 
[+] Running 8/8
 ✔ Network dockerhub_default       Created                                                                      0.0s 
 ✔ Volume "dockerhub_data2"        Created                                                                      0.0s 
 ✔ Volume "dockerhub_setup_state"  Created                                                                      0.0s 
 ✔ Volume "dockerhub_data1"        Created                                                                      0.0s 
 ✔ Container dockerhub-postgres-1  Healthy                                                                     12.3s 
 ✔ Container rabbitmq              Healthy                                                                     12.3s 
 ✘ Container dockerhub-server-1    Error                                                                       11.6s 
 ✔ Container dockerhub-worker_1-1  Created                                                                      0.0s 
dependency failed to start: container dockerhub-server-1 has no healthcheck configured

Errors with dockerhub/docker-compose.yml: sunsolution/betterscan-worker:latest (dockerhub-server-1)

2024-09-13 08:12:52 Traceback (most recent call last):
2024-09-13 08:12:52   File "/srv/betterscan/manage.py", line 22, in <module>
2024-09-13 08:12:52     from quantifiedcode.settings import settings
2024-09-13 08:12:52   File "/srv/betterscan/quantifiedcode/settings/__init__.py", line 22, in <module>
2024-09-13 08:12:52     from .base import * #noqa
2024-09-13 08:12:52   File "/srv/betterscan/quantifiedcode/settings/base.py", line 33, in <module>
2024-09-13 08:12:52     from quantifiedcode.helpers.settings import Settings, load_settings
2024-09-13 08:12:52   File "/srv/betterscan/quantifiedcode/helpers/settings.py", line 37, in <module>
2024-09-13 08:12:52     from blitzdb.backends.sql import SqlBackend
2024-09-13 08:12:52   File "/usr/local/lib/python3.8/site-packages/blitzdb/backends/sql/__init__.py", line 22, in <module>
2024-09-13 08:12:52     from .backend import Backend
2024-09-13 08:12:52 ImportError: cannot import name 'Backend' from 'blitzdb.backends.sql.backend' (/usr/local/lib/python3.8/site-packages/blitzdb/backends/sql/backend.py)

sunsolution/betterscan-worker:latest (dockerhub-worker_1-1)

2024-09-13 08:12:30 Traceback (most recent call last):
2024-09-13 08:12:30   File "/srv/betterscan/manage.py", line 22, in <module>
2024-09-13 08:12:30     from quantifiedcode.settings import settings
2024-09-13 08:12:30   File "/srv/betterscan/quantifiedcode/settings/__init__.py", line 22, in <module>
2024-09-13 08:12:30     from .base import * #noqa
2024-09-13 08:12:30   File "/srv/betterscan/quantifiedcode/settings/base.py", line 33, in <module>
2024-09-13 08:12:30     from quantifiedcode.helpers.settings import Settings, load_settings
2024-09-13 08:12:30   File "/srv/betterscan/quantifiedcode/helpers/settings.py", line 37, in <module>
2024-09-13 08:12:30     from blitzdb.backends.sql import SqlBackend
2024-09-13 08:12:30 ImportError: cannot import name 'SqlBackend' from 'blitzdb.backends.sql' (/usr/local/lib/python3.8/site-packages/blitzdb/backends/sql/__init__.py)
marcinguy commented 2 months ago

Had different error.

Retriggering CICD build now

marcinguy commented 2 months ago

Cannot reproduce

marcinguy commented 2 months ago

@RicardoSette sorry but I had different results/errors. It is like comparing apples to oranges.

Will check again as I will have time/energy

marcinguy commented 2 months ago

Try to make a PR to fix it @RicardoSette

marcinguy commented 2 months ago

@RicardoSette I did pushed some commits. Please try to CLI version. It is acutally a big effort, lots of work to maintain both platforms (CLI and Web based and then also different browser quirks, not to mention dependencies update for modern frontends, it is kinda out of control. Yea, buttons look pretty, but maintaing projects got much more compex, time consuming and maybe even less secure over the longer time (after deployment and initial excitement possibly passing on to the other person/developer or as times goes by)

marcinguy commented 2 months ago

Looking for more helpers on this. Little bit out of my capacity to make it bigger, when it is needed and practical (Asked for this also in the Discussions - what actually people need)