[k8s] initializer

[dsever]

Bug description

I'm using helm (from DEV), when launching initializer there I'm getting:

psycopg2.errors.UndefinedTable: relation "defectdojo_engagement_survey_question" does not exist LINE 1: select polymorphic_ctype_id from defectDojo_engagement_surve...

From logs I can conclude that, problem is with case sensitivity?

helm in master repo is not supporting helm 3

Thanks

Steps to reproduce Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior A clear and concise description of what you expected to happen.

Deployment method (select with an X)

• [X] Kubernetes
• [ ] Docker
• [ ] setup.bash / legacy-setup.bash

Environment information

• Operating System: [e.g. Ubuntu 18.04]
• DefectDojo Commit Message: [use git show -s --format="%h: %s [%ci]"]

Sample scan files (optional) If applicable, add sample scan files to help reproduce your problem.

Screenshots (optional) If applicable, add screenshots to help explain your problem.

Console logs (optional) If applicable, add console logs to help explain your problem.

Additional context (optional) Add any other context about the problem here.

[dsever]

But if calling system_settings it crashes the application, looks like latest docker images were not build by master repo (using docker- compose it looks ok)? For sure announced container inside of dockerhub are actually from DEV????

[dsever]

Looks like problem psycopg2.errors.UndefinedTable: relation "defectdojo_engagement_survey_question" does not exist only exists in postgresql configuration.

Workarround is to put table name under quotes, https://github.com/DefectDojo/django-DefectDojo/blob/dev/dojo/management/commands/import_surveys.py#L25 for postgresql (also tested with pgadmin, it really cann't find table by name without quotes), but at the other hand for mysql as backend it will fail.

[ptrovatelli]

I think i have this error too, still the application is working; i'm not using surveys though

[athurm]

We have this error in postgresql configuration too, but system settings working.

[ptrovatelli]

i never use the images from dockerhub; always build my own.

dockerhub is from master i think but before yesterday master was monthes old and missing many fixes.

we updated master yesterday, the images should be more recent now

[dsever]

@athurm this error reflects only to Survay, with system_settings was something else (I'm not sure 100% what is it)

[athurm]

system_settings not working if rabbitMQ not working ...

we updated master yesterday, the images should be more recent now

https://hub.docker.com/r/defectdojo/defectdojo-django/tags @ptrovatelli latest still seams a month old :/

[ptrovatelli]

my bad, I thought this was automatic :) still, building local images is the best way to use whatever version you need and be sure that the images you work with contain the code you're looking at. you don't even need a private registry with the imagePullPolicy=Never parameter

[dinvlad]

So is there any workaround for this for now? Thanks

[dsever]

@dinvlad, for what, to retrieve new image of for postgres problem? In anycase I'm using option to build new image for myself and I'm pushing it to my private dockerhub

[dinvlad]

@dsever yep, we've just patched and rebuilt it thanks to your https://github.com/DefectDojo/django-DefectDojo/issues/1890#issuecomment-589996536 (I was confused before). It'd be nice to have this patched upstream though. Do I understand correctly that quoting the table in this case makes it incompatible with MySQL (so this patch is blocked atm)?

[dsever]

So there are 2 options, one is to detect mysql vs postgres and than execute query (I see it as antipattern).

The other permanent solution is rename table to proper naming, but I don't know how it will reflect to the logic behind and upgrade procedure.

[ptrovatelli]

@dsever what would be the proper naming?

[dsever]

I would say that issues come from non standard naming of the table (defectdojo_engagement_survey_question), so some reason pqsql doesn't understand it or sees some unsupported chars or keywords.

Or this could be the case: Postgresql treats identifiers case insensitively when not quoted (it actually folds them to lowercase internally), and case sensitively when quoted; many people are not aware of this idiosyncrasy. Using always lowercase you are safe. Anyway, it's acceptable to use camelCase or PascalCase (or UPPER_CASE), as long as you are consistent: either quote identifiers always or never (and this includes the schema creation!).

And we have: in query:

cursor.execute("select polymorphic_ctype_id from defectDojo_engagement_survey_question;")

Or even during the table creation...

This is something that has to be tested

[ptrovatelli]

quoting will be problematic. for mysql it's backquotes, for other engines it's double quotes. If the case is really the issue, I don't see any clean solution except for putting everything in lower case but that's a lot of impacts

Note that the default database when using docker-compose is mysql while with kubernetes it's postgresql. Alghough it's easy to switch to mysql in kubernetes if you really need the surveys (but i find the postgres helm chart is better thanks to the statefulset). the contrary is not true (using postgres with docker-compose is not trivial)

@Maffooch what's your take on this? have you ever tried using surveys with a postgresql database?

[dsever]

As kind of workaround we can build separate images, for init job, but the problem is if we want to use generic helm from the repo.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.