sqlalchemy error

[corviday]

Docker instances of the newest version (3.6.2) throw the following error when any API call is made:

2024-02-20 17:03:47 [13] [ERROR] Exception on /api/multimeta [GET]
Traceback (most recent call last):
File "/root/.cache/pypoetry/virtualenvs/ce-9TtSrW0h-py3.8/lib/python3.8/site-packages/sqlalchemy/util/_collections.py", line 1020, in __call__
return self.registry[key]
KeyError: 140433640835912

This does not seem to be an issue with our python code, as a version with identical code has been deployed as an unmerged branch. It is perhaps caused by an update to sqlalchemy.

[corviday]

• The version of sqlalchemy used by the app has not changed.
• The error does not occur when the same app is run on bare metal
• The file referenced in the error does exist inside the docker
• comparable docker and bare-metal versions use the same version of poetry

[corviday]

I built a docker from 32ce883, and it did not have the error, so I think the error was introduced sometime after that hash, though it is still unclear what sort of error might appear only in docker containers and not on bare metal.

[jameshiebert]

@corviday have you tried running the test suite in a docker container? Does the /multimeta API call fail under that environment? Just trying to get a minimal test case so that we can narrow this down and work on it.

[corviday]

That's a great idea, thanks. The test suite has some failures, but they seem to be unrelated (a needed directory isn't mounted to the container) but other tests pass. I don't know what that means, but it's useful.

EDIT: I have restored the missing directory and all tests pass now. Hmmm.

[jameshiebert]

OK, that being the case can you give a full "steps to reproduce" overview then?

[corviday]

Steps to reproduce:

1. Build a docker of release version 3.6.2, or use the one on docker hub
2. Start the docker
3. access any API endpoint to get an error. /multimeta is an easy one because it has defaults for all the parameters.

Given that tests work, issue is likely flask-related? I don't think flask is used the same way in tests as with the server running live. Hm, except that flask is used when running on bare metal, and that(s fine.

[jameshiebert]

Can you be more specific about what parameters you're using for step 2 and exactly what URL you're using from step 3? I still cannot reproduce this.

[corviday]

That's good news, I think?

Here are the current docker-compose.yaml and be.env files, with their extension changed and passwords redacted. be.env.txt docker-compose.yaml.txt

Here is my test URL.

[corviday]

Things that don't fix it:

• downgrading flask packages to match the versions in the known-good container
• switching database connection string to match the previous, known-good container
• running as a production container instead of a development container (ie, gunicorn instead of flask)

Switching GDAL and / or python back has so far proved to be quite a tangle and I don't have an answer yet.

Working and buggy containers use the same version of sqlalchemy, so that's probably not the cause.