search

orchestracities / ngsi-timeseries-api

QuantumLeap: a FIWARE Generic Enabler to support the usage of NGSIv2 (and NGSI-LD experimentally) data in time-series databases
https://quantumleap.rtfd.io/
MIT License
37 stars 49 forks source link

Broken Docker image #735

Closed c0c0n3 closed 11 months ago

c0c0n3 commented 11 months ago

Describe the bug

Can't build the Docker image at the moment. The step where we install the Python deps fails. Since we don't pin every dep, most likely we've bumped in another instance of self-inflicted dependency hell.

To Reproduce

See e.g. PR #730's build log

Or run docker build -t orchestracities/quantumleap . from the root dir yourself.

Expected behavior

You should be able to build the Docker image.

Environment

Additional context

This is just another installment of the neverending dependency hell saga, see e.g. #273.

c0c0n3 commented 11 months ago

So we should really move away from Python 3.8, but that's too much effort. An avenue we can explore is to upgrade pyyaml from version 5 to version 6. With this upgrade I could make some progress locally, it remains to be seen if that works on CircleCI too. (I still need to upgrade other deps though.) The reason we can't upgrade to version 6 without pain is PR #516 which added a ~=5.4 constraint to the Pipenv dev deps

But I think that was a useless thing to do anyway. In fact, the same PR installs the latest pyyaml version on every CircleCI test session run:

which means at the moment we get version 6 on CircleCI. So what's the use of adding the ~=5.4 constraint to the Pipenv dev deps? Also the main section of the Pipenv file requires pyyaml >= 4.2. So it looks like something fell through the cracks here. My gut is that we should undo this change

and pin pyyaml to version 6 in the main section of Pipfile.

c0c0n3 commented 11 months ago

Ideally we should upgrade requests and redis too b/c of security vulnerabilities---see #729 and #724.

c0c0n3 commented 11 months ago

Notice that the upgrade path isn't straightforward. The problem is that, like us, other Pythonistas out there also have dependency hell issues. Here's an example. If you delete the Pipenv lock file and try regenerating it (or just upgrade deps) you'll see that all the QL tests will break. One of the errors is this

ImportError: cannot import name 'soft_unicode' from 'markupsafe'

(see https://github.com/pallets/markupsafe/issues/282)

Why is that happening? Well, our deps declaration in the Pipfile lead to the following dependency tree---just showing the paths that contain the markupsafe package.

connexion 2.14.2 Connexion - API first applications with OpenAPI/Swagger and Flask
├── flask >=1.0.4,<2.3
|   ├── jinja2 >=2.10.1,<3.0
|       └── markupsafe >=0.23
├── swagger-ui-bundle >=0.0.2,<0.1
    └── jinja2 >=2.0
        └── markupsafe >=0.23

flask 1.1.4 A simple framework for building complex web applications.
├── jinja2 >=2.10.1,<3.0
    └── markupsafe >=0.23

pytest-flask 1.2.0 A set of py.test fixtures to test Flask applications.
├── flask *
    ├── jinja2 >=2.10.1,<3.0
        └── markupsafe >=0.23

With these dependency constraints, the dependency solver can pick any version of the jinja2 package greater or equal to 2.10.1 but less than 3 and any markupsafe version greater or equal to 0.23. So the solver picks

But in actual fact, jinja2 2.11.3 is incompatible with markupsafe 2.1.3 because it tries to import soft_unicode from markupsafe but soft_unicode isn't in markupsafe 2.1.3. The last version containing soft_unicode is markupsafe 2.0.1. So the jinja2 maintainers should've specified an upper version bound of 2.0.1 for markupsafe in their deps, instead of just a lower bound of 0.23:

Anyways, what can we do about it? One option would be to add an explicit dependency on markupsafe 2.0.1 in our Pipfile. This isn't great for all the obvious reasons, but it's an option nonetheless. Another option would be to upgrade connexion and flask to recent versions which depend on jinja2 > 3.0---recent jinja2 versions actually work with the latestmarkupsafe version, 2.1.3. But Python being that wonderful dynamic/untyped soup that it is, I'm a bit scared of upgrading flask without extensive testing.