Isolate job's requirements

[iszulcdeepsense]

A job runs a single Python interpreter process that contains 2 codebases:

• wrapper code (handling http, metrics, swagerUI, etc)
• actual job's logic

Each of them have its third-party dependencies installed with pip install -r requirements.txt. That raises a risk of conflicting packages. Although they are installed in 2 different virtual environments, they're still running in the same single Python process that makes it impossible to load 2 different versions of the same package at the same time. For instance, if wrapper code relies on fastapi==0.100.0, but the job depends on fastapi==0.90.0, there might be the clash, cause the wrapper's library takes precedence when loading modules in Python and the job will use fastapi==0.90.0

I see the only good way out of here: Wrapper code shouldn't have any third-party dependency (that could be overwritten or conflicted by a job). Then, there won't be any conflict. It can either implement the necessary logic on its own or use "forked" versions of libraries (eg. FastAPI renamed to fastapi_racetrack package).

This task has a low priority as it rarely happens to break something (but when it does, it's astonishing and hard to find).

[anders314159]

This issue popped up with Pydantic (ours is ==2.6.1 atm) and a certain Walking Cat (theirs is not 2.6.1) when using the Dockerfile jobtype, so we might want to look into vendoring Pydantic.

[iszulcdeepsense]

Speaking of "vendoring", that might be a good workaround. What we really need is to create a copy of pydantic package with all imports updated so that it can be imported with from pydantic_vendored import ... apart from catwalk's pydantic module. Do you know a tool for doing this?