Better dependency resolution

[mikewilli]

There was a (replicable) issue in colab recently where pip install mozanalysis was getting incompatible dependencies (in this case: polyfactory==2.13.0 and typing-extensions==4.5.0), resulting in an error when trying to use it. In a separate local venv, I got polyfactory==2.13.0 and typing-extensions==4.9.0, which works.

Python is notoriously difficult when it comes to dependency management and package publication, but we should investigate whether there's anything we can do to at least try to address this.

[jaredsnyder]

How do folks feel about poetry? I have used it a lot in the past and liked how clean it made dealing with dependencies. Here is the pro/con list as I see it:

Pro:

• Very reliable dependency resolution
• We could consolidate several config files into a pyproject.toml (though this is also possible with setuptools, but I've found it to be a bit more difficult than with poetry)
• It also manages virtualenvs
• Changes would only affect developers

Cons:

• Dependency resolution is slow, when you add a new package it can take several minutes to figure it out
• Developers would have to modify their workflow

[mikewilli]

I'm fine with poetry and we use it on Experimenter, but it's also a different workflow that folks may not be used to. I'd probably lean towards using the setup that Jetstream has -- pip-tools generates a requirements.txt from requirements.in, and dependabot creates dep update PRs -- it may be an equally different workflow to poetry, but at least there'd be consistency with Jetstream and other similar tooling (like metric-config-parser).

Personally I don't have a strong opinion. I think there's an argument for either side so I'd be interested in getting input and/or votes from the regular contributors before making a decision. What do you think about putting together an ADR to list out the pros/cons of the options and make a recommendation? That would give people an opportunity to comment in the form of a PR. If you're open to the idea but unfamiliar with the ADR process I'd be happy to walk you through it (it's pretty straightforward I think).

[jaredsnyder]

I could put something together! I'm not familiar with the ADR process so I'll take you up on your offer to help

[jaredsnyder]

As per https://github.com/mozilla/mozanalysis/pull/214#discussion_r1524965545, check to make sure there aren't unnecessary pinned dependencies