mozilla / firefox-translations-training

Training pipelines for Firefox Translations neural machine translation models
https://mozilla.github.io/firefox-translations-training/
Mozilla Public License 2.0
135 stars 28 forks source link

Use pip-compile for tracking dependencies #695

Closed La0 closed 2 days ago

La0 commented 4 days ago

Closes #689

La0 commented 3 days ago

@bhearsum pip with pyproject.toml does not seem to parse correctly the requirements.txt file with hashes (removing all white spaces on each line). Have you ever encountered that ?

bhearsum commented 3 days ago

@bhearsum pip with pyproject.toml does not seem to parse correctly the requirements.txt file with hashes (removing all white spaces on each line). Have you ever encountered that ?

That's new to me, but I've also never used a requirements file reference in pyproject.toml. Everywhere else we do this we pip install from a requirements file.

bhearsum commented 3 days ago

I think I'm very confused though. Dependencies are being installed with poetry install. Doesn't it (shouldn't it?) be sourcing dependencies from a Poetry lockfile instead?

La0 commented 3 days ago

AFAIK the tracking code is installed through pip install in the tasks: https://github.com/mozilla/firefox-translations-training/blob/main/taskcluster/kinds/train-teacher/kind.yml#L125 (same for the other dependencies).

I initially did not include the hashes, as the file pointed out by Evgeny did not have those

eu9ene commented 3 days ago

@La0 We should expedite the fix as it's blocking other PRs and likely training too. I would approve but the tests and linter fail.

La0 commented 2 days ago

@eu9ene I have reverted to my initial patch, without hashes, that works on CI.

eu9ene commented 2 days ago

I tested it here and it looks working: https://firefox-ci-tc.services.mozilla.com/tasks/groups/GdnVgLB1Q7Kcjqoej9mEOA