search

mozilla / translations

The code, training pipeline, and models that power Firefox Translations
https://mozilla.github.io/translations/
Mozilla Public License 2.0
154 stars 33 forks source link

Create a new W&B project if it doesn't exist #639

Closed eu9ene closed 5 months ago

eu9ene commented 5 months ago

The script should create a project for a new language if it doesn't exist.

eu9ene commented 5 months ago

I'm marking it as a bug as it blocks training for new languages without creating a project manually.

vrigal commented 5 months ago

@eu9ene I was not able to reproduce this (neither from training nor evaluation task). W&B client automatically creates a new project when initializing the publisher. Can you provide me a link to the Task in failure please ?

eu9ene commented 5 months ago

https://firefox-ci-tc.services.mozilla.com/tasks/TZDN6HwtS7GTPnNa6XoFyA/runs/0/logs/public/logs/live.log

[task 2024-05-25T01:25:46.197Z] [tracking INFO] Publishing experiment config to a 'group_logs' fake run.
[task 2024-05-25T01:25:46.414Z] [tracking INFO] Handling group_logs publication
[task 2024-05-25T01:25:46.414Z] [tracking INFO] Retrieving W&B names from taskcluster attributes
[task 2024-05-25T01:25:46.690Z] [tracking ERROR] Publication failed
[task 2024-05-25T01:25:46.690Z] Traceback (most recent call last):
[task 2024-05-25T01:25:46.690Z]   File "/home/ubuntu/.local/lib/python3.10/site-packages/translations_parser/cli/taskcluster.py", line 147, in main
[task 2024-05-25T01:25:46.690Z]     boot()
[task 2024-05-25T01:25:46.690Z]   File "/home/ubuntu/.local/lib/python3.10/site-packages/translations_parser/cli/taskcluster.py", line 129, in boot
[task 2024-05-25T01:25:46.690Z]     publish_group_logs_from_tasks(config=config)
[task 2024-05-25T01:25:46.690Z]   File "/home/ubuntu/.local/lib/python3.10/site-packages/translations_parser/utils.py", line 263, in publish_group_logs_from_tasks
[task 2024-05-25T01:25:46.690Z]     WandB.publish_group_logs(parents, project, group, existing_runs=[])
[task 2024-05-25T01:25:46.690Z]   File "/home/ubuntu/.local/lib/python3.10/site-packages/translations_parser/publishers.py", line 227, in publish_group_logs
[task 2024-05-25T01:25:46.690Z]     len(wandb.Api().runs(project, filters={"display_name": "group_logs", "group": group}))
[task 2024-05-25T01:25:46.690Z]   File "/home/ubuntu/.local/lib/python3.10/site-packages/wandb/apis/public.py", line 1218, in __len__
[task 2024-05-25T01:25:46.690Z]     self._load_page()
[task 2024-05-25T01:25:46.690Z]   File "/home/ubuntu/.local/lib/python3.10/site-packages/wandb/apis/public.py", line 1248, in _load_page
[task 2024-05-25T01:25:46.690Z]     self.objects.extend(self.convert_objects())
[task 2024-05-25T01:25:46.690Z]   File "/home/ubuntu/.local/lib/python3.10/site-packages/wandb/apis/public.py", line 1815, in convert_objects
[task 2024-05-25T01:25:46.690Z]     raise ValueError("Could not find project %s" % self.project)
[task 2024-05-25T01:25:46.690Z] ValueError: Could not find project en-uk