search

mozilla / translations

The code, training pipeline, and models that power Firefox Translations
https://mozilla.github.io/translations/
Mozilla Public License 2.0
154 stars 33 forks source link

previous_group_ids needs to gracefully handle upstream tasks being expired #623

Open bhearsum opened 5 months ago

bhearsum commented 5 months ago

Currently, we can end up with errors like this:

[task 2024-05-22T18:53:38.882Z] Traceback (most recent call last):
[task 2024-05-22T18:53:38.883Z]   File "/usr/local/lib/python3.11/dist-packages/taskgraph/main.py", line 708, in action_callback
[task 2024-05-22T18:53:38.883Z]     return trigger_action_callback(
[task 2024-05-22T18:53:38.883Z]            ^^^^^^^^^^^^^^^^^^^^^^^^
[task 2024-05-22T18:53:38.883Z]   File "/usr/local/lib/python3.11/dist-packages/taskgraph/actions/registry.py", line 345, in trigger_action_callback
[task 2024-05-22T18:53:38.883Z]     cb(Parameters(**parameters), graph_config, input, task_group_id, task_id)
[task 2024-05-22T18:53:38.883Z]   File "/builds/worker/checkouts/src/taskcluster/translations_taskgraph/actions/train.py", line 399, in train_action
[task 2024-05-22T18:53:38.883Z]     parameters["existing_tasks"] = get_ancestors(start_task_ids)
[task 2024-05-22T18:53:38.883Z]                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[task 2024-05-22T18:53:38.883Z]   File "/usr/local/lib/python3.11/dist-packages/taskgraph/util/taskcluster.py", line 409, in get_ancestors
[task 2024-05-22T18:53:38.883Z]     upstream_tasks.update(_get_deps(tuple(task_def["dependencies"]), use_proxy))

...which appears to be happening when we fetch some way upstream dependencies from a task in previous_group_ids. This may need an upstream fix in taskgraph.