search

mozilla / translations

The code, training pipeline, and models that power Firefox Translations
https://mozilla.github.io/translations/
Mozilla Public License 2.0
154 stars 33 forks source link

GPUs stopped working on bicleaner-ai for id-en #724

Closed eu9ene closed 3 months ago

eu9ene commented 3 months ago

https://firefox-ci-tc.services.mozilla.com/tasks/NWpySVWkT2SVBY3AOkDaIQ

Apparently, bicleaner stopped detecting the GPU midway based on @gregtatum's investigation:

[task 2024-06-21T19:46:44.060Z] 2024-06-21 19:46:42,826 - INFO - Elapsed time 339.78 s
[task 2024-06-21T19:51:43.669Z] 2024-06-21 19:51:42,372 - INFO - Elapsed time 339.33 s
[task 2024-06-21T19:51:45.249Z] 2024-06-21 19:51:43,943 - INFO - Elapsed time 338.51 s
[task 2024-06-21T19:51:45.260Z] 2024-06-21 19:51:41,348 - INFO - Elapsed time 333.78 s
[task 2024-06-21T19:52:37.513Z] 2024-06-21 19:52:36,294 - INFO - Elapsed time 340.27 s
[task 2024-06-21T19:57:36.040Z] 2024-06-21 19:57:34,595 - INFO - Elapsed time 340.03 s
[task 2024-06-21T19:57:37.388Z] 2024-06-21 19:57:35,949 - INFO - Elapsed time 340.50 s
[task 2024-06-21T19:57:40.405Z] 2024-06-21 19:57:39,038 - INFO - Elapsed time 341.95 s
[task 2024-06-21T19:58:42.107Z] 2024-06-21 19:58:39,230 - INFO - Elapsed time 350.04 s
[task 2024-06-21T22:15:59.086Z] 2024-06-21 22:15:55,450 - INFO - Elapsed time 8285.26 s
[task 2024-06-21T22:20:15.833Z] 2024-06-21 22:20:12,330 - INFO - Elapsed time 8541.78 s
[task 2024-06-21T22:20:15.870Z] 2024-06-21 22:16:05,218 - INFO - Elapsed time 8292.14 s
[task 2024-06-21T22:20:15.887Z] 2024-06-21 22:17:56,034 - INFO - Elapsed time 8329.20 s
[task 2024-06-22T00:34:20.137Z] 2024-06-22 00:34:16,682 - INFO - Elapsed time 8272.99 s
[task 2024-06-22T00:34:20.156Z] 2024-06-22 00:34:02,481 - INFO - Elapsed time 8252.49 s
[task 2024-06-22T00:39:10.470Z] 2024-06-22 00:39:06,897 - INFO - Elapsed time 8442.46 s
[task 2024-06-22T00:40:34.463Z] 2024-06-22 00:40:31,056 - INFO - Elapsed time 8390.19 s
[task 2024-06-22T02:53:05.765Z] 2024-06-22 02:53:02,190 - INFO - Elapsed time 8312.60 s
[task 2024-06-27T08:19:40.563Z] 2024-06-27 08:19:37,248 - INFO - Elapsed time 9370.05 s
[task 2024-06-27T08:22:52.818Z] 2024-06-27 08:22:50,177 - INFO - Elapsed time 9465.90 s
[task 2024-06-27T08:27:27.043Z] 2024-06-27 08:27:25,328 - INFO - Elapsed time 8880.36 s
[task 2024-06-27T08:27:27.052Z] 2024-06-27 08:24:29,662 - INFO - Elapsed time 441.93 s
[task 2024-06-27T08:27:27.062Z] 2024-06-27 08:26:56,056 - INFO - Elapsed time 413.68 s
[task 2024-06-27T08:29:34.585Z] 2024-06-27 08:29:33,350 - INFO - Elapsed time 383.57 s
[task 2024-06-27T08:31:08.842Z] 2024-06-27 08:31:07,594 - INFO - Elapsed time 379.11 s
eu9ene commented 3 months ago

It was completed in 2 days after a restart. Also, we'll have #722. Closing