Drop outdated flakey lightning test

iterative / dvclive

📈 Log and track ML metrics, parameters, models with Git and/or DVC

https://dvc.org/doc/dvclive

Apache License 2.0

161 stars 33 forks source link

Drop outdated flakey lightning test #816

Closed shcheklein closed 2 months ago

shcheklein commented 3 months ago

Caused by https://github.com/iterative/dvclive/actions/runs/8872661976/attempts/1

Decision to drop the test: https://github.com/iterative/dvclive/pull/816#issuecomment-2083668012

dberenbaum commented 3 months ago

Were you seeing tests fail locally? Was it the same one failing in the CI here?

shcheklein commented 3 months ago

Same as in this CI - it's flakey, probably since floats comparison is not reliable. It takes a few runs to hit it.

dberenbaum commented 3 months ago

It may be related to https://github.com/iterative/dvclive/pull/814. So far I can't reproduce locally though. As explained here, there is no guarantee now that all expected data calls are sent before the main process dies. I think this is better for users (no reason to make the training process hang for something like a studio server error) but annoying for tests.

dberenbaum commented 3 months ago

I think we should just drop this test. It covered special lightning logic that was removed in https://github.com/iterative/dvclive/pull/788. There's no reason we should need to specifically test lightning for this scenario now.