If a training job job failed MLFlow will not be reported and MLFlow shows job still running

helloworld1 commented 4 months ago

System Info

transformers version: 4.40.0
Platform: Linux-6.8.6-arch1-1-x86_64-with-glibc2.39
Python version: 3.11.7
Huggingface_hub version: 0.20.3
Safetensors version: 0.4.2
Accelerate version: 0.27.0
PyTorch version (GP?): 2.2.2+rocm5.7 (True)
Jax version: not installed
JaxLib version: not installed
Using GPU in script?: True
Using distributed or parallel set-up in script?: <False

Who can help?

No response

Information

[ ] The official example scripts
[X] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction

Setup MLFlow integration correctly,
Run a job.
The job failed due to OOM error.
Go to MLFlow UI and the job experiment shows status "Running"

Expected behavior

MLFlow callback should report the job as failure and call end_run() instead of keeping "Running "status.

amyeroberts commented 4 months ago

cc @muellerzr @pacman100

amyeroberts commented 3 months ago

Gentle ping @pacman100 @muellerzr

amyeroberts commented 2 months ago

cc @muellerzr @SunMarc

github-actions[bot] commented 3 weeks ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

helloworld1 commented 3 weeks ago

The issue is still not resolved

amyeroberts commented 3 weeks ago

Adding a Good Second Issue for anyone who would like to tackle this in the community

huggingface / transformers