wandb can't exit when training process exit accidentally

[suixin1424]

System Info

- `Accelerate` version: 0.33.0
- Platform: Linux-4.15.0-76-generic-x86_64-with-glibc2.27
- `accelerate` bash location: /home/zhuyiming/anaconda3/envs/mix/bin/accelerate
- Python version: 3.10.0
- Numpy version: 1.24.3
- PyTorch version (GPU?): 1.12.1 (True)
- PyTorch XPU available: False
- PyTorch NPU available: False
- PyTorch MLU available: False
- PyTorch MUSA available: False
- System RAM: 503.57 GB
- GPU type: NVIDIA GeForce RTX 3090
- `Accelerate` default config:
        - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: no
        - use_cpu: False
        - debug: False
        - num_processes: 8
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - enable_cpu_affinity: False
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []

Information

• [ ] The official example scripts
• [X] My own modified scripts

Tasks

• [ ] One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
• [X] My own task or dataset (give details below)

Reproduction

1. starting a training process log with wandb
2. when the process exit accidentally caused by oom, we can see wandb print output continuously

Expected behavior

wandb can't exit

[hkproj]

+1

[muellerzr]

Not entirely sure that's something we can do and it's more a wandb issue.