Open kenkalang opened 4 weeks ago
Hi, thank you for using my repository. I haven't tried Stable Audio 1.0 training, but I can help you find the reason. I believe while training the job should save logs at WandB. Could you share some logs of your training such as reconstruction samples, loss curve, and data std?
Btw, some possible issues would be:
Thanks for your response,
https://github.com/user-attachments/assets/38b2a1d9-ab18-43cf-9938-bc79e4fd8494
I noticed some irregularities in the training logs and was optimistic about achieving a good output by 100k steps. I followed the steps outlined in the repository, and everything worked fine with the text prompts, as I confirmed by logging them. I indeed suspect that the VAE might not be loading correctly. How can I verify if the VAE is being loaded properly?
Assuming that the VAEs of Stable Audio 1.0 and 2.0 are slightly different, when I tested the training of the SA 2.0 VAE, the data_std
tended to be around 0.9. Therefore, the logged data_std
value might be suspicious..?
In my training code, I believe reconstruction samples are save to WandB under the name Reconstruction (Pretransform)
as well as the above logs. Can you find them in the WandB log?
I deleted all my previous files since they weren't working anymore, so I retrain the VAE 2.0 following your README instructions and here is the recon pretransform
[original] https://github.com/user-attachments/assets/b5264eee-8d18-4d41-b87d-7f0c7195b51b
[recon]
https://github.com/user-attachments/assets/7a5eac79-f8cd-4ced-bafc-4912a3a9d219
. My question is, could using the DeepSpeed strategy affect the results? Someone mentioned that when they used DeepSpeed, the inference results were noisy, although the demo sounded fine. However, when they used DDP, it worked perfectly. Lastly, may I know the configuration such as batch size, etc and device that you used?
Sorry for the late reply. I believe DeepSpeed should not affect training results dractically. If DeepSpeed affected a lot, there might be a bug in checkpoint saving/loading..?
Unfortunately, since I have never tried DeepSpeed options of Lightning, I have no idea if a DeepSpeed implementation in stable-audio-tools has a problem.
Regarding the reconstruction sample you pasted, from my experience of VAE training, the quality should be able to be improved if the model is trained for longer time.
I've been trying to train diffusion model with stable audio 1.0 config, I also trained the autoencoder with the Stable Audio 1.0 VAE for 50k steps autoencoder result which I then used as the pre-transform ckpt. However, after training the diffusion model for 100k steps, it still results in noise diffusion result. The dataset contains 2,500 hours of songs. I used the same CLAP model as you recomended. i used deepspeed as my training strategy. Is there anything I might have missed in my training?