Error running convert_dance_diffusion_to_diffusers.py

[Skquark]

Describe the bug

When I try to convert a Dance Diffusion .ckpt file to use with my Diffusers implementation, I'm running into this error:

File "./convert_dance_diffusion_to_diffusers.py", line 10, in <module>
from audio_diffusion.models import DiffusionAttnUnet1D
ModuleNotFoundError: No module named 'audio_diffusion'

I dug into the source and couldn't find DiffusionAttnUnet1D or audio_diffusion.models either. Also noticed on line 12 there, it has from diffusion import sampling which looks like a typo of from diffusers, but I can't say for sure... Hopefully an easy fix and not something on my end.. Appreciate any assistance.

Reproduction

!python3 /content/diffusers/scripts/convert_dance_diffusion_to_diffusers.py --model_path /content/dancediffusion_models/jumango-ambient-v1.ckpt --checkpoint_path /content/dancediffusion_models/jumango-ambient

Logs

No response

System Info

Latest on Colab

[patrickvonplaten]

Hey @Skquark,

In order to do:

from audio_diffusion.models import DiffusionAttnUnet1D

You need to git clone the package https://github.com/Harmonai-org/sample-generator and make sure your Python path finds the folder: https://github.com/Harmonai-org/sample-generator/tree/main/audio_diffusion locally.

[Skquark]

Thanks, thought I did have the sample-generator loaded and installed right, but figured out after cloning I was trying to install it from the folder /sample_generator instead of /sample-generator with dash. So now I got past that error and ran into the next error that I suspected I'd get:

File "/content/diffusers/scripts/convert_dance_diffusion_to_diffusers.py", line 12, in <module>
from diffusion import sampling
ModuleNotFoundError: No module named 'diffusion'

I mentioned it looks like diffusion is a typo of diffusers... The more I dug into it, I see sampling comes from v-diffusion-pytorch. I now noticed in the example Colab to train Dance-Diffusion that it's cloning and importing sample-generator and v-diffusion-pytorch there... I thought the scripts were more self-contained in Diffusers library, so added that and got past that error. On to the next, ran the convert script on the ckpt file and got this error:

File "/content/diffusers/scripts/convert_dance_diffusion_to_diffusers.py", line 339, in <module>
main(args)
File "/content/diffusers/scripts/convert_dance_diffusion_to_diffusers.py", line 262, in main
sample_rate = MODELS_MAP[model_name]["sample_rate"]
KeyError: 'jumango-ambient-v1'

I'm passing the file jumango-ambient-v1.ckpt to the --model_path, and it looks like on that line it's trying to use the custom model name, which is not in the MODELS_MAP dict, to get the sample rate & size. Seems like there should be added arguments to pass to the script for --sample_rate and --sample_size to get past that error when trying to convert a model that's not on the official short list. I was hoping it'd auto-detect those values since I don't have that info on all the checkpoints I was planning on loading, but I'll have to figure it out. I may add those arguments to my fork and see if I can get past that error.. Thanks. Thanks.

[Skquark]

I went ahead and modified the script in my fork to get past that error.. Added these args to the bottom:

    parser.add_argument("--sample_rate", default=48000, type=int, required=False, help="Sample Rate of the custom checkpoint.")
    parser.add_argument("--sample_size", default=65536, type=int, required=False, help="Sample Size of the custom checkpoint.")

Then on line 262, changed to this:

    if model_name in MODELS_MAP:
        sample_rate = MODELS_MAP[model_name]["sample_rate"]
        sample_size = MODELS_MAP[model_name]["sample_size"]
    else:
        sample_rate = args.sample_rate
        sample_size = args.sample_size

Simple enough.. Feel free to update that. So now I got past that error, and on to the next issue converting:

File "/content/diffusers/scripts/convert_dance_diffusion_to_diffusers.py", line 345, in <module>
main(args)
File "/content/diffusers/scripts/convert_dance_diffusion_to_diffusers.py", line 278, in main
orig_model.load_state_dict(torch.load(args.model_path, map_location=device)["state_dict"])
File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 795, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 1002, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, '<'.

I'm a little confused by that one... What to do when in a pickle? Tested with a few different .ckpt files from the Harmonai Discord and got the same. Any suggestions?

[patrickvonplaten]

Imm could it be that the file saved in safetensors format instead: https://github.com/huggingface/safetensors

?

[Skquark]

They weren't safetensors, I was running the convert_dance_diffusion script on assorted *.ckpt files that I got from the Harmonai Discord in the model-sharing room, which are all ckpt files, probably created with the Finetune_Dance_Diffusion.ipynb Colab. Can't find any other shared models on HF made for the Diffusers format other than the few offical Harmonai models. Hard for me to say what that last error is, but seemed like I was doing everything right within my UI implementation..

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.