openvpi / DiffSinger

An advanced singing voice synthesis system with high fidelity, expressiveness, controllability and flexibility based on DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism
Apache License 2.0
2.62k stars 275 forks source link

Multi-node batched validation and improvement on strategy selection #148

Closed hrukalive closed 8 months ago

hrukalive commented 8 months ago
  1. Multi-node and batched validation
    1. Binarizer changes:
      1. Split train/valid set now preserves the text prefix order specified in config.
      2. Record all ndarray's first dimension into metadata (pickle).
      3. Also print duration for each speaker separately.
      4. File handle is closed in load_meta_data.
    2. Task code changes:
      1. All devices participate in validation. And diffusion process accepts a batch of inputs.
      2. Custom DsTensorBoardLogger to support logging audio and images from multiple processes.
    3. Plot changes: Added figsize for curves and pitch plots. Added title ("spk - item name")
  2. Multi-device strategy selection: Much easier, does not poke PL internal logic but delegates to its registry. Using name, one could specify all available strategies, and pass other key-value pairs as kwargs. To disable NCCL P2P, now use a new option called nccl_p2p (defaults to true).
  3. Changes due to side-effects and future considerations:
    1. Dataset now admits size_key to specify other "sizes" from metadata for sorting samples.
    2. Dataset now admits preload to load all samples into memory (for TPU).
    3. DsBatchSampler is now unified as the only sampler. Several bug fixes on unwanted randomness, and correctly deals with an insufficient number of samples.
    4. Dataset get_item and collator return the indices of the item for metadata retrieval.
    5. Moved fine-tune related logic to build model.
    6. Removed dynamic validation loss creation in validation step, hoisted to the init function and conformed validation metrics to the same place.

Notes

  1. Even though DeepSpeed is now supported, the checkpoint produced is not in the usual format. Please use it at your own discretion.
  2. FSDP does not support gradient clipping by norm.
  3. This PR does require a re-binarization of the datasets.