[TeleMelody] How to infer using pretained model

[gandolfxu]

Can you provide a detailed description?

I followed the README, but failed.

[EduardoPach]

@gandolfxu which error did you get?

[EduardoPach]

Overal Idea

To use telemelody to infer you'll have to run (I'll focus in infer to english):

python infer_en.py <lyric_checkpoint_directory> <melody_checkpoint_directory> <input_directory> <output_directory>

Where:

• : Is the name of the directory where lyric2rhytm checkpoint is saved
• : Is the name of the directory where template2melody checkpoint is saved
• : Name of the directory inside `data/en` where `lyrics.txt` `syllable.txt` and `chord.txt` are saved
• : Name of the directory where the midi file generated will be stored inside the results directory

Running Inference Example

Follow these steps:

• 1. Download the lyric2rhythm and template2melody checkpoints from the README
• 2. Inside the inference, directory create a new folder called checkpoints and save each model checkpoint to a specific folder and rename the checkpoints to checkpoint_best.py like this

     checkpoints
     ├── <lyric_checkpoint_directory>
     │   └── checkpoint_best.pt
     ├── <melody_checkpoint_directory>
     │   └── checkpoint_best.pt

• 3. Add the dictionaries to data-bin folder like this:

     data-bin
     ├── <lyric_checkpoint_directory>
     │   └── dict.beat.txt
     │   └── dict.lyric.txt
     ├── <melody_checkpoint_directory>
     │   └── dict.notes.txt
     │   └── dict.trend.txt

     dict.beat.txt and dict.lyric.txt are provided in the training/lyrich2rhythm/dict directory and the dict.notes.txt and dict.trend.txt can be generated through the gen_dictionary function. Just make sure that they are correctly named.

• 4. Now just run the test inference to make sure it's working. Assuming:
     • : lyric
     • : melody Do: `python infer_en.py lyric melody test example` This should create an example directory in the results folder containing the `.mid` file for the inference example.

Running Inference on Custom Data

I'm not entirely sure if I did this part 100% correctly (@jzq2000 can verify that), but to run inference on custom data you need to create a new folder in the data directory called <input_directory> containing the lyrics, syllable, and chord .txt. To do that I create two functions to handle lyrics and syllables.

from phonemizer import phonemize
from phonemizer.separator import Separator
from phonemizer.punctuation import Punctuation

def preprocess_lyrics(text: str) -> str:
    text = (" [sep] ".join([t for t in lyrics.split("\n") if t]) + " [sep]").strip()
    text = Punctuation(';:,.!"?()-').remove(text)
    return text

def make_syllables(text: str) -> str:
    text = [t.strip() for t in text.split("[sep]") if t]
    phn = phonemize(
        text,
        language='en-us',
        backend='festival',
        separator=Separator(phone='_', word=' ', syllable='@@'),
        strip=True,
        preserve_punctuation=True,
        njobs=4
    )
    return " [sep] ".join(phn) + " [sep]"

My input lyrics are in the format:

XXXXXX
XXXXXX
XXXXXX

YYYYYY
YYYYYY
YYYYYY

With that being said I used this function to create the lyrics and syllable files and copied the chord file from the inference example and I was able to generate a melody. So let's assume that <input_directory> = my_own_lyrics, then we just need to:

python infer_en.py lyric melody my_own_lyrics my_own_lyrics_melody

[gandolfxu]

I run successfully following this instruction. Thanks!

[jzq2000]

@EduardoPach Thank you for your contribution. Your instructions are correct. As an additional suggestion, you may want to consider replacing punctuations in the line with '[sep]' to improve rhythm quality, but ultimately the choice of whether to remove or replace them is up to you.

[TanMIMI]

error" AttributeError: 'NoneType' object has no attribute 'data' " occurs when trying to load checkpoint, and i find that the args load from model is None, does it mean there are some problem on the checkpoint?

[feiyuehchen]

@TanMIMI Did you solve the problem?

[Hollie7]

@TanMIMI Met the same problem!

[Hollie7]

error" AttributeError: 'NoneType' object has no attribute 'data' " occurs when trying to load checkpoint, and i find that the args load from model is None, does it mean there are some problem on the checkpoint?

Hi, I solved the above issue by referring to the solution in this link This is due to the problem of fairseq. We can solve it by using this specific version of fairseq: pip install soundfile git+git://github.com/pytorch/fairseq.git@b8ea8a9b72c82192da07e3377adf4ebbde16716d