search

IAHispano / Applio

A simple, high-quality voice conversion tool focused on ease of use and performance
https://applio.org
MIT License
1.82k stars 293 forks source link

[Bug]: Large-scale data preprocessing error #884

Open vmsystem opened 1 week ago

vmsystem commented 1 week ago

Project Version

3.2.5

Platform and OS Version

Windows 10 pro 64bit

Affected Devices

PC

Existing Issues

No response

What happened?

image When generating more than 3000 hours of data, the above error occurs. It is unable to process over 800,000 chunks of audio data.

Steps to reproduce

  1. wrote dataset URL
  2. Preprocessing started
  3. While processing normally, the above error message is displayed, making training impossible. ...

Expected behavior

It should have processed the data diligently? I guess?

Attachments

No response

Screenshots or Videos

No response

Additional Information

No response

AznamirWoW commented 1 week ago

what are you gonna do with 3000 hours of audio?

vmsystem commented 1 week ago

err.. I can let the training run and take a break. I’d like to take a break for about a month… Just kidding, I’m working on creating a pre-trained model.

AznamirWoW commented 1 week ago

err.. I can let the training run and take a break. I’d like to take a break for about a month… Just kidding, I’m working on creating a pre-trained model.

what step is actually throwing those warnings? preprocess does not use cuda

if it happens in extract features, then it is likely some audio cuts are too small, but for better training you should just cut all audios to exact 3 sec chunks.

vmsystem commented 1 week ago

I'm not sure but it seems to occur somewhere between 600,000 and 800,000. it doesn't seem to be an issue with the size of the chunks. once the error occurs, the same error keeps being output continuously.

AznamirWoW commented 1 week ago

I'm not sure but it seems to occur somewhere between 600,000 and 800,000. it doesn't seem to be an issue with the size of the chunks. once the error occurs, the same error keeps being output continuously.

you need to start from the start. What you got, what you're doing.

there's a difference in "I have one 100 hour long wav file i'm trying to cut" vs "I have 200000000 1 second wavs I want to load"

vmsystem commented 1 week ago

Linux users seem to overestimate windows users too much. lol trust me, if someone put 200 million 1 second files in a folder, their grandchild would be the one to check it when opening the folder. XD the dataset consists of 10-hour chunks of audio with silence removed.

vmsystem commented 1 week ago

I'm currently testing again after reducing the data volume. If the same issue occurs, I'll share all the data.

AznamirWoW commented 1 week ago

the dataset consists of 10-hour chunks of audio with silence removed.

Details please. Are they already pre-cut? Are there any too small?

vmsystem commented 1 week ago

The data in the folder consists of 10-hour chunks. (It means there are multiple 10-hour audio files.) If you're asking if the pre-cut means they were cut into 3-second pieces, then no. I used Applio's own tool to do the cutting.

AznamirWoW commented 1 week ago

The data in the folder consists of 10-hour chunks. (It means there are multiple 10-hour audio files.) If you're asking if the pre-cut means they were cut into 3-second pieces, then no. I used Applio's own tool to do the cutting.

try setting the number of cpus to 1 for preprocess

vmsystem commented 1 week ago

k, I'll try thanks.