NickleDave
commented
9 months ago Hi @athenasyarifa, I'm sorry that vak is crashing here.
Thank you for the detailed error report. Based on what I see in the console output and the config file you provided, I don't see a reason to think this is a beginner issue.
My guess is that it's just the case that one of the spectrograms ends up being too big to fit on the GPU, and that causes the OOM error.
The quickest way to rule this out might be to make the files smaller and see if we still get the crash. I know you said you tried that already. Just to follow up:
- [ ] Did you happen to work through this tutorial and if so were you able to run predict on all the files in that dataset? That at least tells us you can predict on some files, and also gives us a lower bound on the largest size file you can predict on.
- [ ] In the console output you provided, it looks like the crash happened on file 5. What we want to figure out is if we can run predict if you remove that file and all files larger than it. Here is how you might do that.
- [ ] First, make a copy of the directory representing the dataset prepared by vak for predicting. It's easy enough to regenerate this, but let's copy just in case we need the original to troubleshoot more.
- [ ] Then run the following script, with the constant
DATASET_DIRat the top changed to point to the copy you make. What it does is drop all the files from the dataset csv that have a duration >= the file that causes the crash. Since they are no longer in the csv, vak won't use them when you runvak predict-- it iterates through the rows of that csv
DATASET_DIR = pathlib.Path(
'tests/data_for_tests/generated/prep/predict/audio_cbin_annot_notmat/TweetyNet/032412-vak-frame-classification-dataset-generated-231010_165729/'
)
FILE_NUMBER_THAT_CAUSES_CRASH = 5
metadata = vak.datasets.frame_classification.Metadata.from_dataset_path(DATASET_DIR)
dataset_csv_path = DATASET_DIR / metadata.dataset_csv_filename
df = pd.read_csv(dataset_csv_path)
max_dur = df.loc[FILE_NUMBER_THAT_CAUSES_CRASH, 'duration']
new_df = df[df.duration < max_dur]
assert len(new_df) < len(df) # make sure that worked
assert new_df['duration'].max() < max_dur # make extra extra sure, because we are scientists- [ ] Finally, re-run, after changing
dataset_pathin your config file to point to the copy of the dataset where you changed the csv itself with the script, and then see if you still get an OOM error
If you don't get the OOM error then, we can confirm that the issue is file size.
Assuming that is the problem, there's a couple things we could try:
- use different spectrogram parameters to make the spectrogram smaller, e.g. by setting limits on the frequencies using the
freq_cutoffsoption: https://vak.readthedocs.io/en/latest/reference/config.html#vak.config.spect_params.SpectParamsConfig.freq_cutoffs
so if you knew that your sound of interest is always between 500-12000 Hz you could do
[SPECT_PARAMS]
fft_size = 512
step_size = 64
freq_cutoffs = [500, 12000]- for more detail (perhaps too much detail) see reply here: https://forum.vocalpy.org/t/specifying-the-frequency-range-using-vak/67/2?u=nicholdav
- make clips of the audio, e.g. in Raven or with a Python script -- I can help with that if we need to
I wish we had better numbers on the durations / memory use already to give you -- I did some quick tests to give you a ballpark:
- I just checked the durations of the files from the canary dataset for the TweetyNet paper, looks like the longest one is 39 seconds.
- I am realizing we did not do a great job of documenting the actual hardware we used but I think it would have been a GPU with ~10 GB memory, possibly an RTX 3080?
Looks like your GPU has 2 GB according to the error from pytorch?
Do you see the same thing if you run nvidia-smi in the terminal?
Again, really sorry we don't have a more general solution ready, we know this is an issue -- see for example #514. Ideally we'd estimate based on a user's hardware what we can fit on the GPU and then work with that.
In the meantime I'm happy to work with you to find a workaround.
Please let me know how what you figure out about the file size, I can answer more questions too if needed
athenasyarifa
Hi @NickleDave and everyone, It's me again, sorry. I ran into a
torch.cuda.OutOfMemoryErrorwhen I was runningvak predicton my dataset. Please find below the error message:and my
predict.tomlfile looks like this:I saw similar issue #301 or is it a different case? I tried removing longer files from my predict dataset, but it was still not working. My predict dataset are 27 wav files from 7 to 52 seconds. Can't help thinking this is probably a beginner issue! How can I fix this? Thanks in advance for your help!
Best, Rifa