execute . ./train_uspec.sh show AttributeError: 'AttrDict' object has no attribute 'multitask'

[snowmint]

I set up all environment then try to execute the command . ./train_uspec.sh show the error below:

conditioned-u-net/code$ . ./train_uspec.sh
INFO - unet - Running command 'main'
INFO - unet - Started run with ID "2"
ERROR - unet - Failed after 0:00:00!
Traceback (most recent calls WITHOUT Sacred internals):
  File "train_uspec.py", line 27, in main
    multitask=exp_config.multitask, conditioning=exp_config.conditioning,
AttributeError: 'AttrDict' object has no attribute 'multitask'

Then I check the code from utils.attr_dict import AttrDict but can't find the multitask config. Have any advice how to solve this problem? Thank you for reading my question.

[Veleslavia]

Hello, you need to use a config file where these parameters are specified. Examples of configuration files can be found here https://drive.google.com/drive/folders/1Hv9JwPuFIZ4SMlp0jc2OB3PKctcBCdUw for each experiment described in the paper.

Kind regards, Olga

[snowmint]

I add the multitask = False in config.py And change the runs to checkpoints in action_pipeline.py Then get the error below:

conditioned-u-net/code$ . ./train_uspec.sh
INFO - unet - Running command 'main'
INFO - unet - Started run with ID "5"
Loaded source: Bassoon
Loaded source: Cello
Loaded source: Clarinet
Loaded source: DoubleBass
Loaded source: Flute
Loaded source: Horn
Loaded source: Oboe
Loaded source: Saxophone
Loaded source: Trombone
Loaded source: Trumpet
Loaded source: Tuba
Loaded source: Viola
Loaded source: Violin
Loaded source: Bassoon
Loaded source: Cello
Loaded source: Clarinet
Loaded source: DoubleBass
Loaded source: Flute
Loaded source: Horn
Loaded source: Oboe
Loaded source: Saxophone
Loaded source: Trombone
Loaded source: Trumpet
Loaded source: Tuba
Loaded source: Viola
Loaded source: Violin
INFO - train_logger - Starting epoch 0/1000.
/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/sacred/stdout_capturing.py:184: UserWarning: tee_stdout.wait timeout. Forcibly terminating.
  warnings.warn("tee_stdout.wait timeout. Forcibly terminating.")
/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/sacred/stdout_capturing.py:190: UserWarning: tee_stderr.wait timeout. Forcibly terminating.
  warnings.warn("tee_stderr.wait timeout. Forcibly terminating.")
ERROR - unet - Failed after 0:00:03!
Traceback (most recent calls WITHOUT Sacred internals):
  File "train_uspec.py", line 45, in main
    pipeline.train_model()
  File "/home/mtl/Desktop/Video_Focusing_Music_Separation/conditioned-u-net/code/action_pipeline.py", line 267, in train_model
    self.run_phase(epoch)
  File "/home/mtl/Desktop/Video_Focusing_Music_Separation/conditioned-u-net/code/action_pipeline.py", line 214, in run_phase
    for num_it, (inputs, gt_data, *aux_data) in enumerate(self.loader):
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1333, in _next_data
    return self._process_data(data)
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
    data.reraise()
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/_utils.py", line 543, in reraise
    raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 58, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/mtl/Desktop/Video_Focusing_Music_Separation/conditioned-u-net/code/datasets/solos.py", line 109, in __getitem__
    sample_idx = random.randint(0, len(self.data[source_name])-1)
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/random.py", line 222, in randint
    return self.randrange(a, b+1)
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/random.py", line 200, in randrange
    raise ValueError("empty range for randrange() (%d,%d, %d)" % (istart, istop, width))
ValueError: empty range for randrange() (0,0, 0)

[Veleslavia]

It seems that the dataset that you're using is empty. Please, make sure that you use one of the configs that I mentioned above and specify the folder where the data is located. You may need to pre-process the data before training the model.

[snowmint]

I have download the training data Solos raw25, and set the path in train_uspec.sh: CUDA_VISIBLE_DEVICES=0 python train_uspec.py with exp_config.dataset_dir='./Solos_raw25' I'm curious about how to pre-process the dataset? It seems didn't mention in README.md.

[Veleslavia]

Right. I'm not sure in which format the data come from but you surely need to 1) Extract audio from video data - per folder, I believe 2) Downsample the data to 11025 Hz

[snowmint]

Thank you for your quick reply, I'll try to do that first!

[snowmint]

I have pre-processed all the folders, and change the config.py dataset_dir to my path dataset_dir='./Solos_raw25/videos', But still get the same error message:

conditioned-u-net/code$ . ./train_uspec.sh
INFO - unet - Running command 'main'
INFO - unet - Started run with ID "8"
Loaded source: Bassoon
Loaded source: Cello
Loaded source: Clarinet
Loaded source: DoubleBass
Loaded source: Flute
Loaded source: Horn
Loaded source: Oboe
Loaded source: Saxophone
Loaded source: Trombone
Loaded source: Trumpet
Loaded source: Tuba
Loaded source: Viola
Loaded source: Violin
Loaded source: Bassoon
Loaded source: Cello
Loaded source: Clarinet
Loaded source: DoubleBass
Loaded source: Flute
Loaded source: Horn
Loaded source: Oboe
Loaded source: Saxophone
Loaded source: Trombone
Loaded source: Trumpet
Loaded source: Tuba
Loaded source: Viola
Loaded source: Violin
INFO - train_logger - Starting epoch 0/1000.
/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/sacred/stdout_capturing.py:184: UserWarning: tee_stdout.wait timeout. Forcibly terminating.
  warnings.warn("tee_stdout.wait timeout. Forcibly terminating.")
/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/sacred/stdout_capturing.py:190: UserWarning: tee_stderr.wait timeout. Forcibly terminating.
  warnings.warn("tee_stderr.wait timeout. Forcibly terminating.")
ERROR - unet - Failed after 0:00:03!
Traceback (most recent calls WITHOUT Sacred internals):
  File "train_uspec.py", line 45, in main
    pipeline.train_model()
  File "/home/mtl/Desktop/Video_Focusing_Music_Separation/conditioned-u-net/code/action_pipeline.py", line 267, in train_model
    self.run_phase(epoch)
  File "/home/mtl/Desktop/Video_Focusing_Music_Separation/conditioned-u-net/code/action_pipeline.py", line 214, in run_phase
    for num_it, (inputs, gt_data, *aux_data) in enumerate(self.loader):
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1333, in _next_data
    return self._process_data(data)
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
    data.reraise()
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/_utils.py", line 543, in reraise
    raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 58, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/mtl/Desktop/Video_Focusing_Music_Separation/conditioned-u-net/code/datasets/solos.py", line 109, in __getitem__
    sample_idx = random.randint(0, len(self.data[source_name])-1)
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/random.py", line 222, in randint
    return self.randrange(a, b+1)
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/random.py", line 200, in randrange
    raise ValueError("empty range for randrange() (%d,%d, %d)" % (istart, istop, width))
ValueError: empty range for randrange() (0,0, 0)

[Veleslavia]

Hi! Folders should have the same names as specified in the Dataset class here https://github.com/Veleslavia/conditioned-u-net/blob/master/code/datasets/solos.py

Could you please check (when running your code) that the metadata and the data is loaded correctly? I would advise to put a breakpoint here: https://github.com/Veleslavia/conditioned-u-net/blob/master/code/datasets/solos.py#L57 and check self.meta and self.data for the dataset before running training. You may also need to disable multiprocessing to being able to do debug.

[snowmint]

Seems my dataset didn't have complete data, let me check about my dataset. I'll try to debug as your advice, thank you very much!

[snowmint]

I re-download the Solos dataset and pre-processed all the audio file into 11025Hz (Only 639 file due to download link been delete or setting to private):

Then try to train the code, it shows error below:

conditioned-u-net/code$ python train_uspec.py 
INFO - unet - Running command 'main'
INFO - unet - Started run with ID "13"
Loaded source: Bassoon
Loaded source: Cello
Loaded source: Clarinet
Loaded source: DoubleBass
Loaded source: Flute
Loaded source: Horn
Loaded source: Oboe
Loaded source: Saxophone
Loaded source: Trombone
Loaded source: Trumpet
Loaded source: Tuba
Loaded source: Viola
Loaded source: Violin
Loaded source: Bassoon
Loaded source: Cello
Loaded source: Clarinet
Loaded source: DoubleBass
Loaded source: Flute
Loaded source: Horn
Loaded source: Oboe
Loaded source: Saxophone
Loaded source: Trombone
Loaded source: Trumpet
Loaded source: Tuba
Loaded source: Viola
Loaded source: Violin
INFO - train_logger - Starting epoch 0/1000.
/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/sacred/stdout_capturing.py:184: UserWarning: tee_stdout.wait timeout. Forcibly terminating.
  warnings.warn("tee_stdout.wait timeout. Forcibly terminating.")
/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/sacred/stdout_capturing.py:190: UserWarning: tee_stderr.wait timeout. Forcibly terminating.
  warnings.warn("tee_stderr.wait timeout. Forcibly terminating.")
ERROR - unet - Failed after 0:00:23!
Traceback (most recent calls WITHOUT Sacred internals):
  File "train_uspec.py", line 45, in main
    pipeline.train_model()
  File "/home/mtl/Desktop/Video_Focusing_Music_Separation/conditioned-u-net/code/action_pipeline.py", line 267, in train_model
    self.run_phase(epoch)
  File "/home/mtl/Desktop/Video_Focusing_Music_Separation/conditioned-u-net/code/action_pipeline.py", line 214, in run_phase
    for num_it, (inputs, gt_data, *aux_data) in enumerate(self.loader):
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1333, in _next_data
    return self._process_data(data)
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
    data.reraise()
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/_utils.py", line 543, in reraise
    raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 61, in fetch
    return self.collate_fn(data)
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 265, in default_collate
    return collate(batch, collate_fn_map=default_collate_fn_map)
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 143, in collate
    return [collate(samples, collate_fn_map=collate_fn_map) for samples in transposed]  # Backwards compatibility.
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 143, in <listcomp>
    return [collate(samples, collate_fn_map=collate_fn_map) for samples in transposed]  # Backwards compatibility.
  File "/home/mtl/anaconda3/envs/conditioned_unet/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 139, in collate
    raise RuntimeError('each element in list of batch should be of equal size')
RuntimeError: each element in list of batch should be of equal size

[snowmint]

I am able to do training with batch_size set to 1, but the loss doesn't converge. However, when I set the batch_size > 1, I get this RuntimeError: each element in list of batch should be of equal size error. Could you provide any guidance or clue to what causes this error and how to resolve it?

[snowmint]

Hi, I am able to run the training code with batch_size set to 16 (due to hardware constraint) after I commented out the returned item "aux_source_indices" in the "solos.py" file, and eventually get the checkpoint audio files. However, I opened these audio files but could not hear anything. In addition, could you provide the codes for the inference part for the purpose of testing the trained model?

[Veleslavia]

Hello,

Unfortunately, I don't recall checkpointing audio files, I would rather checkpoint the model, or store some masks and/or masked spectrograms during the validation after each epoch (most likely to tensorboard). I don't think that you would be able to hear anything from the masks.

Once you have trained a model (or you can use a pretrained one as they are shared), you need to use code/test_uspec.py file for inference.

Kind regards, Olga

[snowmint]

Thank you for your reply!

I tried to run the "test_uspec.py" file using the checkpoint files that I trained, and got the errors shown in the screenshot below:

For testing and debugging, I ran the same “tes_uspec.py” file with your pre-trained model and the config.json file from the "Exp. 10" folder using the same command as above: CUDA_VISIBLE_DEVICES=0 python test_uspec.py with experiments_meta/10/config.json exp_config.model_checkpoint='00010/CP0990.pth'

The result produced at the end seems more like an evaluation of a Solos dataset rather than a prediction. Is this understanding correct?

What I intend to do is to input a piece of music to run the inference, and to get a prediction for 13 instruments whether or not presented in the music. Could you kindly provide subsequent help on how to do this?

[snowmint]

It seems that I have to add the "if-statement" shown below to the "test_uspec.py" file to be able to run the test on sepc_unet, but it only works on the Solos dataset.

    if exp_config.model == 'specunet':
        model = Unet(use_dropout=exp_config.use_dropout, complementary=exp_config.complementary,
                     conditioning=exp_config.conditioning)
    elif exp_config.model == 'mhunet':
        model = MHUnet(use_dropout=exp_config.use_dropout, complementary=exp_config.complementary,
                   multitask=exp_config.multitask, conditioning=exp_config.conditioning,
                   use_bias=exp_config.use_bias, n_decoders=exp_config.num_decoders,
                   num_downs=exp_config.num_downblocks)

I tried to run the command CUDA_VISIBLE_DEVICES=0 python test_uspec.py with experiments_meta/10/config.json exp_config.model_checkpoint='00010/CP0990.pth' exp_config.dataset='urmp' exp_config.dataset_dir='./Datasets/urmp', but it did not seem to read the audio file and run the inference...

URMP dataset seems need to do some pre-process, could you give a hint?

[Veleslavia]

Hi! I think you're right, and apparently I don't have access to the pre-processed dataset anymore. I added you to the dev repo for this project (please, accept the invite), you can find a corresponding data processor there: https://github.com/Veleslavia/vimss_torch_dev/blob/master/datasets/urmp_dp.py

From what I remember, you need _process_dataset function from there.

[Veleslavia]

What I intend to do is to input a piece of music to run the inference, and to get a prediction for 13 instruments whether or not presented in the music. Could you kindly provide subsequent help on how to do this?

I think by design the scripts are expecting to have a ground truth to compute the metrics etc. One option is to prepare a compatible dumb ground truth masks, and then save intermediate predictions for all instruments.

[snowmint]

Thank you for your reply! I spent a week trying to figure out how to use the "urmp_dp.py" file to perform data pre-processing. I run the code using the command python urmp_dp.py parallel_data_processor --raw_data_dir=./Datasets/urmp --output_dir=./Datasets/urmpv2numpy, but I am still getting errors despite already resolved quite a few by modifying the code.

One of the errors I am having trouble to resolve is shown in the image below. The error message reads "all input arrays must have the same shape". I can't figured out why and which lines of the code is causing the error. Could it be the np.stack() or pool.map()?

I tried to print out the shape of some data, hope it will help to explain where I am stuck.