According to the comment, the intention is to stop multiprocessing after loading the data into memory. However, setting num_workers = 1 results in a creation of an extra process later on when using the custom data loader.
Having the extra worker resulted in deadlocks when I worked with larger data.
According to the comment, the intention is to stop multiprocessing after loading the data into memory. However, setting num_workers = 1 results in a creation of an extra process later on when using the custom data loader.
Having the extra worker resulted in deadlocks when I worked with larger data.
Corrected by setting num_workers = 0.