In our case, the pipeline runs on a storage that is significantly faster than the one where the interferograms are stored. When using a symlink to the files on that cold storage, the storage can be overwhelmed when 32 containers try to access thousands of interferograms in parallel.
One could add another config variable max_parallel_copying_processes, to only let 4 or 8 processes copy interferograms in parallel.
In our case, the pipeline runs on a storage that is significantly faster than the one where the interferograms are stored. When using a symlink to the files on that cold storage, the storage can be overwhelmed when 32 containers try to access thousands of interferograms in parallel.
One could add another config variable
max_parallel_copying_processes, to only let 4 or 8 processes copy interferograms in parallel.