When generating a waveform dataset with condor (see dingo_generate_dataset_dag), stdout is redirected to files in condor/output/. For each job, a separate output file is created. However, each job may contain several processes (e.g., when splitting waveform generation across nodes). The output of these separate processes is all directed to the same file, which causes output from earlier processes to be overwritten by later ones.
To fix this, we should specify the output file names to include the process numbers. I am not sure how to do this, however, since the pycondorJob constructor only takes the output directory name, not the file name.
When generating a waveform dataset with condor (see
dingo_generate_dataset_dag), stdout is redirected to files incondor/output/. For each job, a separate output file is created. However, each job may contain several processes (e.g., when splitting waveform generation across nodes). The output of these separate processes is all directed to the same file, which causes output from earlier processes to be overwritten by later ones.To fix this, we should specify the output file names to include the process numbers. I am not sure how to do this, however, since the
pycondorJobconstructor only takes the output directory name, not the file name.