CAS-eResearch / TAOImport

TAO Data Conversion and Import Tools
GNU General Public License v3.0
2 stars 3 forks source link

Handle parallel case with small number of input files #20

Open manodeep opened 7 years ago

manodeep commented 7 years ago

Currently, the parallelization is done by distributing the input files over the MPI tasks. However, when the number of input files is less than the number of tasks, the code aborts. For instance, the full meraxes dataset is in one file and therefore can not be converted in parallel currently.

meraxes conversion also takes the longest per GiB and the parallel conversion would be a good thing