nils-braun / b2luigi

Task scheduling and batch running for basf2 jobs made simple
GNU General Public License v3.0
17 stars 11 forks source link

Issue with large projects separated into subfolders #147

Closed Bilokin closed 2 years ago

Bilokin commented 2 years ago

Hello,

thanks for adding the support for multiple sub-directories in the grid projects, i.e. that have multiple sub* folders in their output, but unfortunately, I think there is a bug linked to it: When b2luigi moves the downloaded output from temporary folders of sub00, sub01, subXX, to the output folder, let's call it result/, the contents of sub00 folder is not in the result/, but I suspect that it is in the result/sub00 folder. I tried to run the this code in the python interpreter using sub00 and sub01 input folders and not existing output folder ./result and I get the following tree:

./result
    sub01_content.root
    sub00/
         sub00_content.root

This is not expected by the tasks downstream and I would prefer that the output folder would contain all files from all sub* folders directly. There shouldn't be any conflicts in the file names, but one has to check for that anyway.

meliache commented 2 years ago

Tasks copied over from the duplicate issue #148:

meliache commented 2 years ago

I now created a draft PR #150 where I changed the moving function to move ROOT files individually into the final output directory, with some more checks in place so it should work, but I still haven't tested it yet, which I will do now.