nils-braun / b2luigi

Task scheduling and batch running for basf2 jobs made simple
GNU General Public License v3.0
17 stars 11 forks source link

Gbasf2: Fix moving of downloaded multi-sub datasets #150

Closed meliache closed 3 years ago

meliache commented 3 years ago

For datasets with multiple subs, when moving the contents of the temporary project download directory to the final ouptut dir, previously, there was an error which led to the file structure:

<result_dir>/B.root/job_name*B.root  # sub00 contents
<result_dir>/B.root/sub01/job_name*B.root
<result_dir>/B.root/sub02/job_name*B.root

This commit should fix this back to

<result_dir>/B.root/job_name*B.root  # contents of all subs

Further, since this moving function turns out to aggregate many lines of code, I moved it into an own helper function, which can be independently tested.

Draft so that I can link to it, but I haven't finished yet writing unit tests.

Resolves #147

meliache commented 3 years ago

Added some unittests on dummy directory strucutres and now this should be ready. The only problem I have currently is that when I run test-jobs on the grid I get download failures, but which according to the mailinglist seem to be caused by problems on the SE sites, so I can't test at the moment.