nils-braun / b2luigi

Task scheduling and batch running for basf2 jobs made simple
GNU General Public License v3.0
17 stars 11 forks source link

Allow for gbasf2 projects with multiple output `sub<xy>` directories #122

Closed meliache closed 3 years ago

meliache commented 3 years ago

Before that, the assumption was that all outputs are saved in a final sub00 directory. But in future releases, jobs with many outputs (> 1000) can have outputs in additional sub<XY> directories.

This means that when downloading, we will also get multiple sub<XY> directories in the temporary download directory. However, currently, the b2luigi user expects that all downloaded file names will be in a common directory. Therefore, after the completed download, we move the contents of all sub* directories into the final output directory.

We can then still do the file-comparison cross-check that the download was complete by replace sub<XY>`` with a wildcard when doing the remotegb2_ds_list` command.

So far I just tested that the wildcards work for g2_ds_get and gb2_ds_list, but before merging I will run a complete gbasf2 task with this branch and see that everything still works.

Resolves https://github.com/nils-braun/b2luigi/issues/80

codecov-commenter commented 3 years ago

Codecov Report

Merging #122 (c7a7573) into main (4aaf849) will decrease coverage by 0.16%. The diff coverage is 8.33%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #122      +/-   ##
==========================================
- Coverage   56.89%   56.73%   -0.17%     
==========================================
  Files          23       23              
  Lines        1494     1500       +6     
==========================================
+ Hits          850      851       +1     
- Misses        644      649       +5     
Impacted Files Coverage Δ
b2luigi/batch/processes/gbasf2.py 39.05% <8.33%> (-0.34%) :arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 4aaf849...c7a7573. Read the comment docs.