search

nils-braun / b2luigi

Task scheduling and batch running for basf2 jobs made simple
GNU General Public License v3.0
17 stars 11 forks source link

Replacing `gb2_ds_get` output parsing by `--failed_lfns` option to get file of failed LFNs #130

Closed meliache closed 3 years ago

meliache commented 3 years ago

When download a grid dataset with gb2_ds_get, gbasf2 lists the failed files (LFN) that failed to download. @ArturAkh implemented a feature that we parse the output to save those failed lfns in a text-file. When gbasf2 retries the download, it only tries to download those files from this text file.

Since a recent release, gb2_ds_get has the --failed_lfns option to generate a text file of failed LFNs. I think it would be good to just use that to avoid string-parsing, which seems to work for now, but breaks easily when the output format changes between releases. Sadly it's not easy to reproduce failed LFNs during a download, so I'm not sure how that file would look, but I expect it to be one LFN per line. Is this file even created when no downloads failed? I guess we should code the solution to be prepared for both possibilities of an empty file and a non-existing file.

If @ArturAkh wants to help I'd be happy, but otherwise I might also give this a look in the future

ArturAkh commented 3 years ago

@meliache Thanks for noticing that! Yes, I've also thought about that switch to the option, but haven't had time yet to think about a proper implementation. Maybe next week I would have some time to do so.

meliache commented 3 years ago

Noticed this in the release notes but decided to create an issue to not forget this, but if the current code works then this is not urgent. Actually I thought that maybe you were involved in requesting that feature? After all, the feature in gbasf2 appeared shortly after we implemented supporting for only downloading failed LFNs.

ArturAkh commented 3 years ago

Yep, I've asked about that ;) https://agira.desy.de/browse/BIIDCD-1420 maybe this is partly because of that issue

ArturAkh commented 3 years ago

@meliache a corresponding pull request is created: #132