search

nils-braun / b2luigi

Task scheduling and batch running for basf2 jobs made simple
GNU General Public License v3.0
17 stars 11 forks source link

switch gb2_ds_get to use the --new flag #190

Closed MarcelHoh closed 1 year ago

MarcelHoh commented 1 year ago

Sorry for yet another PR.

I've been using the --new flag in gb2_ds_get in my local version of b2luigi for the last few months. It is significantly faster than the previous download method and I think it would be good to adopt this officially.

To use the flag gbasf2 v5r6 is required. This has now been out for 5+ months. There may be some holdouts but I think it is probably safe to enable the flag. This is of course up to you.

meliache commented 1 year ago

Nice to see improvements on that front! Better download handling on gbasf2 side would resolve many issues.

I'm just worried that there will be a new gbasf2 release in the future which might adopt the new download method as a default and deprecate --new parameter, requiring a new PR to remove it again. Though I hope the gbasf2 devs would then just ignore that parameter for backwards compatibility. Are you aware whether it's planned to be enabled by default in the future?

Also, does the new download method still allow for writing the list of files for which the download failed to a file? B2luigi has some code that repeats a failed download using the list of files from that file. The code was mainly introduced for performance reasons, and I hope with better downloads we might deprecate it.

MarcelHoh commented 1 year ago

I'm not aware of any definitive plans, like you I would expect that the new flag is not removed (at least not immediately) even if it is no longer necessary. I will ask Michel but he is currently on holiday so I will wait until he is back.

For the failed download list, I haven't encountered any issues with this in the 5 months I've used the new flag and b2luigi. But I'll also double check with Michel.