When using laxydl with the aria2c backend, if the same URL is added to the download queue twice (eg, a user submits two jobs in close succession using the same input files that are not already cached), one of the downloads fails to compete an leaves laxydl in an infinite loop state with aria2c errors like:
ERROR: 2021-10-06 01:22:08,465 -- GID aba810727b6a6411 is not found
I suspect this is because the second time the URL is added to the aria2c download queue it is assigned a new GID and the old one is invalidated.
One solution would be to first query the aria2c download queue for the URL we are about to add (if possible). The other might be to abandon the aria2c backend an build relevant functionality into laxydl.
When using laxydl with the aria2c backend, if the same URL is added to the download queue twice (eg, a user submits two jobs in close succession using the same input files that are not already cached), one of the downloads fails to compete an leaves laxydl in an infinite loop state with aria2c errors like:
I suspect this is because the second time the URL is added to the aria2c download queue it is assigned a new GID and the old one is invalidated.
One solution would be to first query the aria2c download queue for the URL we are about to add (if possible). The other might be to abandon the aria2c backend an build relevant functionality into laxydl.