search

ESPRI-Mod / synda

ESGF Downloader (this is a deprecated repository, the tool has now moved to https://github.com/ESGF/esgf-download)
https://espri-mod.github.io/synda/
21 stars 11 forks source link

Restart failed url and survive duplicate files #171

Open painter1 opened 3 years ago

painter1 commented 3 years ago

There are two changes here, submitted together because they both involve sdsqlutils.py.

  1. Upon a retry, the failed_url table should be cleared for the retried files. Otherwise they can't be properly retried.
  2. On rare occasions I have seen Synda crash due to duplicate database entries for the same file or similar problems which aren't supposed to happen. This change deals with them by logging an error message, changing a filename, and assigning a special status. Clearing up the problem has to be done manually, but with this change it is not urgent.