Pull File from S3 Bucket when cURL Fails

RTXteam / RTX-KG2

Build system for the RTX-KG2 biomedical knowledge graph, part of the ARAX reasoning system (https://github.com/RTXTeam/RTX)

MIT License

38 stars 8 forks source link

Pull File from S3 Bucket when cURL Fails #409

Open ecwood opened 2 months ago

ecwood commented 2 months ago

While working on #408, several source links have failed to download. This is really disruptive to the build process. Instead, we should ensure that every source has a backup copy in the S3 bucket so we can download from that when the cURL fails. This might be a tricky task though.