search

web3-storage / backup

A tool to backup all data for disaster recovery.
0 stars 0 forks source link

fix: fetch entire input file before starting processing #16

Closed olizilla closed 1 year ago

olizilla commented 1 year ago

we're seeing backup processes fail where fetching the input url fail mid stream.

this PR tweaks it to fetch the entire data file to disk at the start and then stream them from the local disk.

see: https://github.com/web3-storage/backup/issues/11

of note p-retry won't retry for network errors! Hence the surprising manual retry loop.

It does not retry on most TypeError's, with the exception of network errors. This is done on a best case basis – https://github.com/sindresorhus/p-retry#api

License: MIT

olizilla commented 1 year ago

Gonna merge and deploy. plz to eyeball at your convenience