search

kallsyms / wx_explore

Website that allows everyone to dive into the raw data behind weather predictions
https://www.vortexweather.io/
Apache License 2.0
11 stars 1 forks source link

be defensive about corrupt (incomplete) files #12

Closed kallsyms closed 4 years ago

kallsyms commented 4 years ago

seeing occasional errors from merge numpy.reshape that the contents aren't a multiple of the dimensions which means the files got truncated somehow. any files that are like this should be removed entirely.

kallsyms commented 4 years ago

each merge thread should try/catch the load/reshape and fill with zeros on failure. the func should return a list of each band that failed, and the outer map should gather all of those and remove them from the db. since the offsets will then not be referenced anywhere, the data will be unused (so zeroing is fine) and will be purged on the next merge of the resulting file.

kallsyms commented 4 years ago

debugged this a bit and found that the reshape was actually failing because s3 responded with a 404. added retries around s3_get and s3_put which seems to have solved the problem.