Malformed GZ Files in Planet Minute Replication

[Rub21]

We encountered an issue that hasn’t occurred in the past: malformed GZ files were found in the planet minute replication at Planet OpenHistoricalMap. This caused the tiler and Overpass to stop working and updating the data.

• https://github.com/OpenHistoricalMap/issues/issues/843
• https://github.com/OpenHistoricalMap/issues/issues/575#issuecomment-2229634072

Immediate Solution:

To temporarily solve the issue, I replaced the problematic file with an empty one. Below are the details:

• Malformed file: https://s3.amazonaws.com/planet.openhistoricalmap.org/replication/minute/001/480/848-error.osc.gz
• Replacement file: https://s3.amazonaws.com/planet.openhistoricalmap.org/replication/minute/001/480/848.osc.gz

Something important to highlight is the size of the malformed file is 12 MB, which is unusually large for a replication file.

cc. @batpad

[Rub21]

@batpad do you have some ideas on how to handle the case of a wrong GZ file? Currently, we are monitoring if a minute replication file exists or not in the values.production.template.yaml. We basically check the sequence of files using the S3 API, but in the case of checking for malformed GZ files, we may need to download and check that file,