NewFile SNS message for planet PBF files no longer working?

[mhowell-ims]

We have a lambda function that is triggered by the SNS NewFile message. We use it to trigger some processing on the latest planet-*.osm.pbf file. Several weeks ago that lambda function stopped receiving any messages for the planet PBF file. Now our lambda function only seems to receive messages for object keys like these:

• 2020/history-201207.osm.pbf.torrent
• 2020/history-201207.osm.pbf.torrent.md5
• 2020/history-201207.osm.pbf.md5.

What happened to the NewFile event for the actual planet PBF files, e.g. 2020/planet-201207.osm.pbf ?

[mojodna]

Nothing changed configuration-wise, so I'm not sure what might be up. Are you expecting that notifications occur for the PBFs occur within some window of the .md5 appearing? It's likely to be 4+ hours, or however long it takes for the PBF to mirror.

[mojodna]

I subscribed to the SNS topic (by email) in time to get notifications for planet-201228 and saw the correct number of notifications. HOWEVER, the notification for planet-201228.osm.pbf was delivered when the file started being copied to S3 (which matches the timestamp), not when the multi-part copy completed (which would've happened about 9 hours later).

I'm going to look at the S3 event configuration to see if it's erroneously publishing events for multi-part uploads when they're created rather than completed. This could also affect the Lambda consumer.

[mojodna]

Solved!

I misread the events in my email; what I was reading as the OSM PBF was actually the torrent. This meant that events were missing for all multi-part uploads.

Looking further into the configured event notifications, events for s3:ObjectCreated:CompleteMultipartUpload were not enabled, which would explain this behavior. This is now enabled.

[mhowell-ims]

OK thanks for checking @mojodna . We'll see if we get the message when the next upload happens.

[mhowell-ims]

This seems to be working again, thank @mojodna .