search

mojodna / osm-pds-pipelines

OSM PDS pipeline
https://quay.io/repository/mojodna/osm-pds-pipelines
ISC License
32 stars 4 forks source link

Include planet-latest.osm.pbf file #25

Closed tehnrd closed 2 years ago

tehnrd commented 5 years ago

Definitely open to feedback on this but this PR would populate pbf/planet-latest.osm.pbf file in the osm-pds s3 bucket.

I think it's important for it to be a static path that will not change as this will greatly simplify writing scripts that require the latest planet.pbf file rather than inspecting the contents of the bucket and trying to figure out which file is the latest.

mojodna commented 5 years ago

This looks like it would definitely work, but the need to capture mirrorJobId and conditionally do stuff later based on it really makes me feel like the current structure is falling down. It wasn't particularly easy to follow the flow before, and I think that this affects readability negatively.

My gut suggests that refactoring away from using the async library to using async / await would clean things up tremendously and facilitate this use-case more effectively.

How would you feel about giving that refactor a shot?

tehnrd commented 5 years ago

Cool, ya, and totally agree with the async approach. It took me a while to wrap my head around it and I didn't feel great about storing the mirrorJobId outside the async function.

I can definitely take a look at incorporating async and await approach. Might be a little slower on the new PR as ramping down on the project that involved this so I'll have to squeeze it in when I have some time.

mojodna commented 5 years ago

Sounds good. Thanks!