mojodna / osm-pds-pipelines

OSM PDS pipeline
https://quay.io/repository/mojodna/osm-pds-pipelines
ISC License
32 stars 4 forks source link

Replace osm2orc with osm-pbf-parquet #49

Open mojodna opened 3 months ago

mojodna commented 3 months ago

https://github.com/brad-richardson/osm-pbf-parquet

Pros:

Cons:

Unknowns:

ruanqizhen commented 3 months ago

Hi Seth, I was wondering if it would be feasible to copy the new result to the us-west-2 region on AWS. This is due to the fact that it is the only region accessible from an Overture account.

mojodna commented 3 months ago

As part of #51, I'd like to switch regions for the default bucket, which would address the problem for Overture. However, that's probably at least a few months out.

In the meantime, it may make sense to create an Overture AWS account where the region restriction doesn't apply and where we can register a job that copies ORC files into an internal Overture bucket. Let's chat about this on Slack.