wherobots / OvertureMaps

16 stars 4 forks source link

Overture Map Foundation's new release 2023-12-14-alpha.0 does not work #4

Open PavlosDem99 opened 10 months ago

PavlosDem99 commented 10 months ago

I am trying to download OFM data using the new release 12-14 but I cout an error. The error is: [PATH_NOT_FOUND] Path does not exist: s3a://wherobots-public-data/overturemaps-us-west-2/release/2023-12-14-alpha.0/theme=buildings/type=building.

image image

When i use the 07/26 release all are working perfectly.

Also, when I use this code from here Overture Map Data , also does not work.

from sedona.spark import *

config = SedonaContext.builder().config("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider").getOrCreate() sedona = SedonaContext.create(config)

df = sedona.read.format("geoparquet").load("s3a://overturemaps-us-west-2/release/2023-12-14-alpha.0/theme=places/type=place") df.filter("ST_Contains(ST_GeomFromWKT('POLYGON((-122.48 47.43,-122.20 47.75,-121.92 47.37,-122.48 47.43))'), geometry) = true").show()

jiayuasu commented 9 months ago

@PavlosDem99 Thanks for your interest. Our bucket only contains the first OMF release, the 07/26 release.

We will no longer provide new OMF data from our end because, with our help, OMF is now using Sedona to generate GeoParquet files. However, in OMF Nov and Dec releases, they didn't correctly partition data based on spatial proximity so the spatial filter is extremely slow. W are helping OMF fix this issue in the next release.

In fact, s3a://overturemaps-us-west-2/release/2023-12-14-alpha.0/theme=places/type=place works in our Sedona container and I just tried. But it is extremely slow because of the reason mentioned above.

jiayuasu commented 9 months ago

@PavlosDem99 Took me about 5 min to download places data in Bellevue, WA

image
PavlosDem99 commented 9 months ago

@PavlosDem99 Took me about 5 min to download places data in Bellevue, WA

image

Thank you @jiayuasu for your reply I appreciate it a lot. You are doing amazing job you and your team! Thanks I am waiting for the next release!