search

tannerjt / AGStoShapefile

Convert ArcGIS Server Dynamic Map Service to GeoJSON and Shapefile
MIT License
213 stars 47 forks source link

failed to export bigsize data #9

Closed sanjeevyanala closed 6 years ago

sanjeevyanala commented 8 years ago

Here is a problem when export bigsize data from json to geojson & shapefile. how can i solve this issue.

please help me for this issue. Thank You.

tannerjt commented 8 years ago

Can you provide the REST endpoint you are trying to download. I'm merging this with #8

sanjeevyanala commented 8 years ago

I have tried over 80 MB file, Sorry i can't provide my REST service endpoint. Please u can apply anyof your own service URL. Thank You

mdusch commented 8 years ago

I have a 480MB shp file that I am trying to grab:

https://services5.arcgis.com/GM2ldXfeLnrqaq3j/arcgis/rest/services/AllParcels/FeatureServer/0

I think the program has been running for something like 19 hours and has not had an error message or anything past the initial line to input node AGStoSHP.js.

sanjeevyanala commented 8 years ago

Finally did u get output ? Please try to solve the issue. And this tool will work on any type of arcgis rest service and all arcgis server version (latest 10.4) ?

Is there any programme to grab wms layer of geoserver (export to geojson), If yes please provide information.

Thank you

Thanks & Regards,

_Sanjeeva Reddy http://sanjeevareddyyanala.blogspot.in/ _Yanala http://sanjeevareddyyanala.blogspot.in/

On 23 June 2016 at 19:47, mdusch notifications@github.com wrote:

I have a 480MB shp file that I am trying to grab:

https://services5.arcgis.com/GM2ldXfeLnrqaq3j/arcgis/rest/services/AllParcels/FeatureServer/0

I think the program has been running for something like 19 hours and has not had an error message or anything past the initial line to input node AGStoSHP.js.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/tannerjt/AGStoShapefile/issues/9#issuecomment-228064101, or mute the thread https://github.com/notifications/unsubscribe/ASXwmjeEza9BypQitDc21KKv02gDgJ_Oks5qOpV5gaJpZM4IZ-pF .

mdusch commented 8 years ago

No I had no output, I cancelled the process as the script was running for over a day - I mean I expected a couple of hours because of the amount of lines it had to process but not that long. Is there a max amount of calls that a single IP can grab from a server?

sanjeevyanala commented 8 years ago

​There is no max amount of calls from IP, because we are grabing min 50 MB file also having same problem. Please try to solve this issue,..i am waiting for this. Thank u​

Thanks & Regards,

_Sanjeeva Reddy http://sanjeevareddyyanala.blogspot.in/ _Yanala http://sanjeevareddyyanala.blogspot.in/

On 24 June 2016 at 18:23, mdusch notifications@github.com wrote:

No I had no output, I cancelled the process as the script was running for over a day - I mean I expected a couple of hours because of the amount of lines it had to process but not that long. Is there a max amount of calls that a single IP can grab from a server?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/tannerjt/AGStoShapefile/issues/9#issuecomment-228337661, or mute the thread https://github.com/notifications/unsubscribe/ASXwmp4gYkHruACVyByE8Yd-E1ROZVSEks5qO9MvgaJpZM4IZ-pF .

tannerjt commented 8 years ago

This is a problem due to running out of memory with JSON.stringify(). I tried to convert everything to streams but still get error. You will need to break your query down into chunks using the URL Parameters available in the ArcGIS REST API.

mdusch commented 8 years ago

Would I still be able to use the services.txt file using these smaller chunks?

tannerjt commented 8 years ago

You can use the services.txt file. You will need to pull the latest updates if you cloned the repo (git pull origin master) and do another npm install to get the latest packages. You're query qould look something like this to reduce the payload:

https://services5.arcgis.com/GM2ldXfeLnrqaq3j/arcgis/rest/services/AllParcels/FeatureServer/0?where=FID<5000|parcels1 https://services5.arcgis.com/GM2ldXfeLnrqaq3j/arcgis/rest/services/AllParcels/FeatureServer/0?where=FID>5000 and FID <10000|parcels2

This would give you separate files with chunks of 5000 features. I'll solicit help on this issue because streaming would alleviate this issue.

tannerjt commented 8 years ago

mdusch, were you able to solve this issue using the above solution?

sanjeevyanala commented 8 years ago

I will try, but i am not sure.

Thanks & Regards,

_Sanjeeva Reddy http://sanjeevareddyyanala.blogspot.in/ _Yanala http://sanjeevareddyyanala.blogspot.in/

On 18 July 2016 at 10:05, Joshua Tanner notifications@github.com wrote:

mdusch, were you able to solve this issue using the above solution?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/tannerjt/AGStoShapefile/issues/9#issuecomment-233227353, or mute the thread https://github.com/notifications/unsubscribe-auth/ASXwmi_-CFgqr73xisT94IwA0EVN3SjOks5qWwKegaJpZM4IZ-pF .

mdusch commented 8 years ago

I was able to almost run the entire area through the script, but ran into some issues with the script. The service had 289718 features.

If I ran all queries in a row:

I first got it to work for 2 of 5 queries for the following services (specifically parcels2 and parcels5): The rest had the same error - Cannot read property '0' of undefined.

https://services5.arcgis.com/GM2ldXfeLnrqaq3j/arcgis/rest/services/AllParcels/FeatureServer/0?where=FID>0 and FID <5000|parcels1
https://services5.arcgis.com/GM2ldXfeLnrqaq3j/arcgis/rest/services/AllParcels/FeatureServer/0?where=FID>5000 and FID <10000|parcels2
https://services5.arcgis.com/GM2ldXfeLnrqaq3j/arcgis/rest/services/AllParcels/FeatureServer/0?where=FID>10000 and FID <15000|parcels3
https://services5.arcgis.com/GM2ldXfeLnrqaq3j/arcgis/rest/services/AllParcels/FeatureServer/0?where=FID>15000 and FID <20000|parcels4
https://services5.arcgis.com/GM2ldXfeLnrqaq3j/arcgis/rest/services/AllParcels/FeatureServer/0?where=FID>20000 and FID <25000|parcels5
https://services5.arcgis.com/GM2ldXfeLnrqaq3j/arcgis/rest/services/AllParcels/FeatureServer/0?where=FID>25000 and FID <29000|parcels6

I used the same service.txt queries for the next run, except I only did one query at a time.

All parcels except Parcels3 ran fine, if I ran each query individually (maybe a longer pause is needed for large queries?)

Parcels3 would only create a json, but no geojson or zipped shp file. (TypeError: Cannot read property 'x' of undefined at toGeoJSON)

image

This was run with the newest build.

mdusch commented 8 years ago

I had to switch to OBJECTID to run a different query with the script (the file did not have an FID field), but what was weird was the script skipped 910-999 and 90-99. I was able to grab those features, and checked to see if they were not in the original query (they weren't) based off of another ID field in the feature class.

http://oldhamgis.org:6080/arcgis/rest/services/LocalData/OLDHAM_BASEMAP/MapServer/5?where=PARCELS.OBJECTID>909 and PARCELS.OBJECTID <1000|OldH_Parcel_test1
http://oldhamgis.org:6080/arcgis/rest/services/LocalData/OLDHAM_BASEMAP/MapServer/5?where=PARCELS.OBJECTID>89 and PARCELS.OBJECTID <100|OldH_Parcel_test2
kramertamas commented 8 years ago

I have corrected a bug (issue #13) and made the code work for large shapefiles. It will split a large shapefile but won't merge the parts.

AGStoSHP.js.txt

kramertamas commented 8 years ago

On occasions, esri2geo stopped with an error (x is not a property of 'undefined'). It should be modified to check for typeof ft.geometry !== undefined.

esri2geo.js.txt

tannerjt commented 8 years ago

Thanks @kramertamas! I've added a new branch called bigdata with your modifications. I'd like to keep them as separate branches for now, as I need to test on my end. I would recommend anybody running into larger dataset issues use your modifications. Ideally, I would prefer to stream the output to a single file, but this is a great solution. I added a note in the readme.

mdusch commented 8 years ago

@tannerjt No shapefiles were created by the Bigdata branch.

Just an FYI I tried out the new Bigdata pull with a dataset of about 90080 features. It failed after 30980 (Iteration 31) features with "TypeError: Cannot read property 'attributes' of undefined".

I'll try splitting the amount of queries up in thirds to see if this will help.

mdusch commented 8 years ago

Update - Bigdata works fine in queries of 30,000. Still saves me a ton of time making query strings of 30000, rather than 5000.

rizkyfirmansyah commented 7 years ago

Dear all,

I have the same issue to convert large datasets into shapefile, the same problem with @mdusch. Here the screenshot of the console. error

The following ArcGIS REST service is in here: http://geoportal.menlhk.go.id/arcgis/rest/services/KLHK/BURN_AREA_20151028/MapServer/0

I have used the updated AGtoSHP.js to split huge data provided by @kramertamas, but nothing's work.

Please help me. Any thought is greatly appreciated.

Thank you.

tannerjt commented 7 years ago

Hey @rizkyfirmansyah . This particular issue is happening because you are trying to run the script against a tiled map service. Tiled maps will not work, as there is no geometry information, only tiles. If you look at the root of the service:

http://geoportal.menlhk.go.id/arcgis/rest/services/KLHK/BURN_AREA_20151028/MapServer

You will see the the tiling scheme that lets you know this is a tiled map service, not a dynamic map service.

rizkyfirmansyah commented 7 years ago

Oooh I see. So it should be a dynamic map service to be able generated into a shapefile. Thank you for your explanation @tannerjt

tannerjt commented 7 years ago

I think we could use request and stream the original Esri JSON to file

request('http://../arcgis/rest/.../0').pipe(fs.createWriteStream('file.json'))