Esri / ArcREST

python package for REST API (AGS, AGOL, webmap JSON, etc..)
Apache License 2.0
192 stars 155 forks source link

the replica sample downloads empty file geodatabase. #213

Closed jmk201 closed 8 years ago

jmk201 commented 8 years ago

I am working with the latest script on creating a replica. Sucessfully creates a zipped file geodatabase under C:\temp\ except that there is no data in it.

""" This sample shows how to create a replica from a FS

ArcREST version 3.5.x Python 2/3

""" from future import print_function from arcrest.security import AGOLTokenSecurityHandler from arcrest.agol import FeatureService from arcrest.common.filters import LayerDefinitionFilter

if name == "main": username = "####" password = "######" url = "####" proxy_port = None proxy_url = None agolSH = AGOLTokenSecurityHandler(username=username, password=password) fs = FeatureService( url=url, securityHandler=agolSH, proxy_port=proxy_port, proxy_url=proxy_url, initialize=True) result = fs.createReplica(replicaName='Demo', layers=[1], dataFormat="filegdb", out_path='C:\temp')

print( result)

Please help!!

pLeBlanc93 commented 8 years ago

Julie, could you edit your post to include the code (Create Replica FS) that you are running? Remove sensitive information as needed.

bnadler commented 8 years ago

I am also having an issue with the Create Replica. The sample (create_replica_fs.py) prints the file geodatabase as the response object (not the http response) and does NOT create any file (using 3.5.3). This was working in 3.0.1.

Looks like the response maintype for the zipped file has been changed to 'application', which I hacked in to line 532 of _base.py. Probably not the best way to do it but it fixed things

MikeMillerGIS commented 8 years ago

The current version in master is working for me. Can you share the service that is not working? Is it online or portal?

Things to check: 1: Ensure you are using the url to the service, not the layer in the service 2: Ensure you username and password is correct. ArcRest uses the token service, which is case sensitive, logging in through the web site uses the oauth, which is not. Do not assume the user name is correct because you can log in through the web.

Here is a service to try: http://services2.arcgis.com/PWJUSsdoJDp7SgLj/arcgis/rest/services/CarFS/FeatureServer

jmk201 commented 8 years ago

Yes, your service seems to work. Does it work only on feature layer? I am trying to download a standalone table.

http://services1.arcgis.com/MxjRokvPm7bjslyR/ArcGIS/rest/services/BusinessEditTime/FeatureServer

MikeMillerGIS commented 8 years ago

You cannot replica a layer or table, it has to be the service

If you want to extract just a layer, you will need to look at get_local_copy on FeatureLayer, this hides the logic to call replica on the parent item(service)

MikeMillerGIS commented 8 years ago

It is working on the service provided above without modification, can you please share a service that is not working? Can you link the line you changed? I do not see it on 532 https://github.com/Esri/ArcREST/blob/master/src/arcrest/web/_base.py#L532

bnadler commented 8 years ago

Not sure which version you are working with. It is in the _get function

if maintype.lower() in ('image', 'application/x-zip-compressed') or contentType == 'application/x-zip-compressed' or contentMD5 is not None or (contentDisposition is not None and contentDisposition.lower().find('attachment;') > -1):

Will send you the service info in a pm Ben Nadler | Esri Professional Services | Technical Analyst – Utilities, Telecom, & Energy C: 303.884.9323 | O: 303.449.7779 X8278 [cid:storage_emulated_0_DCIM_image002_png_1454429193608] | Sent from my phone

-------- Original message -------- From: Michael Miller notifications@github.com Date: 4/13/2016 8:03 AM (GMT-08:00) To: Esri/ArcREST ArcREST@noreply.github.com Cc: Ben Nadler BNadler@esri.com Subject: Re: [Esri/ArcREST] the replica sample downloads empty file geodatabase. (#213)

It is working on the service provided above without modification, can you please share a service that is not working? Can you link the line you changed? I do not see it on 532 https://github.com/Esri/ArcREST/blob/master/src/arcrest/web/_base.py#L532https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Esri_ArcREST_blob_master_src_arcrest_web_-5Fbase.py-23L532&d=CwMCaQ&c=n6-cguzQvX_tUIrZOS_4Og&r=2PIwAR_kAfcTQ3m7yzQ_wA&m=Rbw4Y3D8Na3jAq7oHQX52OcMS4ah3AraX4bVbTAnwdM&s=ztPqAqZTCgSAcdeWK-FM3W_pyE4gNrGrYhAgBRDNex8&e=

— You are receiving this because you commented. Reply to this email directly or view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Esri_ArcREST_issues_213-23issuecomment-2D209495531&d=CwMCaQ&c=n6-cguzQvX_tUIrZOS_4Og&r=2PIwAR_kAfcTQ3m7yzQ_wA&m=Rbw4Y3D8Na3jAq7oHQX52OcMS4ah3AraX4bVbTAnwdM&s=57vVzi0NH532L1ElqZXVsU5bC8BP7WIXvpnF4HAcqNY&e=

Fresh-Prince commented 8 years ago

Ben, I've got the same issue as you. Running v 3.5.3 and createReplica will produce the print statement but no file created locally. How did you remedy this?

bnadler commented 8 years ago

DISCLAIMER: THIS IS A TOTAL HACK

Look in the _get function in either .\site-packages\arcrest\web_base.py or .\ site-packages\arcrest\opendata_base.py (Or both?) Search for this line: if maintype.lower() in ('image', 'application/x-zip-compressed') or contentType …

I modified it to include ‘application’ if maintype.lower() in ('application', 'image', 'application/x-zip-compressed') or contentType …

That seems to have fixed it

I also had to handle the zipped file (since arcrest returned the unzipped file)

    result = fs.createReplica(replicaName='Report', layers=layers,returnAttachments=False,dataFormat="filegdb",out_path=outPath)
    print("-> Downloaded features from AGOL to {}".format(result))
    #Create Zipfile instance
    z = zipfile.ZipFile(result,'r')
    z.extractall(result[:-4])
    z.close()
    #Delete download, already extracted
    os.remove(result)
    #Format path unzipped gdb & return
    outputGDB = os.path.join(result[:-4], os.listdir(result[:-4])[0])
    return outputGDB
Fresh-Prince commented 8 years ago

Sweet. That worked. It does download the zipped gdb for me however it is naming the .gdb with a random base32 string. Thanks for your help.

pLeBlanc93 commented 8 years ago

@Fresh-Prince that is as designed. See the Create Replica JSON response example.

{
  "transportType": "esriTransportTypeURL",
  "responseType": "esriReplicaResponseTypeData",
  "URL" : "http://services.myserver.com/ERmEceOGq5cHrItq/ArcGIS/rest/services/example1/replicafiles/600ccf2726d14bfebfcb8576b29edb6f.json"
}
bnadler commented 8 years ago

No problem.

ArcRest used to return the UNZIPPED gdb, now it is returning the zipped download.

Something changed, but at least we got it working…

kdhl commented 8 years ago

I found that removing the where-clause from the create_replica_portal_item.py script (in the exportParameters) made all the difference to me. I'm using arcrest_package 3.5.6, python 2.7 and I'm using a arcmap 10.3.1 basic license although the latter doesn't seem important here.

hth, Bert