Json limit hit when downloading replica

[dougbrowning]

Creating a local replica worked great at 10,000 features. Now at 32,000 I get an error. Have I hit a limit? If so how do I get around it? Thanks

CODE: fs = FeatureService(url=url,securityHandler=agolSH,proxy_port=proxy_port,proxy_url=proxy_url,initialize=True) result = fs.createReplica(replicaName='Demo', layers='0,1,2,3,4,5', keep_replica=False, layerQueries=None, geometryFilter=None, returnAttachments=True,returnAttachmentDatabyURL=False,returnAsFeatureClass=True, out_path=outDir)

ERROR: Traceback (most recent call last): File "", line 34, in File "C:\Python27\ArcGIS10.2\lib\site-packages\arcrest\agol\featureservice.py", line 673, in createReplica proxy_port=self._proxy_port) File "C:\Python27\ArcGIS10.2\lib\site-packages\arcrest\web_base.py", line 82, in _do_post header=header) File "C:\Python27\ArcGIS10.2\lib\site-packages\arcrest\web_base.py", line 232, in _do_post_requests jres = json.loads(content) File "C:\Python27\ArcGIS10.2\lib\jsoninit.py", line 338, in loads return _default_decoder.decode(s) File "C:\Python27\ArcGIS10.2\lib\json\decoder.py", line 365, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\Python27\ArcGIS10.2\lib\json\decoder.py", line 383, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded

[achapkowski]

@dougbrowning - is there anyway you can share your service? I don't have any test that that comes near that size.

[dougbrowning]

Sure just send me your AGO username.

[achapkowski]

@dougbrowning - please use arcresttest

[dougbrowning]

Can not get AGO to find your username. Sent pm.

[achapkowski]

@dougbrowning - I can reproduce the issue, I think you may have found an issue beyond arcrest. Can you contact customer support with this one? The exact message I get back is: "An application error occurred on the server. The current custom error settings for this application prevent the details of the application error from being viewed remotely (for security reasons). It could, however, be viewed by browsers running on the local server machine."

[dougbrowning]

Would support even look at ArcRest issues?

Weird becuase I get this error ValueError: No JSON object could be decoded

On this function of code in decoder.py def raw_decode(self, s, idx=0): """Decode a JSON document from s (a str or unicode beginning with a JSON document) and return a 2-tuple of the Python representation and the index in s where the document ended.

    This can be used to decode a JSON document from a string that may
    have extraneous data at the end.

    """
    try:
        obj, end = self.scan_once(s, idx)
    except StopIteration:
        raise ValueError("No JSON object could be decoded")
    return obj, end

Also some more info as well

1. I have ran this script 20+ times just fine when there were less records.
2. I can take some of the layers out and it works fine. It crashes between 33,517 and 34,178 records.
3. I can also download any layer by itself with no issues so not a data issue.

This does have definition queries in it which I noticed split the data into different layers when publishing to AGO then combines then back together when bringing down a replica. Not sure if that matters but wanted to mention it.

I also am not sure it is a timeout issue as it does not take very long to do all but one of the layers (which is just 661 records). I can do any combination of layers 0-4 just not all of them at once. If I go into AGO and ask for a JSON file the file I get is 66 MB. Seems high since there are only 5 fields in each layer.

Thanks a lot

[MarieAshley]

Esri Support Services does not formally support ArcREST yet, but we do support the requests being made using the REST API. When submitting the ticket, provide the request and responses being sent (i.e. send a Fiddler .saz file over: http://www.telerik.com/fiddler).

[achapkowski]

@dougbrowning @AshleyDesktop The issue doesn't have to deal with ArcREST, but rather a server application error (http 500 error message).

The reason you are getting a decode error is because the value returned is HTML not JSON.

This is definitely something ArcREST needs to address, but you have a greater issue beyond this package.

[dougbrowning]

Now that I ran fiddler on it I do see that there is a error coming back. I did not know using Fiddler is what you meant at first. I have opened a ticket with Esri and I will let report back what comes of it.

Thanks

[achapkowski]

@dougbrowning - thank you! I'm going to close the issue but please comment the results of what you find out.

[dougbrowning]

Esri came back and said they do not know why on the error yet - it is not a time out though they say. They did say that if async was set to true it seemed to work. You have no variable for that so I dug into featureservice.py and changed async. But then it was mad on the unzipping. I commented that out and got a result of Not supported.

[dougbrowning]

I have confirmed that async works. Add /createReplica to the service URL then use this code below. The print response at the end gives you a url to the job. But then the job creates yet another URL to a zip file. Not sure how to get that zip file url back. You would probably know better than me.

Maybe this could be added to the ArcRest code. They are still working on the bug I think...

data = {'f' : 'json', 'replicaName' : 'Nov20Test', 'layers' : [0,1,2,3,4], 'returnAttachments' : 'true', 'returnAttachmentsDatabyURL' : 'false', 'syncModel' : 'none', 'dataFormat' : 'filegdb', 'async' : 'true'} encodedData = urllib.urlencode(data) request = urllib2.Request(url2, encodedData) response = urllib2.urlopen(request) readResponse = response.read() jsonResponse = json.loads(readResponse) print(jsonResponse)

[dougbrowning]

Got final word from Esri. This was all i got. Can not handle the big stuff as usual.

The following defect has been logged:

BUG-000092620: Creating a replica synchronously from a large number of records or vertices results in a 500 - Internal Server Error.