search

Esri / ops-server-config

Operations Server Configuration scripts
Apache License 2.0
34 stars 13 forks source link

Feature services aren't updated by GeoEvent Extension #632

Closed garysheppardjr closed 9 years ago

garysheppardjr commented 10 years ago

I'm following the 2014 Q4 Ops Server installation guide. I'm on page 46 (marked as page 45) and trying to verify that the GeoEvent Extension is updating feature services. It is not updating them. The simulators are running, and the GeoEvent Extension manager page is showing inputs and outputs on the services, but neither the Operations/FriendlyOperations service nor the Operations/FriendlySituation services has any features in it. I checked the logs for GeoEvent Extension and ArcGIS for Server and couldn't find any clues when running just the FriendlyOperations simulator. I don't understand why GeoEvent Extension would say it is writing outputs if those outputs are not actually written to the feature service.

kerryrobinson commented 10 years ago

I noticed the same issue on my ops server before leaving yesterday. Didn't have a chance to look into it, but I also have the issue of Layer 7 missing from the Friendly Situation feature service (I think), which might have something to do with it? I noticed this when opening the Watchkeeper dashboard.

pHill5136 commented 10 years ago

Open the service and check the field enricher (SicToSymbolCode). Look at the enrichment fields. Make sure that symbolrule (lo_friendly_equipment) and code (lo_friendly_units) are the enriched fields. Make sure sic is not one of the enrichment fields. Another thing to check is that the trackrep and units geoevent definitions do not have the field symbolrule. In either case the field enricher tries to add a field with the same name as an existing field in the original event during field enrichment and causes an error. Also the field enricher is designed such that if there is an error enriching, it returns null and waits for the next event. That would explain the counter moving but the symbols not showing.

garysheppardjr commented 10 years ago

Thanks, @pHill5136. For LO_FriendlyEquipment, the enriched fields are symbolrule and symbolname:

image

In the field mapper, only symbolrule is used in the output; symbolname is in neither the field mapper nor the output feature service.

The trackrep and units GeoEvent definitions do not have the field symbolrule.

Here's an idea: the service's output uses the Update a Feature output connector. There is a different output connector called Add a Feature. Can the Update a Feature connector only update existing features and not add new ones? That would explain it; the feature service has zero features at the moment. If this is the case, how does this feature service get populated in the first place?

conklinbd commented 10 years ago

@garys-esri The Update a Feature connector will add new features if one does not exist. That is the correct output. The issue is somewhere in the field enricher/field mapper.

pHill5136 commented 10 years ago

I saw this a few weeks back and the issue was with the enricher adding the sic field and we got an error saying that there were 2 fields with the same name so the new geoevent definition was not getting created. Sounds like in this case the sic field is not the issue.

Gary or Kerry could you send me the log files for geoevent. They are in the /data/logs/ and have names like 'karaf', 'karaf.1', etc. If you have been running for a while there is usually an enourmous amount of data logged... You might just want to zip the log folder and send that. Another thing you might try is to create a text-tcp output service (I think there is already an output for this processor on the OpsServer config, but I have attached a config with just this service that you can deploy on your gee server). In manager attach it just after the field enricher processor (sictosymbolid). Use the tcp console app (attached) to read the output. It will tell you the geoevent definition that is passing through the field enricher. If it is not the geoevent definition created in the field enricher --look in the field enricher's properties for the geoevent definition name-- then we know there is a problem somewhere in the field enricher.

-Patrick

From: Ben Conklin [notifications@github.com] Sent: Monday, November 03, 2014 8:29 AM To: ArcGIS/ops-server-config Cc: Patrick Hill Subject: Re: [ops-server-config] Feature services aren't updated by GeoEvent Extension (#632)

@garys-esrihttps://github.com/garys-esri The Update a Feature connector will add new features if one does not exist. That is the correct output. The issue is somewhere in the field enricher/field mapper.

— Reply to this email directly or view it on GitHubhttps://github.com/ArcGIS/ops-server-config/issues/632#issuecomment-61504541.

garysheppardjr commented 10 years ago

Thanks. I just emailed my logs to @pHill5136.

pHill5136 commented 9 years ago

@garys-esri at some point can you use the message simulator to send a bunch of equipment (id=trackrep) requests to your GEE. Turn off other messages and only send ones from the FriendlyOperations.xml so we can isolate equipment. I turned up the volume on messages to 100 to flood the service with messages. I think the 2 ArcGIS Server messages below are generated when I try to update the equipment, but there is so much other traffic on [dev] portal I can't be sure. Let me know if you are seeing these errors, too.

Error getting list of services. Failed to return all services configurations in the folder 'null'. Server machine 'https://[dev portal]:7443/arcgis/sharing/rest/search' returned an error. 'Invalid token.'

Failed to return all services configurations in the folder 'null'. Server machine 'https://[dev portal]:7443/arcgis/sharing/rest/search' returned an error. 'Invalid token.'

Not sure exactly what this would mean. I have an updated token and other layers update properly. I know behind the scenes AGS passes tokens around for everything.

kerryrobinson commented 9 years ago

@pHill5136 , i tested this a bit today. I checked all the geoevent definitions as you describe above and as far as I can tell they all look OK. but i also noticed all the superfluous simulated messages being run by default (the ones for air ops, using the baseops_* geoevents, which are resulting in a lot of errors in the GEE log files (let me know if you aren't also seeing these and i can send a screenshot i took of the log file). So i paired it down to only the Friendly Ops messages and noticed the friendly equipment was updating fine. I then also ran the situation reports and sustainment messages and for the most part these worked OK (I noticed the spot reports weren't updating but didn't have the chance to look into it).

so i'm also wondering if all of the messages were flooding the system so not all features were getting updated? i'm going to look into this more tomorrow and i'll report back what i find.

pHill5136 commented 9 years ago

@kerryrobinson I've seen all of those error messages and I believe those just mean that a message came through on the simulator, but could not map the tag to a field in the corresponding geoevent definition. I am seeing the same thing in our deployment on Friendly Equipment right now and I believe the error is being thrown by ArcGIS Server -- something to do with an invalid token. I know it is not the token used to connect to portal because we can update other layers on the portal. So I think something is getting corrupted in the communication being passed around behind the scenes, where server is always passing tokens around. I put in a question to the server alias, so I am hoping someone can give me an answer...

garysheppardjr commented 9 years ago

@pHill5136: I wanted to do that this morning (i.e. use only trackrep messages) but we now have so many Ops Servers running in my office that turning off my simulators barely made a difference. Maybe I'll see if I can use a firewall, change ports, etc. so I can use only trackrep messages and then send you the logs.

conklinbd commented 9 years ago

I find changing the port to be the simplest solution. You just need to change the Geomessage input and the simulator output port in that case.

pHill5136 commented 9 years ago

@garys-esri Another question are you seeing this on all of the services or just LO_Friendly_Equipment? It looks like on [dev server] Friendly_Equipment is the only one not updating. I haven't checked every service, but the problem seems to be with that layer in particular. If you find that you are getting that same token error I think we can at least isolate it to a problem on the layer and on the ags side of the equation. I might see if I can grab the mpk from the parent map doc and recreate the service with a different name and then try updating the new one. If that works I think we just have a corrupted feature service.

elinz commented 9 years ago

Does the LO_Friendly_Equipment geoevent service update layer 7 in the Operations/SituationReporting feature service? Looks like this layer is not getting published in the feature service. It does exist in the map service. You can republish this service by unzipping the SituationREporting.sd file using 7zip, open the .mxd contained in the unzipped files, reset the data sources to point to your Ops Server and republishing (with Overwrite option).

From: Patrick Hill [mailto:notifications@github.com] Sent: Thursday, November 6, 2014 9:41 AM To: ArcGIS/ops-server-config Subject: Re: [ops-server-config] Feature services aren't updated by GeoEvent Extension (#632)

@garys-esrihttps://github.com/garys-esri Another question are you seeing this on all of the services or just LO_Friendly_Equipment? It looks like on [dev server] Friendly_Equipment is the only one not updating. I haven't checked every service, but the problem seems to be with that layer in particular. If you find that you are getting that same token error I think we can at least isolate it to a problem on the layer and on the ags side of the equation. I might see if I can grab the mpk from the parent map doc and recreate the service with a different name and then try updating the new one. If that works I think we just have a corrupted feature service.

— Reply to this email directly or view it on GitHubhttps://github.com/ArcGIS/ops-server-config/issues/632#issuecomment-62019386.

kerryrobinson commented 9 years ago

@elinz I have already updated SituationReporting to include Layer 7 and still get issues, so it's not just that...

@pHill5136 I was also experiencing issues with other services. My experiences have been inconsistent but this week I've been trying to run the VM with only 11GB RAM on my 16GB host laptop so i know there is a lot of competition for resources (even though I turned off most services except Operations and Basemaps). Next week I'm going to look at this more using my laptop where I can allocate a lot more RAM to see if that helps narrow down the issue. I also need to check my server logs to confirm I'm seeing the same thing as you and Gary.

garysheppardjr commented 9 years ago

Here's an excerpt of karaf.log during one minute when no simulators are running and only the LO_FriendlyEquipment GeoEvent service is running (all inputs and outputs are running):

2014-11-06 14:38:08,186 | INFO  | ][FeatureServer] | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/NationalSecurity/RIRSpecialEvent/FeatureServer/0/deleteFeatures, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:38:08,186 | ERROR | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  563 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while deleting features in feature service RIRSpecialEvent. Delete String is: f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o.&rollbackOnFailure=true&where=date_ < timestamp '2014-11-06 19:36:08'. Error: null response.
2014-11-06 14:38:08,186 | INFO  | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  564 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | null response
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.cleanup(FeatureServiceOutboundTransport.java:558)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.access$200(FeatureServiceOutboundTransport.java:38)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport$CleanupThread.run(FeatureServiceOutboundTransport.java:637)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
2014-11-06 14:38:27,421 | INFO  | ][FeatureServer] | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/deleteFeatures, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:38:27,421 | ERROR | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  563 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while deleting features in feature service FriendlyOperations. Delete String is: f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o.&rollbackOnFailure=true&where=editedtime < timestamp '2014-11-06 19:24:27'. Error: null response.
2014-11-06 14:38:27,422 | INFO  | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  564 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | null response
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.cleanup(FeatureServiceOutboundTransport.java:558)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.access$200(FeatureServiceOutboundTransport.java:38)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport$CleanupThread.run(FeatureServiceOutboundTransport.java:637)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
2014-11-06 14:38:27,422 | INFO  | ][FeatureServer] | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/6/deleteFeatures, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:38:27,422 | ERROR | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  563 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while deleting features in feature service FriendlyOperations. Delete String is: f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o.&rollbackOnFailure=true&where=datetimesubmitted < timestamp '2014-11-06 19:24:27'. Error: null response.
2014-11-06 14:38:27,422 | INFO  | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  564 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | null response
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.cleanup(FeatureServiceOutboundTransport.java:558)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.access$200(FeatureServiceOutboundTransport.java:38)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport$CleanupThread.run(FeatureServiceOutboundTransport.java:637)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
2014-11-06 14:38:28,192 | INFO  | ][FeatureServer] | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/NationalSecurity/RIRSpecialEvent/FeatureServer/0/deleteFeatures, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:38:28,193 | ERROR | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  563 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while deleting features in feature service RIRSpecialEvent. Delete String is: f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o.&rollbackOnFailure=true&where=date_ < timestamp '2014-11-06 19:36:28'. Error: null response.
2014-11-06 14:38:28,193 | INFO  | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  564 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | null response
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.cleanup(FeatureServiceOutboundTransport.java:558)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.access$200(FeatureServiceOutboundTransport.java:38)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport$CleanupThread.run(FeatureServiceOutboundTransport.java:637)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
2014-11-06 14:38:48,292 | INFO  | ][FeatureServer] | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/NationalSecurity/RIRSpecialEvent/FeatureServer/0/deleteFeatures, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:38:48,292 | ERROR | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  563 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while deleting features in feature service RIRSpecialEvent. Delete String is: f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o.&rollbackOnFailure=true&where=date_ < timestamp '2014-11-06 19:36:48'. Error: null response.
2014-11-06 14:38:48,292 | INFO  | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  564 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | null response
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.cleanup(FeatureServiceOutboundTransport.java:558)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.access$200(FeatureServiceOutboundTransport.java:38)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport$CleanupThread.run(FeatureServiceOutboundTransport.java:637)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
2014-11-06 14:38:57,435 | INFO  | ][FeatureServer] | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/6/deleteFeatures, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:38:57,435 | ERROR | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  563 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while deleting features in feature service FriendlyOperations. Delete String is: f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o.&rollbackOnFailure=true&where=datetimesubmitted < timestamp '2014-11-06 19:24:57'. Error: null response.
2014-11-06 14:38:57,436 | INFO  | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  564 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | null response
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.cleanup(FeatureServiceOutboundTransport.java:558)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.access$200(FeatureServiceOutboundTransport.java:38)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport$CleanupThread.run(FeatureServiceOutboundTransport.java:637)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
2014-11-06 14:38:57,437 | INFO  | ][FeatureServer] | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/deleteFeatures, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:38:57,437 | ERROR | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  563 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while deleting features in feature service FriendlyOperations. Delete String is: f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o.&rollbackOnFailure=true&where=editedtime < timestamp '2014-11-06 19:24:57'. Error: null response.
2014-11-06 14:38:57,437 | INFO  | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  564 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | null response
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.cleanup(FeatureServiceOutboundTransport.java:558)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.access$200(FeatureServiceOutboundTransport.java:38)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport$CleanupThread.run(FeatureServiceOutboundTransport.java:637)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]

If I try to run deleteFeatures on Operations/FriendlyOperations/FeatureServer/11 in a browser, it works fine, even using that same where clause.

garysheppardjr commented 9 years ago

And here's a minute's worth of the log while running a simulator featuring only trackrep messages taken from FriendlyOpertations.xml:

2014-11-06 14:49:02,031 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]pty response), second attempt is (Empty response).
2014-11-06 14:49:02,032 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('3A1-001')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:02,033 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:02,970 | INFO  | ][FeatureServer] | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/deleteFeatures, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:03,037 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:03,400 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('SupplyConvoy2')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:03,400 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:02,976 | INFO  | ][FeatureServer] | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/6/deleteFeatures, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:03,401 | ERROR | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  563 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while deleting features in feature service FriendlyOperations. Delete String is: f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o.&rollbackOnFailure=true&where=datetimesubmitted < timestamp '2014-11-06 19:35:02'. Error: null response.
2014-11-06 14:49:03,401 | INFO  | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  564 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | null response
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.cleanup(FeatureServiceOutboundTransport.java:558)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.access$200(FeatureServiceOutboundTransport.java:38)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport$CleanupThread.run(FeatureServiceOutboundTransport.java:637)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
2014-11-06 14:49:03,402 | ERROR | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  563 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while deleting features in feature service FriendlyOperations. Delete String is: f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o.&rollbackOnFailure=true&where=editedtime < timestamp '2014-11-06 19:35:02'. Error: null response.
2014-11-06 14:49:03,402 | INFO  | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  564 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | null response
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.cleanup(FeatureServiceOutboundTransport.java:558)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.access$200(FeatureServiceOutboundTransport.java:38)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport$CleanupThread.run(FeatureServiceOutboundTransport.java:637)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
2014-11-06 14:49:06,867 | WARN  | SyncThread:1     | FileTxnLog                       | er.server.persistence.FileTxnLog  335 | 17 - org.apache.zookeeper.zookeeper-geoevent - 3.5.0 | fsync-ing the write ahead log in SyncThread:1 took 2315ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2014-11-06 14:49:06,881 | ERROR | In DataProcessor | DefenseInboundAdapter            | geomessage.DefenseInboundAdapter   93 | 388 - com.esri.geoevent.solutions.adapter.geomessage.geomessage-adapter - 10.3.0 | SAXException while trying to parse the incoming xml.
org.xml.sax.SAXParseException: The markup in the document following the root element must be well-formed.
    at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)[:]
    at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLDocumentScannerImpl$TrailingMiscDispatcher.dispatch(Unknown Source)[:]
    at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)[:]
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)[:]
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)[:]
    at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)[:]
    at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)[:]
    at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)[:]
    at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)[:]
    at javax.xml.parsers.SAXParser.parse(Unknown Source)[:2.4.0]
    at com.esri.geoevent.solutions.adapter.geomessage.DefenseInboundAdapter.receive(DefenseInboundAdapter.java:87)[388:com.esri.geoevent.solutions.adapter.geomessage.geomessage-adapter:10.3.0]
    at com.esri.ges.manager.stream.internal.InboundAdapterProxy.receive(InboundAdapterProxy.java:39)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.InboundStreamImpl$DataProcessor.run(InboundStreamImpl.java:79)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
2014-11-06 14:49:06,896 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:06,897 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('3A2-002')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:06,897 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:07,954 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:07,955 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('1B2-002','3C2-007')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:07,955 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:09,367 | WARN  | SyncThread:1     | FileTxnLog                       | er.server.persistence.FileTxnLog  335 | 17 - org.apache.zookeeper.zookeeper-geoevent - 3.5.0 | fsync-ing the write ahead log in SyncThread:1 took 1411ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2014-11-06 14:49:09,372 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:09,372 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('3B2-005')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:09,372 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:11,669 | WARN  | SyncThread:1     | FileTxnLog                       | er.server.persistence.FileTxnLog  335 | 17 - org.apache.zookeeper.zookeeper-geoevent - 3.5.0 | fsync-ing the write ahead log in SyncThread:1 took 2299ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2014-11-06 14:49:11,748 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:11,748 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https:/[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('SupplyConvoy1')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:11,748 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:12,568 | ERROR | In DataProcessor | DefenseInboundAdapter            | geomessage.DefenseInboundAdapter   93 | 388 - com.esri.geoevent.solutions.adapter.geomessage.geomessage-adapter - 10.3.0 | SAXException while trying to parse the incoming xml.
org.xml.sax.SAXParseException: The markup in the document following the root element must be well-formed.
    at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)[:]
    at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLDocumentScannerImpl$TrailingMiscDispatcher.dispatch(Unknown Source)[:]
    at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)[:]
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)[:]
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)[:]
    at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)[:]
    at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)[:]
    at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)[:]
    at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)[:]
    at javax.xml.parsers.SAXParser.parse(Unknown Source)[:2.4.0]
    at com.esri.geoevent.solutions.adapter.geomessage.DefenseInboundAdapter.receive(DefenseInboundAdapter.java:87)[388:com.esri.geoevent.solutions.adapter.geomessage.geomessage-adapter:10.3.0]
    at com.esri.ges.manager.stream.internal.InboundAdapterProxy.receive(InboundAdapterProxy.java:39)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.InboundStreamImpl$DataProcessor.run(InboundStreamImpl.java:79)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
2014-11-06 14:49:12,751 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:12,752 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https:/[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('2C3-003','US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:12,752 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:14,765 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:15,018 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:15,018 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:16,072 | INFO  | ][FeatureServer] | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https:/[server]:6443/arcgis/rest/services/NationalSecurity/RIRSpecialEvent/FeatureServer/0/deleteFeatures, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:16,072 | ERROR | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  563 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while deleting features in feature service RIRSpecialEvent. Delete String is: f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o.&rollbackOnFailure=true&where=date_ < timestamp '2014-11-06 19:47:15'. Error: null response.
2014-11-06 14:49:16,073 | INFO  | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  564 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | null response
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.cleanup(FeatureServiceOutboundTransport.java:558)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.access$200(FeatureServiceOutboundTransport.java:38)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport$CleanupThread.run(FeatureServiceOutboundTransport.java:637)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
2014-11-06 14:49:17,045 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https:/[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:17,136 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:17,137 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:18,597 | WARN  | SyncThread:1     | FileTxnLog                       | er.server.persistence.FileTxnLog  335 | 17 - org.apache.zookeeper.zookeeper-geoevent - 3.5.0 | fsync-ing the write ahead log in SyncThread:1 took 1032ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2014-11-06 14:49:19,197 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:19,197 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:19,197 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:20,201 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:20,202 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:20,202 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:21,205 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:21,205 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https:/[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:21,205 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:22,217 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:22,217 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:22,217 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:23,228 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https:/[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:23,228 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:23,228 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:24,232 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:24,233 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:24,233 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:25,237 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:25,237 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https:/[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:25,237 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:26,240 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:26,241 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:26,241 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:27,245 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:27,245 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:27,246 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:28,249 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:28,249 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:28,250 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:29,259 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:29,259 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:29,260 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:33,620 | INFO  | ][FeatureServer] | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/6/deleteFeatures, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:33,632 | ERROR | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  563 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while deleting features in feature service FriendlyOperations. Delete String is: f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o.&rollbackOnFailure=true&where=datetimesubmitted < timestamp '2014-11-06 19:35:33'. Error: null response.
2014-11-06 14:49:33,634 | INFO  | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  564 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | null response
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.cleanup(FeatureServiceOutboundTransport.java:558)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.access$200(FeatureServiceOutboundTransport.java:38)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport$CleanupThread.run(FeatureServiceOutboundTransport.java:637)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
2014-11-06 14:49:33,688 | INFO  | ][FeatureServer] | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/deleteFeatures, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:33,689 | ERROR | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  563 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while deleting features in feature service FriendlyOperations. Delete String is: f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o.&rollbackOnFailure=true&where=editedtime < timestamp '2014-11-06 19:35:33'. Error: null response.
2014-11-06 14:49:33,689 | INFO  | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  564 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | null response
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.cleanup(FeatureServiceOutboundTransport.java:558)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.access$200(FeatureServiceOutboundTransport.java:38)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport$CleanupThread.run(FeatureServiceOutboundTransport.java:637)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
2014-11-06 14:49:33,827 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:33,828 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:33,828 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:35,515 | WARN  | SyncThread:1     | FileTxnLog                       | er.server.persistence.FileTxnLog  335 | 17 - org.apache.zookeeper.zookeeper-geoevent - 3.5.0 | fsync-ing the write ahead log in SyncThread:1 took 1099ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2014-11-06 14:49:35,859 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:35,859 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:35,859 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:36,893 | INFO  | ][FeatureServer] | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/NationalSecurity/RIRSpecialEvent/FeatureServer/0/deleteFeatures, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:36,900 | ERROR | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  563 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while deleting features in feature service RIRSpecialEvent. Delete String is: f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o.&rollbackOnFailure=true&where=date_ < timestamp '2014-11-06 19:47:36'. Error: null response.
2014-11-06 14:49:36,901 | INFO  | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  564 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | null response
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.cleanup(FeatureServiceOutboundTransport.java:558)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.access$200(FeatureServiceOutboundTransport.java:38)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport$CleanupThread.run(FeatureServiceOutboundTransport.java:637)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
2014-11-06 14:49:39,757 | WARN  | SyncThread:1     | FileTxnLog                       | er.server.persistence.FileTxnLog  335 | 17 - org.apache.zookeeper.zookeeper-geoevent - 3.5.0 | fsync-ing the write ahead log in SyncThread:1 took 2931ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2014-11-06 14:49:40,271 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:40,292 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:40,300 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:41,033 | WARN  | SyncThread:1     | FileTxnLog                       | er.server.persistence.FileTxnLog  335 | 17 - org.apache.zookeeper.zookeeper-geoevent - 3.5.0 | fsync-ing the write ahead log in SyncThread:1 took 1198ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2014-11-06 14:49:41,689 | ERROR | In DataProcessor | DefenseInboundAdapter            | geomessage.DefenseInboundAdapter   93 | 388 - com.esri.geoevent.solutions.adapter.geomessage.geomessage-adapter - 10.3.0 | SAXException while trying to parse the incoming xml.
org.xml.sax.SAXParseException: The markup in the document following the root element must be well-formed.
    at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)[:]
    at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLDocumentScannerImpl$TrailingMiscDispatcher.dispatch(Unknown Source)[:]
    at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)[:]
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)[:]
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)[:]
    at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)[:]
    at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)[:]
    at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)[:]
    at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)[:]
    at javax.xml.parsers.SAXParser.parse(Unknown Source)[:2.4.0]
    at com.esri.geoevent.solutions.adapter.geomessage.DefenseInboundAdapter.receive(DefenseInboundAdapter.java:87)[388:com.esri.geoevent.solutions.adapter.geomessage.geomessage-adapter:10.3.0]
    at com.esri.ges.manager.stream.internal.InboundAdapterProxy.receive(InboundAdapterProxy.java:39)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.InboundStreamImpl$DataProcessor.run(InboundStreamImpl.java:79)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
2014-11-06 14:49:42,322 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:42,322 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:42,323 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:43,345 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:43,346 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:43,346 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:44,356 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:44,356 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:44,357 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:45,399 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:45,532 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:45,533 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:46,540 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:46,755 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:46,755 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:49,898 | WARN  | SyncThread:1     | FileTxnLog                       | er.server.persistence.FileTxnLog  335 | 17 - org.apache.zookeeper.zookeeper-geoevent - 3.5.0 | fsync-ing the write ahead log in SyncThread:1 took 2279ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2014-11-06 14:49:52,123 | WARN  | SyncThread:1     | FileTxnLog                       | er.server.persistence.FileTxnLog  335 | 17 - org.apache.zookeeper.zookeeper-geoevent - 3.5.0 | fsync-ing the write ahead log in SyncThread:1 took 1368ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2014-11-06 14:49:52,565 | ERROR | In DataProcessor | DefenseInboundAdapter            | geomessage.DefenseInboundAdapter   93 | 388 - com.esri.geoevent.solutions.adapter.geomessage.geomessage-adapter - 10.3.0 | SAXException while trying to parse the incoming xml.
org.xml.sax.SAXParseException: The markup in the document following the root element must be well-formed.
    at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)[:]
    at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLDocumentScannerImpl$TrailingMiscDispatcher.dispatch(Unknown Source)[:]
    at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)[:]
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)[:]
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)[:]
    at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)[:]
    at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)[:]
    at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)[:]
    at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)[:]
    at javax.xml.parsers.SAXParser.parse(Unknown Source)[:2.4.0]
    at com.esri.geoevent.solutions.adapter.geomessage.DefenseInboundAdapter.receive(DefenseInboundAdapter.java:87)[388:com.esri.geoevent.solutions.adapter.geomessage.geomessage-adapter:10.3.0]
    at com.esri.ges.manager.stream.internal.InboundAdapterProxy.receive(InboundAdapterProxy.java:39)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.InboundStreamImpl$DataProcessor.run(InboundStreamImpl.java:79)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
2014-11-06 14:49:52,574 | ERROR | In DataProcessor | DefenseInboundAdapter            | geomessage.DefenseInboundAdapter   93 | 388 - com.esri.geoevent.solutions.adapter.geomessage.geomessage-adapter - 10.3.0 | SAXException while trying to parse the incoming xml.
org.xml.sax.SAXParseException: The markup in the document following the root element must be well-formed.
    at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)[:]
    at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source)[:]
    at org.apache.xerces.impl.XMLDocumentScannerImpl$TrailingMiscDispatcher.dispatch(Unknown Source)[:]
    at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)[:]
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)[:]
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)[:]
    at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)[:]
    at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)[:]
    at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)[:]
    at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)[:]
    at javax.xml.parsers.SAXParser.parse(Unknown Source)[:2.4.0]
    at com.esri.geoevent.solutions.adapter.geomessage.DefenseInboundAdapter.receive(DefenseInboundAdapter.java:87)[388:com.esri.geoevent.solutions.adapter.geomessage.geomessage-adapter:10.3.0]
    at com.esri.ges.manager.stream.internal.InboundAdapterProxy.receive(InboundAdapterProxy.java:39)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.InboundStreamImpl$DataProcessor.run(InboundStreamImpl.java:79)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
2014-11-06 14:49:52,904 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:52,904 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:52,904 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:53,913 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:53,913 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:53,913 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:54,918 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:55,979 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:55,986 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:57,208 | INFO  | ][FeatureServer] | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/NationalSecurity/RIRSpecialEvent/FeatureServer/0/deleteFeatures, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:57,208 | ERROR | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  563 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while deleting features in feature service RIRSpecialEvent. Delete String is: f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o.&rollbackOnFailure=true&where=date_ < timestamp '2014-11-06 19:47:56'. Error: null response.
2014-11-06 14:49:57,208 | INFO  | ][FeatureServer] | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  564 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | null response
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.cleanup(FeatureServiceOutboundTransport.java:558)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.access$200(FeatureServiceOutboundTransport.java:38)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport$CleanupThread.run(FeatureServiceOutboundTransport.java:637)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
2014-11-06 14:49:57,240 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:57,241 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:57,242 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:58,429 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:58,430 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:58,430 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
2014-11-06 14:49:59,436 | INFO  | ound/JSON/10.3.0 | Http                             | com.esri.ges.httpclient.Http      273 | 232 - com.esri.ges.framework.httpclient - 10.3.0 | While fetching POST https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query, first attempt (Empty response), second attempt is (Empty response).
2014-11-06 14:49:59,436 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  455 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error Posting to URL: https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('US65')&outfields=uniquedesignation,objectid&f=json&token=LHS_frsMoAV5TqwYRzRLB_HT-dGB9XqSn_XgFw1Z25IGNseY0JGhVohSEADiUmVLsAqrFYqxBPQjuhs_jGwWiiD3gYW7jW56NVQ16gVUnv6tKzJM0m7p4nXI0o2BJFUlA5S-ckIXm7LfpZ-2mttNDYMYUaKsPW-sink22hfevVBIn1CuQ3PyQ-QF7APY3fFhke2kJ8WRJBpOyrP_cVW_u6VX13Lmqx7tFPCZg5rSH_o..
java.io.IOException: null response
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:739)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:451)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:432)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:152)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:83)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0]
    at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
2014-11-06 14:49:59,437 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport  | .FeatureServiceOutboundTransport  168 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.
pHill5136 commented 9 years ago

I used to see the unable to delete feature error a lot if layers were created from feature sets that have versioning turned on. With all the updates sde gets flooded with deltas. Eventually the memory will max out. Things wouldn't update or they would update inconsistently... Do you guys see the postgres memory usage creeping up as the simulator is run?

pHill5136 commented 9 years ago

@jrweakland Is the mxd used to publish FriendlyOperations in a public location somewhere? Do you know if 'Automatically Save Changes after Each Edit' was checked in the map doc? Not sure if there is a way to check that on on the service...

pHill5136 commented 9 years ago

@garys-esri @kerryrobinson Still working on this. Could you guys look in your portal logs (C:\arcgisportal\logs\portal)? Go to any day that you tried running the simulations on FriendlyEquipment (or any layer in FriendlyOperations)See if you are getting a something similar to the following:

Msg time="2014-11-14T15:27:29,766" type="INFO" code="-1" source="Sharing" process="19624" thread="13" methodName="Class.method:com.esri.gw.publish.AgsServicePublisher.getServiceInfo File.line:AgsServicePublisher.java.708 Logger.name:com.esri.gw.publish.AgsServicePublisher" machine="[server]" user="" elapsed="" Get ServiceInfo for https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer</Msg Msg time="2014-11-14T15:27:29,795" type="VERBOSE" code="-1" source="Sharing" process="19624" thread="13" methodName="Class.method:com.esri.gw.security.AccessTokenValidator.a File.line:AccessTokenValidator.java.94 Logger.name:com.esri.gw.security.AccessTokenValidator" machine="[server]" user="" elapsed=""> Token expired. (Token expiry time: 1413146840626, Current time: 1416007649795)

These are 2 messages generated by the same thread. The first is trying to access FriendlyOperations. The second generated a fraction of a second later is an invalid token error. If you look at the token expiration time-stamp it translates to sometime on October 12. There are tons of these on our server. I am thinking that possibly geoevent does not update its token properly. So we sent out a config file with a token that was already expired and then when you try to update it does not work. GeoEvent says it is connected, but I'mm thinking it is not.

I am curious if you guys have a token with an expiration timestamp of 1413146840626.

I just tried deleting the OpsServer instance on our server and creating a completely new one. It will take a while for the services to validate. If my services start tracking or if you are getting the same expiration on the token despite updating the token in geoevent we will be on to something with this.

Thanks -P

garysheppardjr commented 9 years ago

I have no log messages with "ServiceInfo" or "Token expired" in them. I don't see anything suspicious in the Portal logs. Here's what I get in the GeoEvent log for each Friendly Operations trackrep message:

2014-11-17 15:59:02,296 | ERROR |  #20 - Multicast | FeatureServiceDataProvider       | icher.FeatureServiceDataProvider  155 | 278 - com.esri.ges.framework.processor.fieldenricher-processor - 10.3.0 | Unexpected error while parsing the data. Error: null.
2014-11-17 15:59:02,296 | INFO  |  #20 - Multicast | FeatureServiceDataProvider       | icher.FeatureServiceDataProvider  156 | 278 - com.esri.ges.framework.processor.fieldenricher-processor - 10.3.0 | 
java.lang.NullPointerException
    at com.esri.ges.datastore.agsconnection.DefaultArcGISServerConnection.getFeatureAttributes(DefaultArcGISServerConnection.java:1632)[241:com.esri.ges.framework.datastore.agsconnection-datastore:10.3.0]
    at com.esri.ges.processor.fieldenricher.FeatureServiceDataProvider.getData(FeatureServiceDataProvider.java:123)[278:com.esri.ges.framework.processor.fieldenricher-processor:10.3.0]
    at com.esri.ges.processor.fieldenricher.FeatureServiceDataProvider.getData(FeatureServiceDataProvider.java:101)[278:com.esri.ges.framework.processor.fieldenricher-processor:10.3.0]
    at com.esri.ges.processor.fieldenricher.FieldEnricher.process(FieldEnricher.java:85)[278:com.esri.ges.framework.processor.fieldenricher-processor:10.3.0]
    at com.esri.ges.processing.camel.GeoEventServiceRouteBuilder$4.process(GeoEventServiceRouteBuilder.java:399)[245:com.esri.ges.framework.processing.camel-processing:10.3.0]
    at org.apache.camel.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:63)[165:org.apache.camel.camel-core:2.12.3]
    at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)[165:org.apache.camel.camel-core:2.12.3]
    at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)[165:org.apache.camel.camel-core:2.12.3]
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)[165:org.apache.camel.camel-core:2.12.3]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)[165:org.apache.camel.camel-core:2.12.3]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)[165:org.apache.camel.camel-core:2.12.3]
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)[165:org.apache.camel.camel-core:2.12.3]
    at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:51)[165:org.apache.camel.camel-core:2.12.3]
    at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:110)[165:org.apache.camel.camel-core:2.12.3]
    at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)[165:org.apache.camel.camel-core:2.12.3]
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)[165:org.apache.camel.camel-core:2.12.3]
    at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)[165:org.apache.camel.camel-core:2.12.3]
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)[165:org.apache.camel.camel-core:2.12.3]
    at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:105)[165:org.apache.camel.camel-core:2.12.3]
    at org.apache.camel.processor.MulticastProcessor.doProcessParallel(MulticastProcessor.java:712)[165:org.apache.camel.camel-core:2.12.3]
    at org.apache.camel.processor.MulticastProcessor.access$200(MulticastProcessor.java:83)[165:org.apache.camel.camel-core:2.12.3]
    at org.apache.camel.processor.MulticastProcessor$1.call(MulticastProcessor.java:293)[165:org.apache.camel.camel-core:2.12.3]
    at org.apache.camel.processor.MulticastProcessor$1.call(MulticastProcessor.java:278)[165:org.apache.camel.camel-core:2.12.3]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)[:1.7.0_65]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_65]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_65]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
kerryrobinson commented 9 years ago

Patrick, I haven’t been able to test this yet. I should be able to tomorrow and I’ll post what I find.

Thanks, Kerry

Kerry Robinson | Senior Technical Advisor Esri | European Defence Team | Bray, Co. Wicklow | Ireland M +353 86 383 8901callto:+353863838901 | W +1 909 793 2853 x2530callto:+19097932853 krobinson@esri.commailto:krobinson@esri.com | esri.com/defense http://www.esri.com/defense

From: Gary Sheppard [mailto:notifications@github.com] Sent: 17 November 2014 21:02 To: ArcGIS/ops-server-config Cc: Kerry Robinson Subject: Re: [ops-server-config] Feature services aren't updated by GeoEvent Extension (#632)

I have no log messages with "ServiceInfo" or "Token expired" in them. I don't see anything suspicious in the Portal logs. Here's what I get in the GeoEvent log for each Friendly Operations trackrep message:

2014-11-17 15:59:02,296 | ERROR | #20 - Multicast | FeatureServiceDataProvider | icher.FeatureServiceDataProvider 155 | 278 - com.esri.ges.framework.processor.fieldenricher-processor - 10.3.0 | Unexpected error while parsing the data. Error: null.

2014-11-17 15:59:02,296 | INFO | #20 - Multicast | FeatureServiceDataProvider | icher.FeatureServiceDataProvider 156 | 278 - com.esri.ges.framework.processor.fieldenricher-processor - 10.3.0 |

java.lang.NullPointerException

at com.esri.ges.datastore.agsconnection.DefaultArcGISServerConnection.getFeatureAttributes(DefaultArcGISServerConnection.java:1632)[241:com.esri.ges.framework.datastore.agsconnection-datastore:10.3.0]

at com.esri.ges.processor.fieldenricher.FeatureServiceDataProvider.getData(FeatureServiceDataProvider.java:123)[278:com.esri.ges.framework.processor.fieldenricher-processor:10.3.0]

at com.esri.ges.processor.fieldenricher.FeatureServiceDataProvider.getData(FeatureServiceDataProvider.java:101)[278:com.esri.ges.framework.processor.fieldenricher-processor:10.3.0]

at com.esri.ges.processor.fieldenricher.FieldEnricher.process(FieldEnricher.java:85)[278:com.esri.ges.framework.processor.fieldenricher-processor:10.3.0]

at com.esri.ges.processing.camel.GeoEventServiceRouteBuilder$4.process(GeoEventServiceRouteBuilder.java:399)[245:com.esri.ges.framework.processing.camel-processing:10.3.0]

at org.apache.camel.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:63)[165:org.apache.camel.camel-core:2.12.3]

at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)[165:org.apache.camel.camel-core:2.12.3]

at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)[165:org.apache.camel.camel-core:2.12.3]

at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)[165:org.apache.camel.camel-core:2.12.3]

at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)[165:org.apache.camel.camel-core:2.12.3]

at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)[165:org.apache.camel.camel-core:2.12.3]

at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)[165:org.apache.camel.camel-core:2.12.3]

at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:51)[165:org.apache.camel.camel-core:2.12.3]

at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:110)[165:org.apache.camel.camel-core:2.12.3]

at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)[165:org.apache.camel.camel-core:2.12.3]

at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)[165:org.apache.camel.camel-core:2.12.3]

at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)[165:org.apache.camel.camel-core:2.12.3]

at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)[165:org.apache.camel.camel-core:2.12.3]

at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:105)[165:org.apache.camel.camel-core:2.12.3]

at org.apache.camel.processor.MulticastProcessor.doProcessParallel(MulticastProcessor.java:712)[165:org.apache.camel.camel-core:2.12.3]

at org.apache.camel.processor.MulticastProcessor.access$200(MulticastProcessor.java:83)[165:org.apache.camel.camel-core:2.12.3]

at org.apache.camel.processor.MulticastProcessor$1.call(MulticastProcessor.java:293)[165:org.apache.camel.camel-core:2.12.3]

at org.apache.camel.processor.MulticastProcessor$1.call(MulticastProcessor.java:278)[165:org.apache.camel.camel-core:2.12.3]

at java.util.concurrent.FutureTask.run(FutureTask.java:262)[:1.7.0_65]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_65]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_65]

at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]

— Reply to this email directly or view it on GitHubhttps://github.com/ArcGIS/ops-server-config/issues/632#issuecomment-63375034.

kerryrobinson commented 9 years ago

@pHill5136, @elinz , I looked into this more before trying to recreate the Data Store as Patrick suggested to me. I realized that ArcGIS Server was showing an error "Initialization of Layer: Situation Alerts failed". I checked and my FriendlyOperations feature class did not have a Situation Alerts layer. So i recreated the Feature Service using the MXD from the arcgisinputs directory, including the Situation Alerts layer, and now GEE is updating Feature Equipment.

So there is a disconnect or was an error when publishing the Feature Operations layer during the Ops Server install. I know @richgee finished his Ops Server install today and said that he didn't see any problem with Friendly Equipment being updated, but the Danish Esri distributor did notice that problem. (looking on [server], i notice that your Friendly Operations FC does not have a Situation Alerts layer, so it might have just been the process of re-creating the service that fixed the problem)

Might need a little more work to get to the bottom of why this is happening for some people...

pHill5136 commented 9 years ago

@kerryrobinson @garys-esri -- I'll check into this today. We have issues with that layer publishing in a feature service. It publishes to map service but not feature service.

Thanks for the info.

pHill5136 commented 9 years ago

@garys-esri @kerryrobinson @elinz @jrweakland, Kerry, since your issue seems to have been regarding the missing Situation Alerts, the problem may have been caused by the output pointing to the wrong index in the feature service. In our case the layer was not publishing because there were some problems with feature templates. Once updated the Situation Alerts layer is present in the service.

That did not fix the issue for me, though. I've been working with the geoevent team with this issue.and I noticed that the Update Feature Service output connector for Equipment (UpdateFriendlyEquipment) has 'Delete Old Features' property set to 'true'. A time field is required for this setting and it is set to 'Date Time Valid'. Most of the equipment messages do not have the date time valid field set so it is populating null in that field. I recreated the output service and set it to not delete old features. It has been running happily and updating for the last 3 hours. I'll keep monitoring the services, but I think there may be a validation check that was not present in 10.2 and earlier versions of 10.3. That would explain why this was not a problem before. Unfortunately, I was forced to recreate the output service for it to start behaving nicely.

Typically, our feature updates really do not need to have the Delete property set to true. If we do want to force a delete after some time period (deletes if no update has occurred in a given interval), I can make a modification to the geomessage adapter used to consume all of our military related events, such that if the incoming message does not set the Date Time Valid value, the adapter will set it with the current time.

-Patrick

garysheppardjr commented 9 years ago

@pHill5136, you sir have saved the day. I recreated the FriendlyOperationsUpdate output, making sure to set Delete Old Features to false, and it fixed it. It wasn't enough to edit the existing output; I had to delete and recreate it.

This is just in time, as I have a GeoEvent Extension presentation this afternoon. Thanks for the help!

Will this fix be made to the development server so that future Ops Server releases do not have this problem?

joebayles commented 9 years ago

:+1:

yehiahassan commented 9 years ago

Recreating the output connecter works only until the ArcGIS GeoEvent windows service restarted then recreating the output will be needed again.

I also tried removing the ArcGIS server from federated portal servers but the problem still exists. I also tried using the final ArcGIS 10.3 release instead of the daily build used with the TEM release, but unfortunately, this also didn't solve the problem.

this problem is not limited to the FriendlyOperationsUpdate output, all update features output connectors have the same problem.

by the way, I am using Windows Server 2012 R2.

Any Ideas?

kerryrobinson commented 9 years ago

@yehiahassan, I have noticed the same thing. Recreating the output fixed the problem with the friendly equipment not updating at all, but i find that after a while the updates stop again. I just restarted GEE and they are updating again. I will wait to see when they stop updating, will try to restart GEE to see if that temporarily fixed the problem or if I need to recreate the output.

@garys-esri Are you seeing anything similar?

garysheppardjr commented 9 years ago

@kerryrobinson, I did not notice the same thing, but I haven't used the Ops Server much lately, and I won't have time to use it much until after FedGIS.

pHill5136 commented 9 years ago

There are a couple issues here.

@yehiahassan when you restart GEE you will see an error state on any services that have a dependency on a data connection until portal is validated. Validation involves hitting all of the rest endpoints on the server to build a cache. This, depending on your system, takes anywhere between 20 - 45 minutes, after which the services will return (often in a stopped state). Make sure this is not what you are seeing. This problem caused by the large number of services we have deployed on the portal - i.e. it has nothing to do with the number of services in GEE. This is a known problem and an issue has been created for the GEE team to look into it for 10.3.1.

The next problem is the services stopping after a time. After this point, we need to recreate the output connector. One of the GEE developers has been assigned this issue. We think that the problem has something to do with postgres.

If you get a chance, open /etc/org.ops4j.pax.logging.cfg and set the following params:

log4j.appender.out.maxfilesize=10MB log4j.appender.out.maxBackupIndex=20

This will set the logfile size from 1MB to 10MB and create 20 instead of 10 log files before refreshing (only /data/log/karaf.log can be read inside manager the other files -- /data/log/karaf.log.1, etc. can be opened in a text editor and represent older logs that are deleted and refreshed based on the properties above). If you can isolate the time that tracking stops, see if you start getting an error similar to the following:

https://[server]:6443/arcgis/rest/services/Operations/FriendlyOperations/FeatureServer/11/query with parameters: where=uniquedesignation IN ('SupplyConvoy3','War Eagles 1 C','SupplyConvoy4','Wolfpack 1 D','Wolfpack 3 A','Saber 1 D','Shadow 4','23B-019','1A2-008','SupplyConvoy1','1A2-006','Supply In 2c','SupplyConvoy In 3A','Supply In 3C','Wolfpack 2 A','Wolfpack 2 B','Wolfpack 2 D','Wolfpack 3 C','War Eagles 1 B','Wolfpack 3 D','6A1-022','6A1-077','2C3-001','2C3-109','2C3-003','Alpha Forces','Grouper Alpha','LCU 25','Alpha George','Grouper Bravo','RPV UA25','NF26','USV25','NF25','Patrol Fox','3A1-001','3A1-003','SupplyConvoy In 2A','Wolfpack 2 C','Wolfpack 3 B','Cougars 1 D','Cougars 1 A','Cougars 1 B','War Eagles 1 A','Saber 1 B','Saber 1 A','Saber 1 C','Supply In 2b','Wolfpack 1 B','Cougars 1 C','War Eagles 1 D','Wolfpack 1 C','Shadow 1','3A1-004','3A2-002','3A1-006','3A2-001','1B2-002','1B2-008','3C2-007','2A3-009','26A-023','26A-015','26A-005','3B2-005','3B1-029','Shadow 2','Shadow 3')&outfields=uniquedesignation,objectid&f=json&token=ZcADgLuSxUYYF6Nel-GSHZVO5-FzffJ6L5AGrwREyCvXfyqYf66XYb-hVoIaq-u637ZhRPb4AtE6eH7cvGs41A6vjDj6KGLZLGNeFITO1LMJSfK-CCjD14wHt88_GyCmOVwhziJM6EoxHbFXZDckstnxstdJeaz1Hs4HSap3Dao63R8NemnvygD4PVBqlFkgJXnDs2624r_xzRDPKX2bq2h-Xz140X1ZcyZGnUoif1JUq6f7kRVif64GRGuMNVHtNwpk67bfj_FfBCGfZjr9Sg... java.io.IOException: null response at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.validateResponse(FeatureServiceOutboundTransport.java:745)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0] at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.performMissingOIDQuery(FeatureServiceOutboundTransport.java:452)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0] at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.queryForMissingOIDs(FeatureServiceOutboundTransport.java:433)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0] at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.writeToFeatureService(FeatureServiceOutboundTransport.java:153)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0] at com.esri.ges.transport.featureService.FeatureServiceOutboundTransport.receive(FeatureServiceOutboundTransport.java:84)[266:com.esri.ges.framework.transport.featureservice-transport:10.3.0] at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:42)[325:com.esri.ges.manager.internal-streammanager:10.3.0] at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184)[325:com.esri.ges.manager.internal-streammanager:10.3.0] at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44)[253:com.esri.ges.framework.adapter.adapter-api:10.3.0] at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0] at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91)[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0] at java.lang.Thread.run(Thread.java:744)[:1.7.0_51] 2014-12-12 10:32:14,511 | ERROR | ound/JSON/10.3.0 | FeatureServiceOutboundTransport | .FeatureServiceOutboundTransport 169 | 266 - com.esri.ges.framework.transport.featureservice-transport - 10.3.0 | Error while writing to feature service FriendlyOperations. Error: null response.

This will be confirmation that others are getting the same error that we are seeing.

What is happening is that the query above referenced in the error above is executed internally during a rest update operation. For reason's unknown, after querying successfully for some varying interval, that query is returned null and from that point forward GEE is no longer able to update until the output connector is deleted and recreated. If the query is executed via arcgis manager, it is successful.

-Patrick

yehiahassan commented 9 years ago

@pHill5136 my system has a really high specs where restarting the server and reaching the point where all outputs are in stopped state takes about 5-10 minutes, then I start the output connectors manually and it turns green successfully; so, I don't have a problem in this regards despite that I have to start them manually every restart time.

As of stopping the connectors update after a while, I also don't have this problem.

My problem is restarting the windows service for GeoEvent which would be either manually or by restarting the machine will directly stop the output connectors' updates despite that they are showing green live status and this is for all update feature connectors; recreating the connector with the same parameters solves the problem till the next GeoEvent service restart, where I'll have to re-create the connector again.

pHill5136 commented 9 years ago

I haven't not noticed that in my environment yet, but I suspect that it is related to the other issues we have been having. I will attempt to replicate this today. We have one of the GEE developers working on these issues and we have given him full access to our servers to diagnose (so far that has been the biggest impediment to troubleshooting the issue).

One thing from your side that may help is if you post this issue to the sp-geoevent alias. If the GEE developers -who watch the alias pretty closely - see that other people are having the same issues, it might expedite a solution.

-Patrick

jrweakland commented 9 years ago

@pHill5136 - added to the backlog: https://app.asana.com/0/22926133759983/31304190918340

conklinbd commented 9 years ago

This was resolved with the GEE Patch. Closing and updating Asana.