Closed SamoSlaus closed 11 months ago
This message suggests that the code successfully counted the number of records in the u_esm_ci_assoc table (228446) and then got an JSON parsing error while trying to read the first block of data records. My recommendation would be to turn debugging on and reduce the pagesize to a very small number to try to determine which row is failing. In my experience, this error generally occurs when there is a corrupted field in one of the records. It is usually a very large text field (more than 4000 bytes). If you can determine the sys_id of the corrupted record then you can use a filter to exclude it from the extraction. Corrupted records are rare, but I have seen cases where a single table had multiple corrupted records. SNDML cannot recover from a corrupt JSON payload. Since it cannot parse the payload, it cannot display the sys_id of the corrupt record. Unless your pagesize is 1, the displayed sys_id is unlikely to be the sys_id of the corrupt record. It is probably the sys_id of a different record which appeared earlier. Usually if you try to view the corrupted field via the ServiceNow UI in a form then the UI will either hang or get an error. Corrupted records are difficult to find and difficult to deal with. Another approach is to use increasing narrow creation date ranges in SNDML to eventually zero in on the 5 second interval when the record was created -- however, this is very time consuming as you can imagine. Another approach is to use a background script (javascript) to search for the corrupted record. Sometimes you can find them by searching for fields that are unexpectedly large or contain non-printing characters. Once you find the corrupted record you will likely conclude that it is useless, and the easiest thing may be to delete it using javascript.
Strange because the table we are trying to load is a m2m/association table so it is containing only sys_ids. For some reason the pagesize is ignored for this table it loads aprox. 50 records per step/commit, then it fails.
After increesing Transaction quota for: "REST Table API request timeout" we recieve following error: Exception in thread "main" java.io.EOFException: Unexpected end of ZLIB input stream at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:240) at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158) at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117) at org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:70) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.Reader.read(Reader.java:140) at org.apache.http.util.EntityUtils.toString(EntityUtils.java:227) at org.apache.http.util.EntityUtils.toString(EntityUtils.java:308) at sndml.servicenow.JsonRequest.executeRequest(JsonRequest.java:113) at sndml.servicenow.JsonRequest.execute(JsonRequest.java:44) at sndml.servicenow.RestTableAPI.getRecords(RestTableAPI.java:113) at sndml.servicenow.RestTableReader.call(RestTableReader.java:76) at sndml.servicenow.TableReader.getAllRecords(TableReader.java:306) at sndml.datamart.Synchronizer.getServiceNowTimestamps(Synchronizer.java:100) at sndml.datamart.Synchronizer.prepare(Synchronizer.java:66) at sndml.datamart.JobRunner.runSync(JobRunner.java:170) at sndml.datamart.JobRunner.call(JobRunner.java:96) at sndml.datamart.Loader.loadTables(Loader.java:129) at sndml.datamart.Loader.main(Loader.java:83)
Found out the issue, it was an ACL which slowed down API request and resulting an timeout. Thanks for the help.
We get followin error in logs: RestTableAPI [main] u_esm_ci_assoc PROCESS: getStats count=228446 query= Exception in thread "main" com.fasterxml.jackson.core.JsonParseException: Unexpected character ('}' (code 125)): was expecting a colon to separate field name and value at [Source: (String)"{"result":[{"sys_id":"000a51868779355083d10f6e8bbb35ed","sys_created_on":"2023-10-11 19:47:28","sys_updated_on":"2023-10-11