Setting Partitions in Data Pump Results in Critical Error

[AJStacy]

When attempting to sync a large table (such as task) we are getting long running jobs. In an attempt to optimize it, we followed the instructions for using partitions in the article "Optimizing Database Exports with DataPump".

When a job has any partition configured, SNDML3 throws the following error:

ERROR DPRUN0035136: Critical error detected. Halting JVM.
java.lang.AssertionError
java.lang.AssertionError: null
    at sndml.datamart.DatePartitionedTableReader.prepare(DatePartitionedTableReader.java:87) ~[sndml-3.4.7-pg.jar:?]
    at sndml.datamart.JobRunner.runSync(JobRunner.java:179) ~[sndml-3.4.7-pg.jar:?]
    at sndml.datamart.JobRunner.call(JobRunner.java:96) ~[sndml-3.4.7-pg.jar:?]
    at sndml.daemon.AppJobRunner.call(AppJobRunner.java:85) ~[sndml-3.4.7-pg.jar:?]
    at sndml.daemon.AppJobRunner.run(AppJobRunner.java:60) ~[sndml-3.4.7-pg.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
    at java.lang.Thread.run(Unknown Source) [?:?]

I have tracked it to the DatePartitionedTableReader.java src file at line 87 where it is asserting that the dependency named RecordWriter has been injected and is not null.

    @Override
    public void prepare(RecordWriter writer, Metrics metrics, ProgressLogger progress) 
            throws IOException, InterruptedException {
        super.beginPrepare(writer, metrics, progress);
                 assert writer != null; // <-- This line

The error message does not give me enough detail to know if it's something I have configured incorrectly or if this is a bug.

[gflewis]

Please provide the following

1. What command line options are you using?
2. Please provide a copy of the connection profile with URLs and passwords commented out.
3. Please provide a a copy of the JSON from the Job Run (x_108443_sndml_action_run) record. The JSON is in a hidden field. You can see the JSON if you select "Show XML" from the context menu. The number should be DPRUN0035136.

[AJStacy]

Thanks for the speedy response. We are running SNDML3 in daemon mode inside of a docker container with the following command:

java -ea -Dlog4j2.configurationFile=/usr/src/sndml3/log4j2-daemon.xml -Dsndml.logFolder=/usr/src/sndml3/log -Dsndml.logPrefix=SNDML3-Daemon -jar /usr/src/sndml3/sndml-3.4.7-pg.jar --profile=/run/secrets/connection.properties --daemon

Here is the connection.properties contents:

datamart.url=jdbc:postgresql://postgres:5432/servicenow
datamart.schema=public
datamart.autocreate=true
daemon.scope=x_108443_sndml
daemon.agent=main
daemon.interval=60
daemon.threads=3
daemon.continue=false
servicenow.instance=***********
servicenow.username=***********
servicenow.password=***********
datamart.username=***********
datamart.password=***********

As for the JSON output, I'm not finding it on the job run. The job never gets past the "prepare" state because of the error thrown by the SNDML3 daemon. Here is what I see:

[gflewis]

To see the JSON, use the hamburger menu and select "Show XML". (This will probably require admin role).

[gflewis]

Version 3.4.7 was never tagged or released. You must have built the JAR from the downloaded source code. It will be difficult to track down this bug without knowing the commit point.

There has been a huge amount of refactoring and restructuring of the code since 3.4.7. It will be difficult to go back that far, and the bug that you have found may or may not still exist. Are you interested in helping to beta test 3.5?

Here is a summary of what is in 3.5:

• https://github.com/gflewis/sndml3/wiki/~-Release-3.5

[AJStacy]

I wouldn't mind testing it out. We are currently in a time-sensitive rollout of ServiceNow (we are migrating from a different system) so I can't do much tinkering until after we've deployed.

Also, the "Show XML" option is not available to me for the job I was having issues with. I don't know if that's because of permissions issues or because the job never processed.

[gflewis]

Show XML is a global function which should be available from the Form Context Menu for any record in the ServiceNow intance, but it requires admin role. See Form context menu.

3.5 is a very significant release, and the code is still a bit unstable. Could you help with the testing after your rollout is behind you? Please reach out to me on LinkedIn and we can discuss.

Because of all the water that has gone under the bridge, it would be pretty hard at this point to track down and fix a 3.4.7 bug, so I am going to close this thread.

In the interim, if you need to pull a large amount of data from ServiceNow into PostgreSQL, I recommend the command line (YAML) interface and release 3.4.6. Thanks.