figure out HttpMultipartRestClient issues

[jeanetteclark]

Investigate this note in the taskList.csv file

# Admin NOTE: it appears that DataONE HttpMultipartRestClient can't handle two clients being created at the same time, even if they are by different threads. This needs to be
# investigated further and potentially a bug needs to be logged in github for this. Until then, an easy workaround is to ensure that no two tasks are started
# at the same time, so adjust the cron schedule accordingly.

Since soon we are going to have more jobs than there are seconds in a minute, and it is already annoying to find a time to run a job. Related to #374 and might result in a ticket to d1_libclient_java depending on what I figure out when I try to replicate the error

[doulikecookiedough]

I believe this issue is still occurring, error logs below for reference (these were observed as feature-hashstore-support is being tested)

20241108-18:31:10: [ERROR]: quality-test-dataone-fair: error creating rest client: Cannot assign field "before" because "link.after" is null [edu.ucsb.nceas.mdqengine.scheduler.RequestReportJob:190]
20241108-18:31:10: [INFO]: Job metadig.quality-test-dataone-fair threw a JobExecutionException:  [org.quartz.core.JobRunShell:218]
org.quartz.JobExecutionException: java.lang.NullPointerException: Cannot assign field "before" because "link.after" is null [See nested exception: java.lang.NullPointerException: Cannot assign field "before" because "link.after" is null]
    at edu.ucsb.nceas.mdqengine.scheduler.RequestReportJob.execute(RequestReportJob.java:191)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
Caused by: java.lang.NullPointerException: Cannot assign field "before" because "link.after" is null
    at org.apache.commons.collections.map.AbstractLinkedMap.removeEntry(AbstractLinkedMap.java:294)
    at org.apache.commons.collections.map.AbstractHashedMap.removeMapping(AbstractHashedMap.java:543)
    at org.apache.commons.collections.map.AbstractHashedMap.remove(AbstractHashedMap.java:325)
    at org.apache.commons.configuration.BaseConfiguration.clearPropertyDirect(BaseConfiguration.java:133)
    at org.apache.commons.configuration.AbstractConfiguration.clearProperty(AbstractConfiguration.java:503)
    at org.apache.commons.configuration.CompositeConfiguration.clearPropertyDirect(CompositeConfiguration.java:269)
    at org.apache.commons.configuration.AbstractConfiguration.clearProperty(AbstractConfiguration.java:503)
    at org.apache.commons.configuration.AbstractConfiguration.setProperty(AbstractConfiguration.java:483)
    at org.dataone.client.rest.HttpMultipartRestClient.setDefaultTimeout(HttpMultipartRestClient.java:588)
    at org.dataone.client.rest.HttpMultipartRestClient.<init>(HttpMultipartRestClient.java:222)
    at org.dataone.client.rest.HttpMultipartRestClient.<init>(HttpMultipartRestClient.java:199)
    at org.dataone.client.rest.HttpMultipartRestClient.<init>(HttpMultipartRestClient.java:184)
    at edu.ucsb.nceas.mdqengine.scheduler.RequestReportJob.execute(RequestReportJob.java:188)
    ... 2 more

20241108-19:13:10: [ERROR]: quality-test-dataone-fair: error creating rest client: Cannot assign field "after" because "link.before" is null [edu.ucsb.nceas.mdqengine.scheduler.RequestReportJob:190]
20241108-19:13:10: [INFO]: Job metadig.quality-test-dataone-fair threw a JobExecutionException:  [org.quartz.core.JobRunShell:218]
org.quartz.JobExecutionException: java.lang.NullPointerException: Cannot assign field "after" because "link.before" is null [See nested exception: java.lang.NullPointerException: Cannot assign field "after" because "link.before" is null]
    at edu.ucsb.nceas.mdqengine.scheduler.RequestReportJob.execute(RequestReportJob.java:191)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
Caused by: java.lang.NullPointerException: Cannot assign field "after" because "link.before" is null
    at org.apache.commons.collections.map.AbstractLinkedMap.removeEntry(AbstractLinkedMap.java:293)
    at org.apache.commons.collections.map.AbstractHashedMap.removeMapping(AbstractHashedMap.java:543)
    at org.apache.commons.collections.map.AbstractHashedMap.remove(AbstractHashedMap.java:325)
    at org.apache.commons.configuration.BaseConfiguration.clearPropertyDirect(BaseConfiguration.java:133)
    at org.apache.commons.configuration.AbstractConfiguration.clearProperty(AbstractConfiguration.java:503)
    at org.apache.commons.configuration.CompositeConfiguration.clearPropertyDirect(CompositeConfiguration.java:269)
    at org.apache.commons.configuration.AbstractConfiguration.clearProperty(AbstractConfiguration.java:503)
    at org.apache.commons.configuration.AbstractConfiguration.setProperty(AbstractConfiguration.java:483)
    at org.dataone.client.rest.HttpMultipartRestClient.setDefaultTimeout(HttpMultipartRestClient.java:588)
    at org.dataone.client.rest.HttpMultipartRestClient.<init>(HttpMultipartRestClient.java:222)
    at org.dataone.client.rest.HttpMultipartRestClient.<init>(HttpMultipartRestClient.java:199)
    at org.dataone.client.rest.HttpMultipartRestClient.<init>(HttpMultipartRestClient.java:184)
    at edu.ucsb.nceas.mdqengine.scheduler.RequestReportJob.execute(RequestReportJob.java:188)
    ... 2 more

20241111-02:26:10: [ERROR]: quality-test-dataone-fair: error creating rest client: null [edu.ucsb.nceas.mdqengine.scheduler.RequestReportJob:190]
20241111-02:26:10: [INFO]: Job metadig.quality-test-dataone-fair threw a JobExecutionException:  [org.quartz.core.JobRunShell:218]
org.quartz.JobExecutionException: java.lang.NullPointerException [See nested exception: java.lang.NullPointerException]
    at edu.ucsb.nceas.mdqengine.scheduler.RequestReportJob.execute(RequestReportJob.java:191)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
Caused by: java.lang.NullPointerException