Consider adding these modifications to .../src/main/scala/dpla/ingestion3/utils/HttpUtils.scala to new release, and align Combine-Playbook build with that:
protected lazy val timeout = 240 // TODO load from config?
protected lazy val retryMax = 10
protected lazy val backoffMax = 500
protected lazy val httpClient: OkHttpClient = new OkHttpClient.Builder()
.connectTimeout(timeout, TimeUnit.SECONDS)
.readTimeout(timeout, TimeUnit.SECONDS)
.retryOnConnectionFailure(true)
.followRedirects(true)
.build()
where readTimeout method is new, and the values for val timeout and val retryMax have been bumped.
Currently, Combine is using a fork of DPLA's Ingestion3 engine: https://github.com/WSULib/ingestion3
And is pulling this release: https://github.com/WSULib/ingestion3/releases/tag/combine_spark-2.1.0_livy-0.5.0
Consider adding these modifications to
.../src/main/scala/dpla/ingestion3/utils/HttpUtils.scala
to new release, and align Combine-Playbook build with that:where
readTimeout
method is new, and the values forval timeout
andval retryMax
have been bumped.