MI-DPLA / combine

Combine /kämˌbīn/ - Metadata Aggregator Platform
MIT License
26 stars 11 forks source link

Add timeout configurations to WSULib Ingestion 3 fork #250

Closed ghukill closed 6 years ago

ghukill commented 6 years ago

Currently, Combine is using a fork of DPLA's Ingestion3 engine: https://github.com/WSULib/ingestion3

And is pulling this release: https://github.com/WSULib/ingestion3/releases/tag/combine_spark-2.1.0_livy-0.5.0

Consider adding these modifications to .../src/main/scala/dpla/ingestion3/utils/HttpUtils.scala to new release, and align Combine-Playbook build with that:

  protected lazy val timeout = 240 // TODO load from config?
  protected lazy val retryMax = 10
  protected lazy val backoffMax = 500

  protected lazy val httpClient: OkHttpClient = new OkHttpClient.Builder()
    .connectTimeout(timeout, TimeUnit.SECONDS)
    .readTimeout(timeout, TimeUnit.SECONDS) 
    .retryOnConnectionFailure(true)
    .followRedirects(true)
    .build()

where readTimeout method is new, and the values for val timeout and val retryMax have been bumped.

ghukill commented 6 years ago

Done: https://github.com/WSULib/combine-playbook/commit/4bfe726df81465e75aa90ad7908b8b2a733a4c60