When internet connection is lost and batch of variants is not returned from variant validator, no error is presented

[marikaris]

Run the service, turn internet off, turn it back on.

Expected: For variants that were lost in the process, we get an error in the errorlog

Observed: They are lost in a void of nothingness

[dennishendriksen]

Notes:

• https://camel.apache.org/manual/latest/error-handling-in-camel.html
• Some routes have a completion timeout, but https://github.com/molgenis/data-transform-vkgl/blob/master/src/main/java/org/molgenis/core/MySpringBootRouter.java#L202 does not

[dennishendriksen]

issue

data-transform-vkgl appears to complete succesfully (result files and error files are produced, no errors are logged) but data is missing

hypothesis

this is caused by the setting of a 'completionTimeout' (based on completion.timeout=60000 in application.properties):

from("direct:validate")
    .routeId("validateRoute")
    .process()
    .body(Map.class, refValidator::validateOriginalRef)
    .aggregate(header(FILE_NAME), new GroupedBodyAggregationStrategy())
    .completionTimeout(fileCompletionTimeout)
    .process(geneValidator::getVariantsWithCorrectGenes)
    .to("direct:check_unique");

completionTimeout JavaDoc

Time in millis that an aggregated exchange should be inactive before its complete (timeout).

what happens

The route is inactive for one minute because of a HTTP request (including retries) that takes longer than one minute to complete succesfully.

solution

a) do not completionTimeout but complete in a robust way that guarantees that all input files were processed https://camel.apache.org/components/latest/eips/aggregate-eip.html#_about_completion

b) make sure that completionTimeout is larger than the total expected inactivity time, for example:

• http client
  • 5 retries
  • 60 seconds timeout --> completionTimeout=300s + overhead

connectionsPerRoute 1 maxTotalConnections 1 connectionRequestTimeout -1 connectTimeout 10000 socketTimeout 60000