search

AbsaOSS / enceladus

Dynamic Conformance Engine
Apache License 2.0
31 stars 14 forks source link

SparkJobs working with LoadBalanced Menas #1869

Closed Zejnilovic closed 2 years ago

Zejnilovic commented 3 years ago

Background

When we move to the cloud, we will not have 2 URLs for Menas, there will be a proper LB. That means the CrossHostApiCaller setup will not work. This means that if there is a glitch in the network once, there will be no retries, maybe even for long jobs no new authentication.

Feature

Figure a strategy for retries even if CrossHostApiCaller gets only one URL and implement a solution

Proposed Solution [Optional]

Tell CrossHostApiCaller to retry "all" URLs X times. Be able to set the number of all retries in configuration. The default can be 1 or 2.

benedeki commented 2 years ago

Release notes: It's now possible to set the retry count of each Menas API url. and how they are approached (equal and chosen randomly versus the first as primary and the followups being backups).