Open ivan-sukhomlyn opened 3 weeks ago
Thanks for the request @ivan-sukhomlyn I included to the platform team backlog.
@davinchia now without the limit of reading large catalog maybe this is something necessary to make possible to now OOM during the discover schema.
Topic
worker config
Relevant information
Could you extend discover jobs configuration at the
worker
side in the same manner as for check jobs with the possibility to define custom resources instead of default ones?The current behavior leads to overprovisioning of the Kubernetes cluster with such higher resource requests than usually needed for replication jobs
For example, the
check
job has this possibility - https://github.com/airbytehq/airbyte-platform/blob/main/airbyte-workers/src/main/resources/application.yml#L151But there's no for discover ones - https://github.com/airbytehq/airbyte-platform/blob/main/airbyte-workers/src/main/resources/application.yml#L154
Proposal