Open ro8inmorgan opened 5 months ago
We have the same probelm in Helm 0.64.205 and version 0.57.3. The connector level setting doesn't work. However, connection level works
We're still seeing the same thing when trying to override on the connector level (Helm chart version 0.293.4
/app version 0.63.8
).
Unable to override at the actor definition level in helm 0.453.2 and airbyte 0.63.19.
Here is my SQL request to update resourceRequirements:
UPDATE
"public"."actor_definition"
SET
resource_requirements = '{"jobSpecific": [{"jobType": "sync", "resourceRequirements": {"cpu_limit": "4", "cpu_request": "0.1", "memory_limit": "2Gi", "memory_request": "1Gi"}}]}'
WHERE
id IN (
SELECT
actor_definition_id
FROM
"public"."actor_definition_version"
WHERE
docker_repository LIKE 'airbyte/destination-bigquery' );
I need to do it on actor definition level because bigquery destination does not need much cpu but orchestrator and read pods do need much more.
Can confirm the issue for version 0.58.0. We are trying to set different resource requirements for sync and check/discover pods since they require different amount of resources, but unsuccessful
Helm Chart Version
0.67.17
What step the error happened?
During the Sync
Relevant information
I've set the resource_requirements for my connectors in the postgress database in table actor_definition. For example
{"jobSpecific": [{"jobType": "sync", "resourceRequirements": {"cpu_limit": "1", "cpu_request": "0.5", "memory_limit": "1Gi", "memory_request": "1Gi"}}]}
But all syncs revert to the default settings set in the airbyte-env configmap
No matter what I do the actor_defintion resource_requirements is never used. I would like to set a per connector resource base as these widely differ per connector.
For example logs below come from a connector set with the resource_requirements above as you can see it doesn't follow the resource_requirements but uses the settings from airbyte-env configmap which are:
JOB_MAIN_CONTAINER_CPU_LIMIT: "2" JOB_MAIN_CONTAINER_CPU_REQUEST: "1" JOB_MAIN_CONTAINER_MEMORY_LIMIT: 2Gi JOB_MAIN_CONTAINER_MEMORY_REQUEST: 2Gi
Relevant log output