search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.77k stars 4.04k forks source link

Cannot pass environment variables to read and write pods #38158

Open onurmus opened 4 months ago

onurmus commented 4 months ago

Helm Chart Version

0.53.3 and 0.64.388

What step the error happened?

During the Sync

Relevant information

Hi! I need to pass HTTP_PROXY environment variables to my connector during sync. I add the environment variables to extraEnv or _envvars for the worker, global, and all the other sections in values.yaml file. As a result, these variables are passed to check and discover pods but not passed to the orchestrator, read and write pods. I checked the environment variables using kubectl describe pod and also tried squid proxy. I expect to see these variables in all pods. Here is an example configuration for worker in values.yaml:

env_vars:
    JOB_DEFAULT_ENV_HTTP_PROXY: loremIpsum
    JOB_DEFAULT_ENV_HTTPS_PROXY: loremIpsum
    JOB_DEFAULT_ENV_NO_PROXY: loremIpsum
    JOB_DEFAULT_ENV_JAVA_TOOL_OPTIONS: loremIpsum
extraEnv:
    - name: JOB_DEFAULT_ENV_HTTP_PROXY
      value: loremIpsum
......

Relevant log output

No response

ro8inmorgan commented 4 months ago

I noticed too that the orchestrator pod doesn't pick up enviroment vars and so doesn't pass them on to the read and write pods. Currently searching if I can find anything about the orchestrator pod to pick up the vars, but no luck yet.