airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.75k stars 4.04k forks source link

WEBAPP_URL not overriding webhooks notifications #33942

Closed joeybenamy closed 5 months ago

joeybenamy commented 9 months ago

Helm Chart Version

0.50.20

What step the error happened?

Other

Revelant information

In helm values we are setting webapp.url to our public URL in the format "https://......" and it is properly being set in the configmap, but the webhook notifications still show the internal URL. We also see that on the airbyte-server pod, WEBAPP_URL is being set properly.

Relevant log output

No response

HamiltonHord commented 8 months ago

+1 here.

I've confirmed that the webapp pod is getting the proper ENV var for the public url, but doesn't seem to do anything with it. Notifications just come up with the default URL.

In addition to setting the value webapp.url in the helm values, which does seem to update the config map, I noticed that with that setting alone, I didn't see the ENV var for WEBAPP_URL in the env for the webapp pod. I then tried again by setting the webapp.extraEnv in the values.yaml to:

extraEnv:
  - name: WEBAPP_URL
    valueFrom:
      configMapKeyRef:
        name: airbyte-env
        key: WEBAPP_URL

This change did show that it was pulling the expected url value, and was showing up properly within the pod. Ultimately though, the notifications still weren't using the URL in this value.

We're using v0.50.20 as well.

joeybenamy commented 8 months ago

+1 here.

I've confirmed that the webapp pod is getting the proper ENV var for the public url, but doesn't seem to do anything with it. Notifications just come up with the default URL.

In addition to setting the value webapp.url in the helm values, which does seem to update the config map, I noticed that with that setting alone, I didn't see the ENV var for WEBAPP_URL in the env for the webapp pod. I then tried again by setting the webapp.extraEnv in the values.yaml to:

extraEnv:
  - name: WEBAPP_URL
    valueFrom:
      configMapKeyRef:
        name: airbyte-env
        key: WEBAPP_URL

This change did show that it was pulling the expected url value, and was showing up properly within the pod. Ultimately though, the notifications still weren't using the URL in this value.

We're using v0.50.20 as well.

Thanks for confirming this issue and providing the great observation! I hope it will be fixed soon.

nataliekwong commented 7 months ago

@malikdiarra I recall you were working on this area - did this end up getting resolved?

malikdiarra commented 7 months ago

Yes, this should be fixed for the following kind of notifications:

Some other minor notifications are not fixed just yet. Let's keep that bug open until they are; I'll provide a list.

malikdiarra commented 7 months ago

Remaining notifications to address:

joeybenamy commented 6 months ago

Remaining notifications to address:

  • [ ] Warning before a connection is disabled because it failed too many times.
  • [ ] Connection disabled because it failed too many times.

Do we still have to set this with an env var or is this exposed as a helm value?

malikdiarra commented 6 months ago

Hi @joeybenamy, it still requires setting up an env var. We want to expose it as an helm value but we are not working on that at the moment.

joeybenamy commented 6 months ago

Hi @joeybenamy, it still requires setting up an env var. We want to expose it as an helm value but we are not working on that at the moment.

Understood. Thank you!

joeybenamy commented 6 months ago

+1 here.

I've confirmed that the webapp pod is getting the proper ENV var for the public url, but doesn't seem to do anything with it. Notifications just come up with the default URL.

In addition to setting the value webapp.url in the helm values, which does seem to update the config map, I noticed that with that setting alone, I didn't see the ENV var for WEBAPP_URL in the env for the webapp pod. I then tried again by setting the webapp.extraEnv in the values.yaml to:

extraEnv:
  - name: WEBAPP_URL
    valueFrom:
      configMapKeyRef:
        name: airbyte-env
        key: WEBAPP_URL

This change did show that it was pulling the expected url value, and was showing up properly within the pod. Ultimately though, the notifications still weren't using the URL in this value.

We're using v0.50.20 as well.

This worked for me with some modifications since we are wrapping the airbyte helm chart in a custom chart for our custom ingress. We're also using helmfile and go templates. But in the end, the same approach worked for us. We still set webapp.url but now we explicitly set the WEBAPP_URL env var in the webapp to the value from the configmap via the configmap key reference.