Status of testing of Apache Airflow Helm Chart 1.8.0rc1

[jedcunningham]

We have a kind request for all the contributors to the latest Apache Airflow Helm Chart 1.8.0rc1.

Could you please help us to test the RC versions of the Apache Airflow Helm Chart?

Please let us know in the comment if the issue is addressed in the latest RC.

• [x] Add nodePort declaration to chart/values.schema.json (#26945): @joseph-max-coalfire Linked issues:
  • Add NodePort Option to the values schema (#26812)
• [x] Make custom env vars optional for job templates (#27148): @richardbadman @tseruga Linked issues:
  • [Helm Chart] Decouple custom_airflow_environment from all pods and jobs (#26045)
• [x] Add workeer log-groomer-sidecar enable option in helm chart (#27178): @BobDu @jedcunningham Linked issues:
  • Adds support for custom command and args (#16153)
• [x] Chart: Default airflow version to 2.4.2 (#27230): @ephraimbuddy
• [x] Add webserverConfigConfigMapName (#27419): @JulesTriomphe
• [ ] ExtraVolumeMounts in sidecars and initContainers (#27420): @JulesTriomphe
• [x] allow pgbouncer replicas to be configurable (#27439): @pgvishnuram
• [x] [BUMP] helm chart - redis 7 (#27443): @raphaelauv
• [x] Chart: add hostaliases to pod template file (#27544): @Aakcht
• [x] Chart: Decrease default gitSync wait (#27625): @jedcunningham
• [x] Chart: Update default airflow version to 2.4.3 (#27668): @ephraimbuddy
• [x] Correct repo example for cloning via ssh - comments in helm chart (#27671): @schudoku @morkot Linked issues:
  • helm chart - mounting-dags-from-a-private-github-repo-using-git-sync-sidecar (#21970)
• [x] Replace helm chart icon (#27704): @ronahk
• [x] Chart: add airflow_local_settings to all airflow containers (#27779): @rauljmz @Aakcht @venkatnpedada Linked issues:
  • Not able to use custom logging with airflow helm chart (#27498)
  • Cannot reference airflow_local_settings in config settings (#25637)
• [x] Add global volume & volumeMounts to the chart (#27781): @csp33 Linked issues:
  • Allow global volume & volumeMounts in Helm chart (#27687)
• [x] git-sync 3.6.3 (#27848): @danielhoherd
• [x] Enhance chart to allow over-riding command-line args to statsd exporter (#28041): @rob-1126
• [x] Chart: Default airflow version to 2.5.0 (#28074): @ephraimbuddy
• [ ] Fix helm chart NOTES.txt to show correct URL (#28264): @okue
• [ ] Remove checks for 1.19 api checks (#28461): @potiuk @Jmdaming Linked issues:
  • Wrong apiVersion Detected During Ingress Creation (#22657)
• [x] Add missing annotations on deployments [#28604] (#28688): @sudeepgupta90 Linked issues:
  • Missing annotations on Deployments (#28604)
• [x] Add worker service account for LocalKubernetesExecutor (#28813): @Chen-Oliver
• [x] Fix helm chart create scheduler service to serve task logs for LocalKubernetesExecutor (#28828): @snjypl
• [x] Fix incorrect comments in worker-kedaautoscaler.yaml (#28858): @EricGao888
• [x] Chart: add doc note about podtemplate images (#29032): @Aakcht
• [ ] Upgrade bitnami/postgresql helm depenency from 10.5.3 to 12.1.9 (#29071): @snjypl
• [x] Update Airflow version to 2.5.1 (#29074): @pierrejeambrun
• [ ] Fix PgBouncer after postgres subchart upgrade (#29207): @snjypl @jedcunningham Linked issues:
  • Upgrade bitnami/postgresql helm depenency from 10.5.3 to 12.1.9 (#29071)

Thanks to all who contributed to the release (probably not a complete list!): @richardbadman @ronahk @pgvishnuram @sudeepgupta90 @Jmdaming @joseph-max-coalfire @Aakcht @tseruga @JulesTriomphe @morkot @rauljmz @pierrejeambrun @csp33 @raphaelauv @rob-1126 @jedcunningham @okue @venkatnpedada @Chen-Oliver @schudoku @EricGao888 @danielhoherd @BobDu @snjypl @potiuk @ephraimbuddy

[danielhoherd]

I have validated https://github.com/apache/airflow/pull/27848 which was easy to validate. It was just a version bump.

[EricGao888]

I have validated #28858 which fixed some comments in the source code.

[Aakcht]

Checked #27544 , #27779 and #29032 - all good.

[sudeepgupta90]

Checked #28604 https://github.com/apache/airflow/issues/28604 - all good

On Wed, Feb 1, 2023 at 10:46 AM Aakcht @.***> wrote:

Checked #27544 https://github.com/apache/airflow/pull/27544 , #27779 https://github.com/apache/airflow/pull/27779 and #29032 https://github.com/apache/airflow/pull/29032 - all good.

— Reply to this email directly, view it on GitHub https://github.com/apache/airflow/issues/29273#issuecomment-1411481922, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5XLD44466PUGRJMP5D6RLWVHWTJANCNFSM6AAAAAAUM3WRA4 . You are receiving this because you were mentioned.Message ID: @.***>

[csp33]

Checked #27781 Works as expected

[pgvishnuram]

Checked #27439 works as expected

[JulesTriomphe]

I have spotted multiple issues:

• There is a missing prefixing - here to cleanly render the volume mounts for the migrate-database-job, same here for the dag-processor, here, here and here for the scheduler, here and here for the triggerer, here and here for the webserver, here, here, here and here for the workers
• Globally defined volumes are missing in the pod-template-file which will make pod spawning fail on clusters where readOnlyRootFilesystem is True and a volume for airflow home has been set globally. Workaround would be to set that volume in each system's extraVolumes and extraVolumeMounts but that would greatly reduce the functionality of global volumes and volume mounts.
• There is a missing trailing - here to cleanly render the "airflow_dags_mount"

• There is a missing prefixing - here to cleanly render webserver service ports, and I didn't manage to clean the rendering of the fields themselves (name aligned with the dash). The current state looks like this:

  ports:
  
  - 
    name: airflow-ui
    port: 8080

I will try adding the missing globally defined volumes to the pod template file, deploy and then open a PR to correct this and another for the rendering issues.

[richardbadman]

Checked #27148, works as expected

[ronahk]

Checked #27704 https://github.com/apache/airflow/pull/27704 works as expected.

On Tue, Jan 31, 2023 at 10:09 PM Jed Cunningham @.***> wrote:

We have a kind request for all the contributors to the latest Apache Airflow Helm Chart 1.8.0rc1 https://dist.apache.org/repos/dist/dev/airflow/helm-chart/1.8.0rc1.

Could you please help us to test the RC versions of the Apache Airflow Helm Chart?

Please let us know in the comment if the issue is addressed in the latest RC.

• Add nodePort declaration to chart/values.schema.json (#26945) https://github.com/apache/airflow/pull/26945: @joseph-max-coalfire https://github.com/joseph-max-coalfire Linked issues:
  • Add NodePort Option to the values schema (#26812) https://github.com/apache/airflow/issues/26812
• Make custom env vars optional for job templates (#27148) https://github.com/apache/airflow/pull/27148: @richardbadman https://github.com/richardbadman @tseruga https://github.com/tseruga Linked issues:
  • [Helm Chart] Decouple custom_airflow_environment from all pods and jobs (#26045) https://github.com/apache/airflow/issues/26045
• Add workeer log-groomer-sidecar enable option in helm chart (#27178) https://github.com/apache/airflow/pull/27178: @BobDu https://github.com/BobDu @jedcunningham https://github.com/jedcunningham Linked issues:
  • Adds support for custom command and args (#16153) https://github.com/apache/airflow/pull/16153
• Chart: Default airflow version to 2.4.2 (#27230) https://github.com/apache/airflow/pull/27230: @ephraimbuddy https://github.com/ephraimbuddy
• Add webserverConfigConfigMapName (#27419) https://github.com/apache/airflow/pull/27419: @JulesTriomphe https://github.com/JulesTriomphe
• ExtraVolumeMounts in sidecars and initContainers (#27420) https://github.com/apache/airflow/pull/27420: @JulesTriomphe https://github.com/JulesTriomphe
• allow pgbouncer replicas to be configurable (#27439) https://github.com/apache/airflow/pull/27439: @pgvishnuram https://github.com/pgvishnuram
• [BUMP] helm chart - redis 7 (#27443) https://github.com/apache/airflow/pull/27443: @raphaelauv https://github.com/raphaelauv
• Chart: add hostaliases to pod template file (#27544) https://github.com/apache/airflow/pull/27544: @Aakcht https://github.com/Aakcht
• Chart: Decrease default gitSync wait (#27625) https://github.com/apache/airflow/pull/27625: @jedcunningham https://github.com/jedcunningham
• Chart: Update default airflow version to 2.4.3 (#27668) https://github.com/apache/airflow/pull/27668: @ephraimbuddy https://github.com/ephraimbuddy
• Correct repo example for cloning via ssh - comments in helm chart (#27671) https://github.com/apache/airflow/pull/27671: @schudoku https://github.com/schudoku @morkot https://github.com/morkot Linked issues:
  • helm chart - mounting-dags-from-a-private-github-repo-using-git-sync-sidecar (#21970) https://github.com/apache/airflow/issues/21970
• Replace helm chart icon (#27704) https://github.com/apache/airflow/pull/27704: @ronahk https://github.com/ronahk
• Chart: add airflow_local_settings to all airflow containers (#27779) https://github.com/apache/airflow/pull/27779: @rauljmz https://github.com/rauljmz @Aakcht https://github.com/Aakcht @venkatnpedada https://github.com/venkatnpedada Linked issues:
  • Not able to use custom logging with airflow helm chart (#27498) https://github.com/apache/airflow/issues/27498
  • Cannot reference airflow_local_settings in config settings (#25637) https://github.com/apache/airflow/issues/25637
• Add global volume & volumeMounts to the chart (#27781) https://github.com/apache/airflow/pull/27781: @csp33 https://github.com/csp33 Linked issues:
  • Allow global volume & volumeMounts in Helm chart (#27687) https://github.com/apache/airflow/issues/27687
• git-sync 3.6.3 (#27848) https://github.com/apache/airflow/pull/27848: @danielhoherd https://github.com/danielhoherd
• Enhance chart to allow over-riding command-line args to statsd exporter (#28041) https://github.com/apache/airflow/pull/28041: @rob-1126 https://github.com/rob-1126
• Chart: Default airflow version to 2.5.0 (#28074) https://github.com/apache/airflow/pull/28074: @ephraimbuddy https://github.com/ephraimbuddy
• Fix helm chart NOTES.txt to show correct URL (#28264) https://github.com/apache/airflow/pull/28264: @okue https://github.com/okue
• Remove checks for 1.19 api checks (#28461) https://github.com/apache/airflow/pull/28461: @potiuk https://github.com/potiuk @Jmdaming https://github.com/Jmdaming Linked issues:
  • Wrong apiVersion Detected During Ingress Creation (#22657) https://github.com/apache/airflow/issues/22657
• Add missing annotations on deployments [#28604] (#28688) https://github.com/apache/airflow/pull/28688: @sudeepgupta90 https://github.com/sudeepgupta90 Linked issues:
  • Missing annotations on Deployments (#28604) https://github.com/apache/airflow/issues/28604
• Add worker service account for LocalKubernetesExecutor (#28813) https://github.com/apache/airflow/pull/28813: @Chen-Oliver https://github.com/Chen-Oliver
• Fix helm chart create scheduler service to serve task logs for LocalKubernetesExecutor (#28828) https://github.com/apache/airflow/pull/28828: @snjypl https://github.com/snjypl
• Fix incorrect comments in worker-kedaautoscaler.yaml (#28858) https://github.com/apache/airflow/pull/28858: @EricGao888 https://github.com/EricGao888
• Chart: add doc note about podtemplate images (#29032) https://github.com/apache/airflow/pull/29032: @Aakcht https://github.com/Aakcht
• Upgrade bitnami/postgresql helm depenency from 10.5.3 to 12.1.9 (#29071) https://github.com/apache/airflow/pull/29071: @snjypl https://github.com/snjypl
• Update Airflow version to 2.5.1 (#29074) https://github.com/apache/airflow/pull/29074: @pierrejeambrun https://github.com/pierrejeambrun
• Fix PgBouncer after postgres subchart upgrade (#29207) https://github.com/apache/airflow/pull/29207: @snjypl https://github.com/snjypl @jedcunningham https://github.com/jedcunningham Linked issues:
  • Upgrade bitnami/postgresql helm depenency from 10.5.3 to 12.1.9 (#29071) https://github.com/apache/airflow/pull/29071

Thanks to all who contributed to the release (probably not a complete list!): @richardbadman https://github.com/richardbadman @ronahk https://github.com/ronahk @pgvishnuram https://github.com/pgvishnuram @sudeepgupta90 https://github.com/sudeepgupta90 @Jmdaming https://github.com/Jmdaming @joseph-max-coalfire https://github.com/joseph-max-coalfire @Aakcht https://github.com/Aakcht @tseruga https://github.com/tseruga @JulesTriomphe https://github.com/JulesTriomphe @morkot https://github.com/morkot @rauljmz https://github.com/rauljmz @pierrejeambrun https://github.com/pierrejeambrun @csp33 https://github.com/csp33 @raphaelauv https://github.com/raphaelauv @rob-1126 https://github.com/rob-1126 @jedcunningham https://github.com/jedcunningham @okue https://github.com/okue @venkatnpedada https://github.com/venkatnpedada @Chen-Oliver https://github.com/Chen-Oliver @schudoku https://github.com/schudoku @EricGao888 https://github.com/EricGao888 @danielhoherd https://github.com/danielhoherd @BobDu https://github.com/BobDu @snjypl https://github.com/snjypl @potiuk https://github.com/potiuk @ephraimbuddy https://github.com/ephraimbuddy

— Reply to this email directly, view it on GitHub https://github.com/apache/airflow/issues/29273, or unsubscribe https://github.com/notifications/unsubscribe-auth/AR36W4FYUKOGRX2WB5DSY3DWVFWPJANCNFSM6AAAAAAUM3WRA4 . You are receiving this because you were mentioned.Message ID: @.***>

[rob-1126]

Checked #28041 - works as expected.

[jedcunningham]

@JulesTriomphe, thanks for testing this out! Since the issue you found isn't a regression from 1.7.0, it'll end up going in the next helm chart release instead of blocking this one.

[joseph-max-coalfire]

Tested #26945 -- looks good, working as intended 💯

[JulesTriomphe]

@jedcunningham are you planning to release a patch version sometime soon to include these bug fixes ?

[JulesTriomphe]

Checked #27419, works as expected

[jedcunningham]

@JulesTriomphe, I fully intend to do releases more quickly than we have in the past. So "sometime soon" meaning in the next month, yes, in the next week or so, no.

[morkot]

27671 has been checked. Changes are there, as expected. You can tick that one off.

[JulesTriomphe]

Very cool, thanks for the clarification @jedcunningham !

[csp33]

Guess we can confirm that #27230, #27668, #28074 and #29074 are working✌🏻

[csp33]

27178 works as expected

[csp33]

27443 is also OK

[csp33]

All good in #27625

[csp33]

28813 and #28828 look good

[BobDu]

27178 is ok!

[jedcunningham]

Awesome, thanks @csp33! (I'd intended to remove all those core Airflow version bumps but missed it!)

[jedcunningham]

1.8.0 is out! Thanks everyone for taking the time to test the RC!