Syncing dags from multiple git-repos

[somasays]

Is it possible to sync from several repositories into the dags foder using git-sync, example below. The intention being, to have a sort of multi-tenent aiflow cluster, where dags can be deployed from several git repositories

extraContainers:
    - name: gitsync-repo-1
      image: k8s.gcr.io/git-sync/git-sync:v3.2.2
      imagePullPolicy: IfNotPresent
      env:
        - name: GIT_SYNC_REPO
          value: https://github.com/company/repo-1.git
        - name: GIT_SYNC_ROOT
          ......
      volumeMounts:
        - mountPath: /usr/local/airflow/dags
          name: dags
          readOnly: false
      extraVolumeMounts:
        - name: dags
          mountPath: /usr/local/airflow/dags
      extraVolumes:
        - name: dags
          emptyDir: {}

    - name: gitsync-repo-2
      image: k8s.gcr.io/git-sync/git-sync:v3.2.2
      imagePullPolicy: IfNotPresent
      env:
        - name: GIT_SYNC_REPO
          value: https://github.com/company/repo-2.git
        - name: GIT_SYNC_ROOT
          ......
      volumeMounts:
        - mountPath: /usr/local/airflow/dags
          name: dags
          readOnly: false
      extraVolumeMounts:
        - name: dags
          mountPath: /usr/local/airflow/dags
      extraVolumes:
        - name: dags
          emptyDir: {}

[jedcunningham]

Yes you could, you'd just add extra extraInitContainers and extraContainers. The extraVolumes portion would only need to be done once.

The example on the readme (which it looks like you may have already found) has the right structure (your indentation is off for extraVolumes): https://github.com/astronomer/airflow-chart#deploying-dags-using-git-sync

One word of caution though: Airflow has limited support for multi-tenancy, so if you have any real security needs you are better off with separate instances anyways.

[somasays]

Thank you for the clarification. By multi-tenant I meant that one team managing several projects and using the same cluster, where each project corresponds to a single repo

[danielhoherd]

We do not have any plans to support that feature in this chart. Astronomer handles multi-tenancy at the Astronomer app layer, and each tenant can have multiple airflow deployments, each with its own git-sync configurations.