Add privateKey property to the AbstractGitTask to support SSH key-based authentication for all Git tasks

corincorvus commented 4 months ago

Describe the issue

This Flow not work with the WorkingDirectory inputFiles. Before i used the deprecated LocalFiles and it worked fine.

id: dev-test
namespace: dev

tasks:
  - id: start
    type: io.kestra.plugin.core.flow.WorkingDirectory
    inputFiles:
      id_rsa: "test"

    tasks:
      - id: clone_repository
        type: io.kestra.plugin.git.SyncNamespaceFiles
        namespace: prod
        gitDirectory: _files # optional; set to _files by default
        delete: false # optional; by default, it's set to false to avoid destructive behavior
        url: "{{ globals['repository-url'] }}"
        branch: "{{ globals['repository-branch'] }}"
        username: "{{ globals['repository-username'] }}"
        password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
        dryRun: true

Log:

2024-06-07T06:40:15.026Z INFO Provided 1 input(s).
2024-06-07T06:40:16.187Z INFO Start cloning from 'https://gitlab.xy.xy/xy/xy'
2024-06-07T06:40:16.204Z ERROR Destination path "spgOiPTaewpXqQS8vgGL0" already exists and is not an empty directory

If i use the old "LocalFiles" it works:

id: dev-test
namespace: dev

tasks:
  - id: start
    type: io.kestra.plugin.core.flow.WorkingDirectory

    tasks:
      - id: clone_repository
        type: io.kestra.plugin.git.SyncNamespaceFiles
        namespace: prod
        gitDirectory: _files # optional; set to _files by default
        delete: false # optional; by default, it's set to false to avoid destructive behavior
        url: "{{ globals['repository-url'] }}"
        branch: "{{ globals['repository-branch'] }}"
        username: "{{ globals['repository-username'] }}"
        password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
        dryRun: true

      - id: load_ssh_key
        type: io.kestra.core.tasks.storages.LocalFiles
        inputs:
          id_rsa: "{{ secret('SSH_KEY') }}"

Log:

2024-06-07T06:43:30.430Z INFO Start cloning from 'https://gitlab.c-lab.one/c-lab/automatisierung'
2024-06-07T06:43:31.851Z INFO Provided 1 input(s).
2024-06-07T06:43:31.857Z INFO Captured 0 output(s).

Environment

Kestra Version: 0.17.1
Operating System (OS/Docker/Kubernetes): Debian/Docker

anna-geller commented 3 months ago

it seems that you authenticate with your Git VCS using an SSH key. We may explore adding sshKey property to the AbstractGitTask so that you don't need to use WorkingDirectory anymore:

id: dev-test
namespace: dev

tasks:
      - id: clone_repository
        type: io.kestra.plugin.git.SyncNamespaceFiles
        namespace: prod
        gitDirectory: _files # optional; set to _files by default
        delete: false # optional; by default, it's set to false to avoid destructive behavior
        url: "{{ globals['repository-url'] }}"
        branch: "{{ globals['repository-branch'] }}"
        username: "{{ globals['repository-username'] }}"
        password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
        dryRun: true
        sshKey: "{{ secret('SSH_KEY') }}"

anna-geller commented 3 months ago

and https://github.com/kestra-io/kestra/issues/3477

corincorvus commented 3 months ago

I use she "ssh_key" to add the ssh_key for ansible tasks for login on my target hosts. The Gitlab Token is another key (GITHUB_ACCESS_TOKEN in the secret file for kestra) and i dont have problems with gitlab. The Gitlab fails, cause the directory is already used. I think the new ssh_key on workindirectory method make the problem. At the moment i use localfiles for the ssh_key to use the ssh_key in the container, that have to connect to my targets. often i need other files too, like inventory of my gitlab. so i need a directory local for my container.

With the deprecated localfiles the ssh key provide after the repository.

I follow here the tutorial of christian lempa: timestamp: 12:30 https://www.youtube.com/watch?v=PJG1-7hMHsE&t=778

I need this private key for all scripts (ansbile and more) outside of kestra. otherwise the flow failed, cause its not allowed to connect the target.

for example:

id: update-apt-packages
namespace: lab-0

inputs:
  - id: host
    type: STRING
    required: true
    description: "Der Host oder die Gruppe, die verwendet werden soll"
    defaults:
      - "update"

  - id: playbook_update
    type: STRING
    required: true
    description: "Der Host oder die Gruppe, die verwendet werden soll"
    defaults:
      - "ansible/update/update-apt-packages.yaml"

  - id: playbook_autoremove
    type: STRING
    required: true
    description: "Der Host oder die Gruppe, die verwendet werden soll"
    defaults:
      - "ansible/update/update-apt-autoremove.yaml"

  - id: inventory
    type: STRING
    required: true
    defaults: 
      - "ansible/inventory/inventory.yaml"

tasks:
  - id: start-task # this Task download the repository
    type: io.kestra.plugin.core.flow.WorkingDirectory
    tasks:
      - id: clone-repository
        type: io.kestra.plugin.git.Clone
        url: "{{ globals['repository-url'] }}"
        branch: "{{ globals['repository-branch'] }}"
        username: "{{ globals['repository-username'] }}"
        password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"

      - id: load_ssh_key # this task add ssh_key for container
        type: io.kestra.core.tasks.storages.LocalFiles
        inputs:
          id_rsa: "{{ secret('SSH_KEY') }}"

      - id: update-servers # this task using repository inventory, playbook and ssh key to connect and update the targets
        type: io.kestra.plugin.ansible.cli.AnsibleCLI
        docker:
          image: cytopia/ansible:latest-tools
          pullPolicy: IF_NOT_PRESENT
          user: "1000"
        commands:
          - ansible-playbook -i "{{ inputs.inventory }}" --key-file id_rsa --extra-vars "hosts={{ inputs.host }}" "{{ inputs.playbook_update }}"
        env:
          ANSIBLE_HOST_KEY_CHECKING: "False"
          ANSIBLE_REMOTE_USER: "{{ globals['automation-ssh-user'] }}"

      - id: autoremove-old-packages
        type: io.kestra.plugin.ansible.cli.AnsibleCLI
        docker:
          image: cytopia/ansible:latest-tools
          pullPolicy: IF_NOT_PRESENT
          user: "1000"
        commands:
          - ansible-playbook -i "{{ inputs.inventory }}" --key-file id_rsa --extra-vars "hosts={{ inputs.host }}" "{{ inputs.playbook_autoremove }}"
        env:
          ANSIBLE_HOST_KEY_CHECKING: "False"
          ANSIBLE_REMOTE_USER: "{{ globals['automation-ssh-user'] }}"

triggers:
  - id: schedule_trigger
    type: io.kestra.plugin.core.trigger.Schedule
    cron: 0 22 * * *
    inputs:
      host: "update"

kestra-io / kestra

Add privateKey property to the AbstractGitTask to support SSH key-based authentication for all Git tasks #3950

Describe the issue

Environment