drone-ssh command never returns in a particular task in a Tekton pipeline run

[yeeyaa]

We are using drone-ssh in our tekton pipeline for remote execution of some commands on our servers. As advised by the tekton community: https://hub.tekton.dev/tekton/task/remote-ssh-commands.

However, in one of our tasks, the drone-ssh command is never returned which is causing the task to hang and eventually timeout.

From the log output, we can see the scripts passed to the drone-ssh command have been successfully executed. But it's just not returning and ending the drone-ssh command.

Could you please help identify where the issue might be? I'm using latest tag.

Same issue has been reported to tekton as well: https://github.com/tektoncd/catalog/issues/1277

Task configuration in the pipeline:

    - displayName: Install PAC - BRT
      name: task-install-pac-brt-cluster
      params:
        - name: HOST
          value: $(params.pac-fyre-host)
        - name: USERNAME
          value: $(params.default-brt-login-user)
        - name: SSH_SCRIPT
          value: |
            #!/usr/bin/bash
            $(params.path-to-pipeline-scripts)/install_lsf_brt.sh
            sleep 60
            $(params.path-to-pipeline-scripts)/install_pac_brt.sh
      runAfter:
        - task-build-hpc-package
        - copy-repo-to-remote-host
      taskRef:
        kind: Task
        name: remote-ssh-commands
      timeout: 30m0s
      when:
        - input: $(params.should-run-install-pac-brt-cluster)
          operator: in
          values:
            - 'yes'
            - 'y'
            - 'YES'
            - 'yes'
      workspaces:
        - name: credentials

Task definition, as shown in the tekton catalog source here: https://github.com/tektoncd/catalog/blob/main/task/remote-ssh-commands/0.1/remote-ssh-commands.yaml

[yeeyaa]

I'd done some further tests today and it turns out even if I am running the drone-ssh binary on its own, the session won't close. Here's what I tested.

1. Defined a script environment variable using

   # export script="/path/to/my/script.sh
   echo 'Finished script'
   "

2. Then I am running the latest drone-ssh to get connect to my remote host and run the script (which is present on the remote host).

   ./drone-ssh -H my-host -p 22 -u root -T 480m -s "$script" -i /path/to/my/id_ed25519 

Then I am able to execute the script.sh on the remote host. I can see from the console it reaches to the end of the script. However, it was at this point the session won't return and the drone-ssh process looking like hanging and doing nothing.

[yeeyaa]

Not sure if this helps but it's some test I did.

I did another test, which in the first terminal window the drone-ssh runs to the end of the remote script. I opened another terminal window and executed same command. Then after a couple of seconds, the drone-ssh process in the first window successfully returned and runs to a complete state.

But then the second window would run to the end and kept hanging there.