Failed to push workdir to the guest when the provision is beaker

[guoguojenna]

$ 
summary: Demo
discover:
  - name: demo
    how: fmf
    url: https://gitlab.com/ZhaojuanGuo/kernel-tests.git
    ref: 523-rdma
    test:
      - /rdma/opa-fm
provision:
    how: beaker
    image: RHEL-9.3.0-20230821.32
    hardware:
        hostname: "rdma-cki-04.rdma.lab.eng.rdu2.redhat.com"
    key: "/home/zguo/.ssh/id_rsa"
    user: root
    password: QwAo2U6GRxyNPKiZaOCx
execute:
    how: tmt

$ tmt run plans --name /rdma/plans/example-provision-beaker -d -vvv
/var/tmp/tmt/run-023
    warn: /rdma/plans/example-provision-beaker:provision - {'how': 'beaker', 'image': 'RHEL-9.3.0-20230821.32', 'hardware': {'hostname': 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com'}, 'key': '/home/zguo/.ssh/id_rsa', 'user': 'root', 'password': 'QwAo2U6GRxyNPKiZaOCx'} is not valid under any of the given schemas
Workdir '/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker' created.

/rdma/plans/example-provision-beaker
summary: Demo
    Workdir '/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/discover' created.
    Workdir '/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/provision' created.
    Workdir '/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/prepare' created.
    Workdir '/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/execute' created.
    Workdir '/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/report' created.
    Report step always force mode enabled.
    Workdir '/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/report' created.
    Workdir '/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/finish' created.
    discover
    workdir: /var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/discover
        how: fmf
        name: demo
        order: 50
        Workdir '/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/discover/demo' created.
        url: https://gitlab.com/ZhaojuanGuo/kernel-tests.git
        Clone 'https://gitlab.com/ZhaojuanGuo/kernel-tests.git' to '/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/discover/demo/tests'.
        ref: 523-rdma
        Checkout ref '523-rdma'.
        hash: 10c24c72
        tests: /rdma/opa-fm
        Check metadata tree in '/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/discover/demo/tests'.
        summary: 1 test selected
            /rdma/opa-fm
    provision
    workdir: /var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/provision
        Workdir '/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/provision/default-0' created.
        how: beaker
        order: 50
        hardware: hostname: rdma-cki-04.rdma.lab.eng.rdu2.redhat.com

        image: RHEL-9.3.0-20230821.32
        Transformed hardware:
            and:
              - hostname:
                    _op: ==
                    _value: rdma-cki-04.rdma.lab.eng.rdu2.redhat.com
        guest: has been requested
        job id: rdma-cki-04.rdma.lab.eng.rdu2.redhat.com
        wait: waiting for condition 'get_new_state' with timeout 1:00:00, deadline in 3600.0 seconds, checking every 60.00 seconds
        status: Waiting        wait: 'get_new_state' still pending, 3598.26 seconds left, current tick 60.00 seconds
        status: Installing        wait: 'get_new_state' still pending, 3536.70 seconds left, current tick 60.00 seconds
        status: Installing        wait: 'get_new_state' still pending, 3474.87 seconds left, current tick 60.00 seconds
        status: Installing        wait: 'get_new_state' still pending, 3413.12 seconds left, current tick 60.00 seconds
        status: Installing        wait: 'get_new_state' still pending, 3351.29 seconds left, current tick 60.00 seconds
        status: Installing        wait: 'get_new_state' still pending, 3289.22 seconds left, current tick 60.00 seconds
        status: Installing        wait: 'get_new_state' still pending, 3227.26 seconds left, current tick 60.00 seconds
        status: Installing        wait: 'get_new_state' still pending, 3165.52 seconds left, current tick 60.00 seconds
        status: Installing        wait: 'get_new_state' still pending, 3103.26 seconds left, current tick 60.00 seconds
        status: Installing        wait: 'get_new_state' still pending, 3041.59 seconds left, current tick 60.00 seconds
        status: Installing        wait: 'get_new_state' still pending, 2979.86 seconds left, current tick 60.00 seconds
        status: Installing        wait: 'get_new_state' still pending, 2918.22 seconds left, current tick 60.00 seconds
        status: Installing        wait: 'get_new_state' still pending, 2856.37 seconds left, current tick 60.00 seconds
        status: Installing        wait: 'get_new_state' still pending, 2794.62 seconds left, current tick 60.00 seconds
        status: Waiting           wait: 'get_new_state' still pending, 2732.88 seconds left, current tick 60.00 seconds
        status: Waiting        wait: 'get_new_state' still pending, 2671.13 seconds left, current tick 60.00 seconds
        status: Waiting        wait: 'get_new_state' still pending, 2609.38 seconds left, current tick 60.00 seconds
        status: Waiting        wait: 'get_new_state' still pending, 2547.63 seconds left, current tick 60.00 seconds
        status: Reserved        wait: 'get_new_state' finished successfully, 2485.89 seconds left

        address: rdma-cki-04.rdma.lab.eng.rdu2.redhat.com
        multihost name: default-0
        Create the master ssh connection: ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oServerAliveInterval=60 -oServerAliveCountMax=5 -S/run/user/1000/tmt/tmp7vl8n74z -MNnT root@rdma-cki-04.rdma.lab.eng.rdu2.redhat.com
        Execute command 'export TMT_PLAN_DATA=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/data; export TMT_TREE=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/tree; cat /etc/os-release' on guest 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com'.
        Workdir '/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/provision/default-0' already exists.
root@rdma-cki-04.rdma.lab.eng.rdu2.redhat.com's password:         Execute command 'export TMT_PLAN_DATA=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/data; export TMT_TREE=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/tree; cat /etc/lsb-release' on guest 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com'.
        Execute command 'export TMT_PLAN_DATA=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/data; export TMT_TREE=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/tree; arch' on guest 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com'.
        query: Command 'arch' produced no usable output.
        Execute command 'export TMT_PLAN_DATA=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/data; export TMT_TREE=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/tree; cat /etc/redhat-release' on guest 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com'.
        query: Command 'cat /etc/redhat-release' produced no usable output.
        Execute command 'export TMT_PLAN_DATA=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/data; export TMT_TREE=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/tree; cat /etc/fedora-release' on guest 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com'.
        query: Command 'cat /etc/fedora-release' produced no usable output.
        Execute command 'export TMT_PLAN_DATA=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/data; export TMT_TREE=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/tree; uname -r' on guest 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com'.
        query: Command 'uname -r' produced no usable output.
        Execute command 'export TMT_PLAN_DATA=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/data; export TMT_TREE=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/tree; stat /run/ostree-booted' on guest 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com'.
        Execute command 'export TMT_PLAN_DATA=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/data; export TMT_TREE=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/tree; dnf5 --version' on guest 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com'.
        Execute command 'export TMT_PLAN_DATA=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/data; export TMT_TREE=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/tree; dnf --version' on guest 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com'.
        Execute command 'export TMT_PLAN_DATA=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/data; export TMT_TREE=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/tree; yum --version' on guest 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com'.
        Execute command 'export TMT_PLAN_DATA=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/data; export TMT_TREE=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/tree; cat /proc/filesystems' on guest 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com'.
        Execute command 'export TMT_PLAN_DATA=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/data; export TMT_TREE=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/tree; whoami' on guest 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com'.
        arch: unknown
        distro: unknown
        kernel: unknown
        package manager: unknown
        selinux: no
        is superuser: no
        summary: 1 guest provisioned
    prepare
    workdir: /var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/prepare
        queued push task #1: push on default-0

        push task #1: push on default-0
    Push workdir to guest 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com'.
    Ensure that rsync is installed on the guest.
    Execute command 'export TMT_PLAN_DATA=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/data; export TMT_TREE=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/tree; rsync --version' on guest 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com'.
        cmd: rsync --version
        err: Warning: Permanently added 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com' (ED25519) to the list of known hosts.
        err: ssh_askpass: exec(/usr/libexec/openssh/gnome-ssh-askpass): No such file or directory
        err: Permission denied, please try again.
        err: ssh_askpass: exec(/usr/libexec/openssh/gnome-ssh-askpass): No such file or directory
        err: Permission denied, please try again.
        err: ssh_askpass: exec(/usr/libexec/openssh/gnome-ssh-askpass): No such file or directory
        err: root@rdma-cki-04.rdma.lab.eng.rdu2.redhat.com: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
    Check the package manager.
    Execute command 'export TMT_PLAN_DATA=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/data; export TMT_TREE=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/tree; dnf --version' on guest 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com'.
        cmd: dnf --version
        err: Warning: Permanently added 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com' (ED25519) to the list of known hosts.
        err: ssh_askpass: exec(/usr/libexec/openssh/gnome-ssh-askpass): No such file or directory
        err: Permission denied, please try again.
        err: ssh_askpass: exec(/usr/libexec/openssh/gnome-ssh-askpass): No such file or directory
        err: Permission denied, please try again.
        err: ssh_askpass: exec(/usr/libexec/openssh/gnome-ssh-askpass): No such file or directory
        err: root@rdma-cki-04.rdma.lab.eng.rdu2.redhat.com: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
    Check for a read-only distro.
    Execute command 'export TMT_PLAN_DATA=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/data; export TMT_TREE=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/tree; rpm-ostree --version' on guest 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com'.
        cmd: rpm-ostree --version
        err: Warning: Permanently added 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com' (ED25519) to the list of known hosts.
        err: ssh_askpass: exec(/usr/libexec/openssh/gnome-ssh-askpass): No such file or directory
        err: Permission denied, please try again.
        err: ssh_askpass: exec(/usr/libexec/openssh/gnome-ssh-askpass): No such file or directory
        err: Permission denied, please try again.
        err: ssh_askpass: exec(/usr/libexec/openssh/gnome-ssh-askpass): No such file or directory
        err: root@rdma-cki-04.rdma.lab.eng.rdu2.redhat.com: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
    Execute command 'export TMT_PLAN_DATA=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/data; export TMT_TREE=/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/tree; yum install -y rsync' on guest 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com'.
        cmd: yum install -y rsync
        err: Warning: Permanently added 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com' (ED25519) to the list of known hosts.
        err: ssh_askpass: exec(/usr/libexec/openssh/gnome-ssh-askpass): No such file or directory
        err: Permission denied, please try again.
        err: ssh_askpass: exec(/usr/libexec/openssh/gnome-ssh-askpass): No such file or directory
        err: Permission denied, please try again.
        err: ssh_askpass: exec(/usr/libexec/openssh/gnome-ssh-askpass): No such file or directory
        err: root@rdma-cki-04.rdma.lab.eng.rdu2.redhat.com: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
    fail: Failed to push workdir to the guest. This usually means that login as 'root' to the guest does not work.
    fail: Command 'yum install -y rsync' returned 255.
    finish
    workdir: /var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/finish

    Prune '/rdma/plans/example-provision-beaker' plan workdir '/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker'.
        Workdir '/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/execute/default-0' created.
        Workdir '/var/tmp/tmt/run-023/rdma/plans/example-provision-beaker/report/default-0' created.
        summary: 0 tasks completed

plan failed

The exception was caused by 1 earlier exceptions

Cause number 1:

    prepare step failed
    The exception was caused by 1 earlier exceptions
    Cause number 1:
        Command 'yum install -y rsync' returned 255.
        stderr (7 lines)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        Warning: Permanently added 'rdma-cki-04.rdma.lab.eng.rdu2.redhat.com' (ED25519) to the list of known hosts.
        ssh_askpass: exec(/usr/libexec/openssh/gnome-ssh-askpass): No such file or directory
        Permission denied, please try again.
        ssh_askpass: exec(/usr/libexec/openssh/gnome-ssh-askpass): No such file or directory
        Permission denied, please try again.
        ssh_askpass: exec(/usr/libexec/openssh/gnome-ssh-askpass): No such file or directory
        root@rdma-cki-04.rdma.lab.eng.rdu2.redhat.com: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[idorax]

Hi @guoguojenna, would you please paste your fmf file? Then I will take a look to figure out why.

[guoguojenna]

Hi @idorax , thanks for the checking, here is my fmf file:

# $ cat a.fmf
summary: Demo
discover:
  - name: demo
    how: fmf 
    url: https://gitlab.com/ZhaojuanGuo/kernel-tests.git
    ref: 523-rdma
    test:
      - /rdma/opa-fm
provision:
    how: beaker
    image: RHEL-9.3.0-20230821.32
    hardware:
        hostname: "rdma-cki-04.rdma.lab.eng.rdu2.redhat.com"
    key: "/home/zguo/.ssh/id_rsa"
    user: root
    password: ##the beaker default password##
execute:
    how: tmt 

# $ tmt run plans --name /rdma/plans/example-provision-beaker -d -vvv

[idorax]

@guoguojenna, to run your fmf file, we have to configure both beaker client and kerberos on the SUT running tmt. Will talk to you via Slack about how to configure beaker client.

[guoguojenna]

Is there any way to avoid the password input for testing with beaker on tmt

[guoguojenna]

Update, I copied my ssh public key here, and this issue is gone. Thanks Huanian for checking this!

https://beaker.engineering.redhat.com/prefs/#ssh-public-keys

[idorax]

Update, I copied my ssh public key here, and this issue is gone.

@guoguojenna, can we close this issue?

[guoguojenna]

Closing.