ppc6446 commented 3 years ago

Additional Information

salt-ssh appears to be successful from the WebUI for Tumbleweed clients, but throws an error for Fedora 33 and CentOS Stream

Version of Uyuni Server and Proxy (if used)

Information for package Uyuni-Server-release:
---------------------------------------------
Repository     : uyuni-server-stable
Name           : Uyuni-Server-release
Version        : 2021.02-116.3.uyuni
Arch           : x86_64
Vendor         : obs://build.opensuse.org/systemsmanagement:Uyuni
Support Level  : Level 3
Installed Size : 1.4 KiB
Installed      : Yes
Status         : up-to-date
Source package : Uyuni-Server-release-2021.02-116.3.uyuni.src
Summary        : Uyuni Server
Description    :
    Uyuni lets you efficiently manage physical, virtual,
    and cloud-based Linux systems. It provides automated and cost-effective
    configuration and software management, asset management, and system
    provisioning.

Details about issue

Registering Tumbleweed clients in the WebUI with the Manage systems completely using SSH box check results in apparent success, but the client never shows up in the WebUI systems list or salt "*" test.ping

The WebUI reports this at addition time -

Successfully bootstrapped host! Your system should appear in systems shortly.

Registering Fedora 33 clients in the WebUI throws an error message -

Traceback (most recent call last):
File "/var/tmp/.root_bc7b7c_salt/salt-call", line 27, in <module>
salt_call()
File "/var/tmp/.root_bc7b7c_salt/pyall/salt/scripts.py", line 445, in salt_call
client.run()
File "/var/tmp/.root_bc7b7c_salt/pyall/salt/cli/call.py", line 47, in run
caller = salt.cli.caller.Caller.factory(self.config)
File "/var/tmp/.root_bc7b7c_salt/pyall/salt/cli/caller.py", line 63, in factory
return ZeroMQCaller(opts, **kwargs)
File "/var/tmp/.root_bc7b7c_salt/pyall/salt/cli/caller.py", line 312, in __init__
super(ZeroMQCaller, self).__init__(opts)
File "/var/tmp/.root_bc7b7c_salt/pyall/salt/cli/caller.py", line 87, in __init__
self.minion = salt.minion.SMinion(opts)
File "/var/tmp/.root_bc7b7c_salt/pyall/salt/minion.py", line 843, in __init__
opts['grains'] = salt.loader.grains(opts)
File "/var/tmp/.root_bc7b7c_salt/pyall/salt/loader.py", line 819, in grains
ret = funcs[key]()
File "/var/tmp/.root_bc7b7c_salt/pyall/salt/grains/core.py", line 2304, in ip_fqdn
ret['ipv6'] = salt.utils.network.ip_addrs6(include_loopback=True)
File "/var/tmp/.root_bc7b7c_salt/pyall/salt/utils/network.py", line 1263, in ip_addrs6
return _ip_addrs(interface, include_loopback, interface_data, 'inet6')
File "/var/tmp/.root_bc7b7c_salt/pyall/salt/utils/network.py", line 1244, in _ip_addrs
ret.add(addr)
File "/usr/lib64/python3.9/ipaddress.py", line 1920, in __hash__
return hash((self._ip, self._scope_id))
AttributeError: _scope_id

Registering CentOS Stream clients also throws an error message:

pkg_|-remove_traditional_stack_all_|-remove_traditional_stack_all_|-removed(retcode=2): State 'pkg.removed' was not found in SLS 'ssh_bootstrap'
Reason: 'pkg' __virtual__ returned False
pkg_|-remove_traditional_stack_|-remove_traditional_stack_|-removed(retcode=2): State 'pkg.removed' was not found in SLS 'ssh_bootstrap'
Reason: 'pkg' __virtual__ returned False

ppc6446 commented 3 years ago

Update - after allowing Pubkey in sshd_config on the Tumbleweed client, it has started showing up in the Uyuni WebUI.

It is still not showing up in salt "*" test.ping however (or any of salt-run manage.alived, salt-run manage.not_alived, salt-run manage.not_joined or salt-run manage.not_allowed) However, scheduling a reboot using the Uyuni console succeeded.

This message in /var/log/rhn/rhn_taskomatic_daemon.log may be related -

2021-04-20 12:55:20,536 [DefaultQuartzScheduler_Worker-17] WARN com.suse.manager.utils.SaltUtils - No product match found for: openSUSE 20210418 0 x86_64 2021-04-20 12:56:00,065 [DefaultQuartzScheduler_Worker-2] INFO com.redhat.rhn.taskomatic.task.ErrataCacheTask - In the queue: 1

Shirocco88 commented 3 years ago

@ppc6446 could you please run salt command with debug option

salt minion -l debug test.ping

and poste output.

paususe commented 3 years ago

@ppc6446 Regarding Tumbleweed: did your Tumbleweed have salt-minion installed already? what were the installation settings? what repositories did you mirror? I'd like to try to reproduce it?

@ppc6446 Regarding Fedora and CentOS Tream: where does your salt-minion come from? Fedora/CentOS Stream or Uyuni SaltStack's salt-minion is part of the Fedora and generally newer than Uyuni's. Problem with that is our salt-minion package includes 100+ patches, so some features may not work with SaltStack's (although I would be surprised if registration fails due to that, that's very basic functionality).

ppc6446 commented 3 years ago

salt minion -l debug test.ping returns the following output:

[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/py26-compat-salt.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/py26-compat-salt.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/susemanager.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/susemanager.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/susemanager_engine.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/susemanager_engine.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/virtualization-host-formula.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/virtualization-host-formula.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: snip
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/py26-compat-salt.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/py26-compat-salt.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/susemanager.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/susemanager.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/susemanager_engine.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/susemanager_engine.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/virtualization-host-formula.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/virtualization-host-formula.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: snip
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'snail_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
No minions matched the target. No command was sent, no jid was assigned.
[DEBUG   ] LazyLoaded nested.output
ERROR: No return received
[DEBUG   ] Closing IPCMessageSubscriber instance

ppc6446 commented 3 years ago

@paususe -

No salt is installed. The goal for it to be ssh only since I don't think there are client tools for openSUSE Tumbleweeed -

# zypper lr Repository priorities are without effect. All enabled repositories share the same priority.

# | Alias | Name | Enabled | GPG Check | Refresh --+----------------------------------+----------------------------+---------+-----------+-------- 1 | download.opensuse.org-non-oss | Main Repository (NON-OSS) | No | ---- | ---- 2 | download.opensuse.org-oss | Main Repository (OSS) | No | ---- | ---- 3 | download.opensuse.org-tumbleweed | Main Update Repository | No | ---- | ---- 4 | openSUSE-20201227-0 | openSUSE-20201227-0 | No | ---- | ---- 5 | repo-debug | openSUSE-Tumbleweed-Debug | No | ---- | ---- 6 | repo-source | openSUSE-Tumbleweed-Source | No | ---- | ---- # rpm -qa | grep salt #

Shirocco88 commented 3 years ago

[DEBUG ] Trying to connect to: tcp://127.0.0.1:4506

I couldn't see a connection to port tcp/22 (ssh); are you shure that "salt-ssh" is using "tcp://127.0.0.1:4506" for outgoing connections ?

Is the roaster file correct configured on the salt-master ?

ppc6446 commented 3 years ago

I have been able to find roster files containing that minion's hostname in /var/cache/salt -

# find . -name '.minions*' -exec grep snip {} \; Binary file ./master/jobs/58/cb0b55cb956877366778bebd85cd2f50f4143e7c900a71049cac14c5d0ca8b/.minions.p matches Binary file ./master/jobs/dc/c383acd4da4292d59eaf62d9dac4283d52f98a912c08d2f1e1a662752ea684/.minions.p matches

Also, the CentOS Stream, Tumbleweed and a random Focal Fossa system have a directory tree under /var/cache/salt/master/salt-ssh

/var/cache/salt/master/salt-ssh # ls -l total 0 drwx------ 3 salt salt 19 Jan 6 17:18 dog drwx------ 3 salt salt 19 Apr 20 11:19 puppy drwx------ 3 salt salt 19 Apr 20 11:28 snip

Shirocco88 commented 3 years ago

for salt-ssh you need roster file

cat /etc/salt/roster
# Sample salt-ssh config file
#web1:
#  host: 192.168.42.1 # The IP addr or DNS hostname
#  user: fred         # Remote executions will be executed as user fred
#  passwd: foobarbaz  # The password to use for login, if omitted, keys are used
#  sudo: True         # Whether to sudo to root, not enabled by default
#web2:
#  host: 192.168.42.2
#

ppc6446 commented 3 years ago

/etc/salt # cat /etc/salt/roster # Sample salt-ssh config file #web1: # host: 192.168.42.1 # The IP addr or DNS hostname # user: fred # Remote executions will be executed as user fred # passwd: foobarbaz # The password to use for login, if omitted, keys are used # sudo: True # Whether to sudo to root, not enabled by default #web2: # host: 192.168.42.2 snip: host: snip user: root /etc/salt # salt "snip" test.ping No minions matched the target. No command was sent, no jid was assigned. ERROR: No return received /etc/salt # salt "snip" cmd.run "uptime" No minions matched the target. No command was sent, no jid was assigned. ERROR: No return received

The -l debug command output is similar as well: [DEBUG ] Trying to connect to: tcp://127.0.0.1:4506 [DEBUG ] Closing AsyncZeroMQReqChannel instance No minions matched the target. No command was sent, no jid was assigned. [DEBUG ] LazyLoaded nested.output ERROR: No return received [DEBUG ] Closing IPCMessageSubscriber instance

Shirocco88 commented 3 years ago

I've checked the salt documentation: https://docs.saltproject.io/en/getstarted/ssh/connect.html

you need to use "salt-ssh" instead of "salt":

salt-ssh [target] [command] [arguments]

Please test

salt-ssh "snip" test.ping

and poste results.

ppc6446 commented 3 years ago

First, I tried salt-ssh, which was not installed, so I installed the salt-ssh package (the Uyuni packages should include salt-ssh as a requires dependency).

Once installing the packages and trying salt-ssh snip test.ping again, the salt-ssh command was hanging indefinitely. I tried making a connection with the ssh command with the ssh key it was using from the shell. This error message resulted -

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0755 for '/etc/salt/pki/master/ssh' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "/etc/salt/pki/master/ssh": bad permissions

To work around this error with the permissions -

snail:~ # chmod 700 /etc/salt/pki/master/ssh

Next, a salt key import was requested and it asked for the password to be entered -

snail:~ # salt-ssh  snip  test.ping
Permission denied for host snip, do you want to deploy the salt-ssh key? (password required):
[Y/n]

snail:~ # salt-ssh  snip  test.ping
Permission denied for host snip, do you want to deploy the salt-ssh key? (password required):
[Y/n]
Password for root@snip:
snip:
    Permission denied, no authentication information

A subsequent try succeeded, however:

snail:~ # salt-ssh snip test.ping Permission denied for host snip, do you want to deploy the salt-ssh key? (password required): [Y/n] y Password for root@snip: snip: True snail:~ # salt-ssh snip test.ping snip: True

uyuni-project / uyuni

Manage system completely via SSH registration not working for Fedora 33, Tumbleweed or CentOS Stream clients #3584

Additional Information

Version of Uyuni Server and Proxy (if used)

Details about issue