aaannz
commented
2 years ago Formulas and saltboot images will not work without this. Is this a new installation of migration from previous Uyuni?
Closed phibid closed 2 years ago
aaannz
commented
2 years ago Formulas and saltboot images will not work without this. Is this a new installation of migration from previous Uyuni?
phibid
commented
2 years ago @aaannz Thanks. I don't use either one, so that explains why I did not noticed anything.
This is an upgrade from 2022.02 to 2022.06. I have checked the logs, those stack traces begin to appear after the upgrade to 2022.06.
aaannz
commented
2 years ago Can you check your /etc/salt/master.d/susemanager_db.conf if your access credentials to the database are correctly set?
phibid
commented
2 years ago The content of that file:
# cat /etc/salt/master.d/susemanager_db.conf
postgres:
db: uyuni
host: localhost
pass: xxx
port: 5432
user: adminNo references to the spacewalk psql user. I have checked the admin account and all good:
postgres@uyuni:~> psql -U admin -d uyuni
Password for user admin:
psql (14.3)
Type "help" for help.
uyuni=# This seems fine. Can you check ownership and permissions of the /etc/salt/master.d/susemanager_db.conf file
phibid
commented
2 years ago Here it is:
uyuni:~ # ls -al /etc/salt/master.d/susemanager_db.conf
-rw-r----- 1 salt salt 84 Jul 25 08:10 /etc/salt/master.d/susemanager_db.conf
ftigges-comdev
commented
2 years ago Hi,
I have the same error message after upgrading.
When I try to bootstrap a new system the gui gives the error:
"An error has occurred during salt execution: unable to parse json."
--------------------------------------------------------------------
I am able to connect to postgres with
--------------------------------------------------------------------
psql -h localhost -p 5432 -U uyuni --password $password_from_susemanager_db.conf
Thanks and best regards
In
--------------------------------------------------------------------
/var/log/salt/api the following errors are written:
--------------------------------------------------------------------
2022-09-14 16:48:36,327 [salt.pillar :1169][ERROR ][22498] Exception caught loading ext_pillar 'suma_minion':
File "/usr/lib/python3.6/site-packages/salt/pillar/__init__.py", line 1158, in ext_pillar
ext = self._external_pillar_data(pillar, val, key)
File "/usr/lib/python3.6/site-packages/salt/pillar/__init__.py", line 1098, in _external_pillar_data
ext = self.ext_pillars[key](self.minion_id, pillar, val)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1203, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
return callable(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1218, in _run_as
return _func_or_method(*args, **kwargs)
File "/usr/share/susemanager/modules/pillar/suma_minion.py", line 122, in ext_pillar
with _get_cursor() as cursor:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/usr/share/susemanager/modules/pillar/suma_minion.py", line 99, in _get_cursor
port=options['port'])
File "/usr/lib64/python3.6/site-packages/psycopg2/__init__.py", line 127, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
2022-09-14 16:48:36,331 [salt.pillar :1224][CRITICAL][22498] Pillar render error: Failed to load ext_pillar suma_minion: connection to server at "localhost" (::1), port 5432 failed: FATAL: Ident authentication failed for user "spacewalk"
connection to server at "localhost" (::1), port 5432 failed: FATAL: Ident authentication failed for user "spacewalk"
--------------------------------------------------------------------
Ownership and rights susemanager_db.conf
--------------------------------------------------------------------
ls -al /etc/salt/master.d/susemanager_db.conf
-rw-r----- 1 salt salt 92 Sep 7 10:45 /etc/salt/master.d/susemanager_db.conf
--------------------------------------------------------------------
spacewalk seems to have no problem with connection
--------------------------------------------------------------------
echo "show server_version;" | spacewalk-sql --select-mode -
Prototype mismatch: sub Spacewalk::Setup::idn_to_ascii: none vs ($;$$) at /usr/lib/perl5/5.26.1/Exporter.pm line 66.
at /usr/lib/perl5/vendor_perl/5.26.1/Spacewalk/Setup.pm line 29.
server_version
14.5
(1 row)
Hmm... seems to me that for some reason suma_minion pillar module is not using configured user and instead using default ( suma_minion.py#L84 ). Is there no error reported on salt master start or before this pillar error?
ftigges-comdev
commented
2 years ago Sadly this is not the case.. When I change the defaults to the actual user I get:
2022-09-15 08:50:37,160 [salt.utils.templates:274 ][ERROR ][10343] Rendering exception occurred
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
output = template.render(**decoded_context)
File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
return original_render(self, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "<template>", line 21, in top-level template code
File "/var/tmp/.root_d6d5aa_salt/running_data/var/cache/venv-salt-minion/minion/files/base/channels/disablelocalrepos.sls", line 24, in top-level template code
{% if (repos_disabled.match_str in alias)|string == repos_disabled.matching|string and data.get('enabled', True) in [True, '1'] %}
File "/usr/lib/python3.6/site-packages/jinja2/sandbox.py", line 436, in call
if not __self.is_safe_callable(__obj):
File "/usr/lib/python3.6/site-packages/jinja2/sandbox.py", line 338, in is_safe_callable
return not (getattr(obj, 'unsafe_callable', False) or
jinja2.exceptions.UndefinedError: 'str object' has no attribute 'get'This error is however something different. For some reason pkg.list_repos is returning unexpected data.
To check if pillars work, try salt $minion pillar.items.
And for the problem in last comment, see what salt $minion pkg.list_repos is returning, but please open a new issue for it.
ftigges-comdev
commented
2 years ago addon_group_types:
- monitoring_entitled
- salt_entitled
beacons:
----------
pkgset:
----------
interval:
5
channels:
----------
opensuse_leap15_4-x86_64:
----------
alias:
susemanager:opensuse_leap15_4-x86_64
autorefresh:
1
cloned_nonmodular:
False
enabled:
1
gpgcheck:
0
host:`
Does that mean, pillar is working? I can ignore the error msg for now?
Thanks
Florian
aaannz
commented
2 years ago Seems like it's working for you. Assuming you did not paste complete pillar.items output. That would confirm the suspicion about using default values instead of configured one. Note that update of uyuni will overwrite your change in the salt-minion.py file.
ftigges-comdev
commented
2 years ago Will do.
I will open a new Ticket for the repo problem.
Thanks for your time.
vzhestkov
commented
2 years ago For some reason I can't reproduce the issue on my test system.
@phibid @ftigges-comdev could you please try to apply this patch to suma_minion pillar module:
--- a/suma_minion.py
+++ b/suma_minion.py
@@ -87,6 +87,7 @@
'port': 5432,
}
opts = __opts__.get("postgres", {})
+ log.critical("OPTS DEBUG: %s", opts)
options = {}
for attr, default in defaults.items():
options[attr] = opts.get(attr, default)Save it to the patch file and call: cd /usr/share/susemanager/modules/pillar ; patch -p1 < /PATH/TO/this.patch
Restart the salt-api service and check if the logging messages in the /var/log/salt/api will contain the proper username and password from the config file.
@ftigges-comdev could you please set the trace logging level to the salt-master and restart salt-api service and try to reproduce the issue?
Additional note: no need to restart salt-master as salt ssh operations are performed with salt-api.
echo "log_level_logfile: trace" > /etc/salt/master.d/logging.confDon't forget to remove this config after the tests, otherwise salt-api will state too noisy.
ftigges-comdev
commented
2 years ago The debug msg shows the correct values (the ones from the config file).
My problem with the bootstrapping is solved.. (The message on the gui doesn't reflect that..)
Warning: the ECDSA host key for xxx.yyyy.zzz differs from the key for the IP address...
Do you still want the extended Trace?
vzhestkov
commented
2 years ago If there is no more issues with the opts from the config and render issue, I don't need the traces. If there is a case when the issue still exists, I can check the traces, but note it could contain some sensitive data.
ftigges-comdev
commented
2 years ago Part of the Trace
2022-09-15 13:22:42,518 [salt.client.ssh :37 ][TRACE ][14781] STDOUT HOST.DOMAIN.TLD
Password:
DELETED
{
"local": {
"jid": "20220915112241939451",
"return": {
"interface": "0.0.0.0",
"master": "salt",
"master_type": "str",
"master_uri_format": "default",
"source_interface_name": "",
"source_address": "",
"source_ret_port": 0,
"source_publish_port": 0,
"master_port": 4506,
"master_finger": "",
"master_shuffle": false,
"master_alive_interval": 0,
"master_failback": false,
"master_failback_interval": 0,
"verify_master_pubkey_sign": false,
"sign_pub_messages": false,
"always_verify_signature": false,
"master_sign_key_name": "master_sign",
"syndic_finger": "",
"user": "root",
"root_dir": "/var/tmp/.root_d6d5aa_salt/running_data",
"pki_dir": "/var/tmp/.root_d6d5aa_salt/running_data/etc/venv-salt-minion/pki/minion",
"id": "HOST.DOMAIN.TLD",
"id_function": {},
"cachedir": "/var/tmp/.root_d6d5aa_salt/running_data/var/cache/venv-salt-minion/minion",
"append_minionid_config_dirs": [],
"cache_jobs": false,
"grains_blacklist": [],
"grains_cache": true,
"grains_cache_expiration": 300,
"grains_deep_merge": false,
"conf_file": "/var/tmp/.root_d6d5aa_salt/minion",
"sock_dir": "/var/tmp/.root_d6d5aa_salt/running_data",
"sock_pool_size": 1,
"backup_mode": "",
"renderer": "jinja|yaml",
"renderer_whitelist": [],
"renderer_blacklist": [],
"random_startup_delay": 0,
"failhard": false,
"autoload_dynamic_modules": true,
"saltenv": null,
"lock_saltenv": false,
"pillarenv": null,
"pillarenv_from_saltenv": false,
"pillar_opts": false,
"pillar_source_merging_strategy": "smart",
"pillar_merge_lists": false,
"pillar_includes_override_sls": false,
"pillar_cache": false,
"pillar_cache_ttl": 3600,
"pillar_cache_backend": "disk",
"gpg_cache": false,
"gpg_cache_ttl": 86400,
"gpg_cache_backend": "disk",
"extension_modules": "/var/tmp/.root_d6d5aa_salt/running_data/var/cache/venv-salt-minion/minion/extmods",
"state_top": "top.sls",
"state_top_saltenv": null,
"startup_states": "",
"sls_list": [],
"start_event_grains": [],
"top_file": "",
"thoriumenv": null,
"thorium_top": "top.sls",
"thorium_interval": 0.5,
"thorium_roots": {
"base": [
"/srv/thorium"
]
},
"file_client": "local",
"local": true,
"use_master_when_local": false,
"file_roots": {
"base": [
"/srv/salt",
"/srv/spm/salt"
]
},
"top_file_merging_strategy": "merge",
"env_order": [],
"default_top": "base",
"fileserver_limit_traversal": false,
"file_recv": false,
"file_recv_max_size": 100,
"file_ignore_regex": [],
"file_ignore_glob": [],
"fileserver_backend": [
"roots"
],
"fileserver_followsymlinks": true,
"fileserver_ignoresymlinks": false,
"pillar_roots": {
"base": [
"/srv/pillar",
"/srv/spm/pillar"
]
},
"on_demand_ext_pillar": [
"libvirt",
"virtkey"
],
"decrypt_pillar": [],
"decrypt_pillar_delimiter": ":",
"decrypt_pillar_default": "gpg",
"decrypt_pillar_renderers": [
"gpg"
],
"roots_update_interval": 60,
"azurefs_update_interval": 60,
"gitfs_update_interval": 60,
"git_pillar_update_interval": 60,
"hgfs_update_interval": 60,
"minionfs_update_interval": 60,
"s3fs_update_interval": 60,
"svnfs_update_interval": 60,
"git_pillar_base": "master",
"git_pillar_branch": "master",
"git_pillar_env": "",
"git_pillar_fallback": "",
"git_pillar_root": "",
"git_pillar_ssl_verify": true,
"git_pillar_global_lock": true,
"git_pillar_user": "",
"git_pillar_password": "",
"git_pillar_insecure_auth": false,
"git_pillar_privkey": "",
"git_pillar_pubkey": "",
"git_pillar_passphrase": "",
"git_pillar_refspecs": [
"+refs/heads/*:refs/remotes/origin/*",
"+refs/tags/*:refs/tags/*"
],
"git_pillar_includes": true,
"gitfs_remotes": [],
"gitfs_mountpoint": "",
"gitfs_root": "",
"gitfs_base": "master",
"gitfs_fallback": "",
"gitfs_user": "",
"gitfs_password": "",
"gitfs_insecure_auth": false,
"gitfs_privkey": "",
"gitfs_pubkey": "",
"gitfs_passphrase": "",
"gitfs_saltenv_whitelist": [],
"gitfs_saltenv_blacklist": [],
"gitfs_global_lock": true,
"gitfs_ssl_verify": true,
"gitfs_saltenv": [],
"gitfs_ref_types": [
"branch",
"tag",
"sha"
],
"gitfs_refspecs": [
"+refs/heads/*:refs/remotes/origin/*",
"+refs/tags/*:refs/tags/*"
],
"gitfs_disable_saltenv_mapping": false,
"unique_jid": false,
"hash_type": "sha256",
"optimization_order": [
0,
1,
2
],
"disable_modules": [],
"disable_returners": [],
"whitelist_modules": [],
"module_dirs": [],
"returner_dirs": [],
"grains_dirs": [],
"states_dirs": [],
"render_dirs": [],
"outputter_dirs": [],
"utils_dirs": [
"/var/cache/venv-salt-minion/minion/extmods/utils"
],
"publisher_acl": {},
"publisher_acl_blacklist": {},
"providers": {},
"clean_dynamic_modules": true,
"open_mode": false,
"auto_accept": true,
"autosign_timeout": 120,
"multiprocessing": true,
"process_count_max": -1,
"mine_enabled": true,
"mine_return_job": false,
"mine_interval": 60,
"ipc_mode": "ipc",
"ipc_write_buffer": 0,
"ipv6": null,
"file_buffer_size": 262144,
"tcp_pub_port": 4510,
"tcp_pull_port": 4511,
"tcp_authentication_retries": 5,
"tcp_reconnect_backoff": 1,
"log_file": "/var/log/salt-ssh.log",
"log_level": "quiet",
"log_datefmt": "%H:%M:%S",
"log_datefmt_logfile": "%Y-%m-%d %H:%M:%S",
"log_fmt_console": "[%(levelname)-8s] %(message)s",
"log_fmt_logfile": "%(asctime)s,%(msecs)03d [%(name)-17s:%(lineno)-4d][%(levelname)-8s][%(process)d] %(message)s",
"log_fmt_jid": "[JID: %(jid)s]",
"log_granular_levels": {},
"log_rotate_max_bytes": 0,
"log_rotate_backup_count": 0,
"max_event_size": 1048576,
"enable_legacy_startup_events": true,
"test": false,
"ext_job_cache": "",
"cython_enable": false,
"enable_fqdns_grains": true,
"enable_gpu_grains": true,
"enable_zip_modules": false,
"state_verbose": true,
"state_output": "full",
"state_output_diff": false,
"state_output_profile": true,
"state_auto_order": true,
"state_events": false,
"state_aggregate": false,
"snapper_states": false,
"snapper_states_config": "root",
"acceptance_wait_time": 10,
"acceptance_wait_time_max": 0,
"rejected_retry": false,
"loop_interval": 1,
"verify_env": true,
"grains": {
"cwd": "/root",
"ip_gw": false,
"ip4_gw": false,
"ip6_gw": false,
"dns": {
"nameservers": [
"IPADDRESS
"IPADDRESS"
],
"ip4_nameservers": [
"IPADDRESS
"IPADDRESS"
],
"ip6_nameservers": [],
"sortlist": [],
"domain": "",
"search": [
"DOMAIN.TLD"
],
"options": []
},
"fqdns": [
"HOSTNAME"
],
"machine_id": "aaf3a9bd244f43adbaf2d821ef786128",
"master": "salt",
"server_id": 836354377,
"localhost": "HOSTNAME",
"fqdn": "HOSTNAME",
"host": "HOSTNAME",
"domain": "",
"hwaddr_interfaces": {
"lo": "00:00:00:00:00:00",
"eth0": "00:16:3e:4e:0a:a4"
},
"id": "HOST.DOMAIN.TLD",
"ip4_interfaces": {
"lo": [
"127.0.0.1"
],
"eth0": [
"IPVADDRESS"
]
},
"ip6_interfaces": {
"lo": [
"::1"
],
"eth0": [
"IPV6ADDRESS"
]
},
"ipv4": [
"127.0.0.1",
"IPVADDRESS"
],
"ipv6": [
"::1",
"IPV6ADDRESS"
],
"fqdn_ip4": [
"IPVADDRESS"
],
"fqdn_ip6": [],
"ip_interfaces": {
"lo": [
"127.0.0.1",
"::1"
],
"eth0": [
"IPADDRESS
"IPV6ADDRESS"
]
},
"kernelparams": [
[
"BOOT_IMAGE",
"/boot/vmlinuz-5.3.18-150300.59.93-default"
],
[
"root",
"/dev/mapper/system-root"
],
[
"resume",
"/dev/system/swap"
],
[
"splash",
"silent"
],
[
"quiet",
null
],
[
"mitigations",
"auto"
]
],
"locale_info": {
"defaultlanguage": "de_DE",
"defaultencoding": "UTF-8",
"detectedencoding": "utf-8",
"timezone": "unknown"
},
"num_gpus": 1,
"gpus": [
{
"vendor": "unknown",
"model": "Device 1111"
}
],
"kernel": "Linux",
"nodename": "HOSTNAME",
"kernelrelease": "5.3.18-150300.59.93-default",
"kernelversion": "#1 SMP Tue Sep 6 05:05:37 UTC 2022 (7acce37)",
"cpuarch": "x86_64",
"systemd": {
"version": "246",
"features": "+PAM +AUDIT +SELINUX -IMA +APPARMOR -SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid"
},
"init": "systemd",
"lsb_distrib_id": "openSUSE Leap",
"lsb_distrib_release": "15.3",
"lsb_distrib_codename": "openSUSE Leap 15.3",
"os": "SUSE",
"osfullname": "Leap",
"osrelease": "15.3",
"oscodename": "openSUSE Leap 15.3",
"num_cpus": 2,
"cpu_model": "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz",
"cpu_flags": [
"fpu",
"vme",
"de",
"pse",
"tsc",
"msr",
"pae",
"mce",
"cx8",
"apic",
"sep",
"mtrr",
"pge",
"mca",
"cmov",
"pat",
"pse36",
"clflush",
"acpi",
"mmx",
"fxsr",
"sse",
"sse2",
"ss",
"ht",
"syscall",
"nx",
"pdpe1gb",
"rdtscp",
"lm",
"constant_tsc",
"rep_good",
"nopl",
"cpuid",
"pni",
"pclmulqdq",
"ssse3",
"cx16",
"pcid",
"sse4_1",
"sse4_2",
"x2apic",
"popcnt",
"tsc_deadline_timer",
"aes",
"xsave",
"avx",
"hypervisor",
"lahf_lm",
"cpuid_fault",
"pti",
"xsaveopt"
],
"os_family": "Suse",
"osarch": "x86_64",
"mem_total": 1941,
"swap_total": 2007,
"biosversion": "4.14.4_04-150300.3.",
"productname": "HVM domU",
"manufacturer": "Xen",
"biosreleasedate": "07/15/2022",
"uuid": "b78491d0-2ac2-4ba0-a55e-a1a132794879",
"serialnumber": "b78491d0-2ac2-4ba0-a55e-a1a132794879",
"virtual": "xen",
"virtual_subtype": "Xen HVM DomU",
"virtual_hv_version": "4.14.4_04-150300.3.",
"virtual_hv_version_info": [
"4",
"14",
".4_04-150300.3."
],
"virtual_hv_features": "00002705",
"virtual_hv_features_list": [
"writable_page_tables",
"auto_translated_physmap",
"hvm_callback_vector",
"hvm_safe_pvclock",
"hvm_pirqs",
"memory_op_vnode_supported"
],
"ps": "ps -efHww",
"osrelease_info": [
15,
3
],
"osmajorrelease": 15,
"osfinger": "Leap-15",
"path": "/var/tmp/venv-salt-minion/bin:/var/tmp/venv-salt-minion/bin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin",
"systempath": [
"/var/tmp/venv-salt-minion/bin",
"/var/tmp/venv-salt-minion/bin",
"/sbin",
"/usr/sbin",
"/usr/local/sbin",
"/root/bin",
"/usr/local/bin",
"/usr/bin",
"/bin"
],
"pythonexecutable": "/var/tmp/venv-salt-minion/bin/python",
"pythonpath": [
"/var/tmp/venv-salt-minion/bin",
"/var/lib/venv-salt-minion/local",
"/var/tmp/venv-salt-minion/lib64/python310.zip",
"/var/tmp/venv-salt-minion/lib64/python3.10",
"/var/tmp/venv-salt-minion/lib64/python3.10/lib-dynload",
"/var/tmp/venv-salt-minion/lib64/python3.10/site-packages",
"/var/tmp/venv-salt-minion/lib/python3.10/site-packages"
],
"pythonversion": [
3,
10,
5,
"final",
0
],
"saltpath": "/var/tmp/venv-salt-minion/lib/python3.10/site-packages/salt",
"saltversion": "3004",
"saltversioninfo": [
3004
],
"zmqversion": "4.2.3",
"cpu_cores": "2",
"cpu_numanodes": "1",
"cpu_vendor": "GenuineIntel",
"cpu_stepping": "7",
"cpusockets": 1,
"total_num_cpus": 2,
"is_mgr_server": false,
"has_report_db": false,
"is_uyuni": true,
"disks": [],
"ssds": [
"xvda"
],
"shell": "/bin/bash",
"__suse_reserved_pkg_all_versions_support": true,
"__suse_reserved_pkg_patches_support": true,
"__suse_reserved_saltutil_states_support": true,
"transactional": false,
"efi": false,
"efi-secure-boot": false,
"lvm": {
"system": [
"root",
"swap"
]
},
"username": "root",
"groupname": "root",
"pid": 7182,
"gid": 0,
"uid": 0,
"zfs_support": false,
"zfs_feature_flags": false
},
"permissive_pki_access": false,
"default_include": "minion.d/*.conf",
"update_url": false,
"update_restart_services": [],
"retry_dns": 30,
"retry_dns_count": null,
"resolve_dns_fallback": true,
"recon_max": 10000,
"recon_default": 1000,
"recon_randomize": true,
"return_retry_timer": 5,
"return_retry_timer_max": 10,
"return_retry_tries": 3,
"random_reauth_delay": 10,
"winrepo_source_dir": "salt://win/repo-ng/",
"winrepo_dir": "/srv/salt/win/repo",
"winrepo_dir_ng": "/srv/salt/win/repo-ng",
"winrepo_cachefile": "winrepo.p",
"winrepo_cache_expire_max": 604800,
"winrepo_cache_expire_min": 1800,
"winrepo_remotes": [
"https://github.com/saltstack/salt-winrepo.git"
],
"winrepo_remotes_ng": [
"https://github.com/saltstack/salt-winrepo-ng.git"
],
"winrepo_branch": "master",
"winrepo_fallback": "",
"winrepo_ssl_verify": true,
"winrepo_user": "",
"winrepo_password": "",
"winrepo_insecure_auth": false,
"winrepo_privkey": "",
"winrepo_pubkey": "",
"winrepo_passphrase": "",
"winrepo_refspecs": [
"+refs/heads/*:refs/remotes/origin/*",
"+refs/tags/*:refs/tags/*"
],
"pidfile": "/var/tmp/.root_d6d5aa_salt/running_data/var/run/salt-minion.pid",
"range_server": "range:80",
"reactor_refresh_interval": 60,
"reactor_worker_threads": 10,
"reactor_worker_hwm": 10000,
"engines": [],
"tcp_keepalive": true,
"tcp_keepalive_idle": 300,
"tcp_keepalive_cnt": -1,
"tcp_keepalive_intvl": -1,
"modules_max_memory": -1,
"grains_refresh_every": 0,
"minion_id_caching": true,
"minion_id_lowercase": false,
"minion_id_remove_domain": false,
"keysize": 2048,
"transport": "zeromq",
"auth_timeout": 5,
"auth_tries": 7,
"master_tries": 1,
"master_tops_first": false,
"auth_safemode": false,
"random_master": false,
"cluster_mode": false,
"restart_on_error": false,
"ping_interval": 0,
"username": null,
"password": null,
"zmq_filtering": false,
"zmq_monitor": false,
"cache_sreqs": true,
"cmd_safe": true,
"sudo_user": "",
"http_connect_timeout": 20.0,
"http_request_timeout": 3600.0,
"http_max_body": 107374182400,
"event_match_type": "startswith",
"minion_restart_command": [],
"pub_ret": true,
"proxy_host": "",
"proxy_username": "",
"proxy_password": "",
"proxy_port": 0,
"minion_jid_queue_hwm": 100,
"ssl": null,
"multifunc_ordered": false,
"beacons_before_connect": false,
"scheduler_before_connect": false,
"cache": "localfs",
"salt_cp_chunk_size": 65536,
"extmod_whitelist": {},
"extmod_blacklist": {},
"minion_sign_messages": false,
"discovery": false,
"schedule": {},
"ssh_merge_pillar": true,
"disabled_requisites": [],
"reactor_niceness": null,
"fips_mode": false,
"server_id_use_crc": false,
"__role": "minion",
"fileserver_list_cache_time": 3,
"__cli": "salt-call",
"beacons": {},
"selected_output_option": "state_output",
"fun": "test.opts_pkg",
"arg": [],
"config_dir": "/var/tmp/.root_d6d5aa_salt",
"args_stdin": false,
"hard_crash": false,
"no_parse": [],
"grains_run": false,
"doc": false,
"return": "",
"retcode_passthrough": true,
"print_metadata": true,
"skip_grains": false,
"refresh_grains_cache": false,
"no_return_event": false,
"log_level_logfile": "warning",
"output": "json",
"output_file_append": false,
"no_color": false,
"force_color": false,
"profiling_path": "/tmp/stats",
"profiling_enabled": false,
"log_datefmt_console": "%H:%M:%S",
"caller": true,
"pillar": {}
},
"retcode": 0,
"id": "HOST.DOMAIN.TLD",
"fun": "test.opts_pkg",
"fun_args": []
}
}
2022-09-15 13:22:42,520 [salt.client.ssh :37 ][TRACE ][14781] STDERR HOST.DOMAIN.TLD
SALT_ARGV: ['/var/tmp/venv-salt-minion/bin/python', '/var/tmp/venv-salt-minion/bin/salt-call', '--retcode-passthrough', '--local', '--metadata', '--out', 'json', '-l', 'quiet', '-c', '/var/tmp/.root_d6d5aa_salt', '--', 'test.opts_pkg']
_edbc7885e4f9aac9b83b35999b68d015148caf467b78fa39c05f669c0ff89878
2022-09-15 13:22:42,521 [salt.client.ssh :1554][DEBUG ][14781] RETCODE HOST.DOMAIN.TLD: 0
2022-09-15 13:22:42,586 [salt.utils.entrypoints:57 ][DEBUG ][14781] Using importlib_metadata to load entry points
2022-09-15 13:22:42,687 [salt.utils.lazy :99 ][DEBUG ][14781] LazyLoaded jinja.render
2022-09-15 13:22:42,688 [salt.utils.lazy :99 ][DEBUG ][14781] LazyLoaded yaml.render
2022-09-15 13:22:42,711 [salt.utils.lazy :99 ][DEBUG ][14781] LazyLoaded suma_minion.ext_pillar
2022-09-15 13:22:42,712 [salt.loaded.ext.pillar.suma_minion:115 ][DEBUG ][14781] Getting pillar data for the minion "HOST.DOMAIN.TLD"
2022-09-15 13:22:42,734 [salt.loaded.ext.pillar.suma_minion:102 ][DEBUG ][14781] Connected to DB
2022-09-15 13:22:42,735 [salt.loaded.ext.pillar.suma_minion:158 ][DEBUG ][14781] Loading global pillars from db
2022-09-15 13:22:42,755 [salt.utils.lazy :99 ][DEBUG ][14781] LazyLoaded state.apply
2022-09-15 13:22:42,760 [salt.state :750 ][DEBUG ][14781] Gathering pillar data for state run
2022-09-15 13:22:42,761 [salt.pillar :70 ][DEBUG ][14781] Determining pillar cache
2022-09-15 13:22:42,838 [salt.utils.lazy :99 ][DEBUG ][14781] LazyLoaded jinja.render
2022-09-15 13:22:42,839 [salt.utils.lazy :99 ][DEBUG ][14781] LazyLoaded yaml.render
2022-09-15 13:22:42,867 [salt.utils.lazy :99 ][DEBUG ][14781] LazyLoaded suma_minion.ext_pillar
2022-09-15 13:22:42,868 [salt.loaded.ext.pillar.suma_minion:115 ][DEBUG ][14781] Getting pillar data for the minion "HOST.DOMAIN.TLD"
2022-09-15 13:22:42,881 [salt.pillar :1169][ERROR ][14781] Exception caught loading ext_pillar 'suma_minion':
File "/usr/lib/python3.6/site-packages/salt/pillar/__init__.py", line 1158, in ext_pillar
ext = self._external_pillar_data(pillar, val, key)
File "/usr/lib/python3.6/site-packages/salt/pillar/__init__.py", line 1098, in _external_pillar_data
ext = self.ext_pillars[key](self.minion_id, pillar, val)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1203, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
return callable(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1218, in _run_as
return _func_or_method(*args, **kwargs)
File "/usr/share/susemanager/modules/pillar/suma_minion.py", line 122, in ext_pillar
with _get_cursor() as cursor:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/usr/share/susemanager/modules/pillar/suma_minion.py", line 99, in _get_cursor
port=options['port'])
File "/usr/lib64/python3.6/site-packages/psycopg2/__init__.py", line 127, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
2022-09-15 13:22:42,881 [salt.pillar :1224][CRITICAL][14781] Pillar render error: Failed to load ext_pillar suma_minion: connection to server at "localhost" (::1), port 5432 failed: FATAL: Ident authentication failed for user "spacewalk"
connection to server at "localhost" (::1), port 5432 failed: FATAL: Ident authentication failed for user "spacewalk"
2022-09-15 13:22:42,881 [salt.state :765 ][DEBUG ][14781] Finished gathering pillar data for state run
2022-09-15 13:22:42,967 [salt.utils.lazy :99 ][DEBUG ][14781] LazyLoaded jinja.render
2022-09-15 13:22:42,968 [salt.utils.lazy :99 ][DEBUG ][14781] LazyLoaded yaml.render
2022-09-15 13:22:42,972 [salt.fileserver :149 ][DEBUG ][14781] Returning file list from cache: age=5 cache_time=20 /var/cache/salt/master/file_lists/roots/base.p
2022-09-15 13:22:42,975 [salt.fileclient :1136][DEBUG ][14781] Could not find file 'salt://certs.sls' in saltenv 'base'
2022-09-15 13:22:42,976 [salt.fileclient :1159][DEBUG ][14781] In saltenv 'base', looking at rel_path 'certs/init.sls' to resolve 'salt://certs/init.sls'
2022-09-15 13:22:42,976 [salt.fileclient :1168][DEBUG ][14781] In saltenv 'base', ** considering ** path '/var/tmp/.root_d6d5aa_salt/running_data/var/cache/venv-salt-minion/minion/files/base/certs/init.sls' to resolve 'salt://certs/init.sls'
2022-09-15 13:22:42,976 [salt.template :53 ][DEBUG ][14781] compile template: /var/tmp/.root_d6d5aa_salt/running_data/var/cache/venv-salt-minion/minion/files/base/certs/init.sls
2022-09-15 13:22:42,977 [salt.utils.jinja :92 ][DEBUG ][14781] Jinja search path: ['/var/tmp/.root_d6d5aa_salt/running_data/var/cache/venv-salt-minion/minion/files/base']
2022-09-15 13:22:42,981 [salt.fileclient :1159][DEBUG ][14781] In saltenv 'base', looking at rel_path 'certs/suse.sls' to resolve 'salt://certs/suse.sls'
2022-09-15 13:22:42,981 [salt.fileclient :1168][DEBUG ][14781] In saltenv 'base', ** considering ** path '/var/tmp/.root_d6d5aa_salt/running_data/var/cache/venv-salt-minion/minion/files/base/certs/suse.sls' to resolve 'salt://certs/suse.sls'
2022-09-15 13:22:42,983 [salt.template :28 ][PROFILE ][14781] Time (in seconds) to render '/var/tmp/.root_d6d5aa_salt/running_data/var/cache/venv-salt-minion/minion/files/base/certs/init.sls' using 'jinja' renderer: 0.0063207149505615234
2022-09-15 13:22:42,983 [salt.template :120 ][DEBUG ][14781] Rendered data from file: /var/tmp/.root_d6d5aa_salt/running_data/var/cache/venv-salt-minion/minion/files/base/certs/init.sls:
mgr_absent_ca_package:
pkg.removed:
- name: rhn-org-trusted-ssl-cert
mgr_ca_cert:
file.managed:
- name: /etc/pki/trust/anchors/RHN-ORG-TRUSTED-SSL-CERT
- source: salt://certs/RHN-ORG-TRUSTED-SSL-CERT
update-ca-certificates:
cmd.run:
- name: /usr/sbin/update-ca-certificates
- runas: root
- onchanges:
- file: mgr_ca_cert
- retry:
attempts: 5
interval: 5
until: True
@ftigges-comdev could you please take one more shot with the patch (which I put before) applied? This part is bit confusing:
2022-09-15 13:22:42,712 [salt.loaded.ext.pillar.suma_minion:115 ][DEBUG ][14781] Getting pillar data for the minion "HOST.DOMAIN.TLD"
2022-09-15 13:22:42,734 [salt.loaded.ext.pillar.suma_minion:102 ][DEBUG ][14781] Connected to DB
2022-09-15 13:22:42,735 [salt.loaded.ext.pillar.suma_minion:158 ][DEBUG ][14781] Loading global pillars from db
2022-09-15 13:22:42,755 [salt.utils.lazy :99 ][DEBUG ][14781] LazyLoaded state.apply
2022-09-15 13:22:42,760 [salt.state :750 ][DEBUG ][14781] Gathering pillar data for state run
2022-09-15 13:22:42,761 [salt.pillar :70 ][DEBUG ][14781] Determining pillar cache
2022-09-15 13:22:42,838 [salt.utils.lazy :99 ][DEBUG ][14781] LazyLoaded jinja.render
2022-09-15 13:22:42,839 [salt.utils.lazy :99 ][DEBUG ][14781] LazyLoaded yaml.render
2022-09-15 13:22:42,867 [salt.utils.lazy :99 ][DEBUG ][14781] LazyLoaded suma_minion.ext_pillar
2022-09-15 13:22:42,868 [salt.loaded.ext.pillar.suma_minion:115 ][DEBUG ][14781] Getting pillar data for the minion "HOST.DOMAIN.TLD"
2022-09-15 13:22:42,881 [salt.pillar :1169][ERROR ][14781] Exception caught loading ext_pillar 'suma_minion':It calls the module twice for some reason, but on the first shot it doesn't fail, but fails on the second one. In my testing environment it is called twice also, but with no issue on the second call. Need to dig deeper, but it's better to understand how to reproduce it locally.
ftigges-comdev
commented
2 years ago Second time it's empty..
"fun": "test.opts_pkg",
"fun_args": []
}
}
2022-09-15 14:51:41,239 [salt.client.ssh :37 ][TRACE ][500] STDERR HOST.DOMAIN.TLD
SALT_ARGV: ['/var/tmp/venv-salt-minion/bin/python', '/var/tmp/venv-salt-minion/bin/salt-call', '--retcode-passthrough', '--local', '--metadata', '--out', 'json', '-l', 'quiet', '-c', '/var/tmp/.root_d6d5aa_salt', '--', 'test.opts_pkg']
_edbc7885e4f9aac9b83b35999b68d015148caf467b78fa39c05f669c0ff89878
2022-09-15 14:51:41,239 [salt.client.ssh :1554][DEBUG ][500] RETCODE HOST.DOMAIN.TLD: 0
2022-09-15 14:51:41,270 [salt.utils.entrypoints:57 ][DEBUG ][500] Using importlib_metadata to load entry points
2022-09-15 14:51:41,342 [salt.utils.lazy :99 ][DEBUG ][500] LazyLoaded jinja.render
2022-09-15 14:51:41,343 [salt.utils.lazy :99 ][DEBUG ][500] LazyLoaded yaml.render
2022-09-15 14:51:41,414 [salt.utils.lazy :99 ][DEBUG ][500] LazyLoaded suma_minion.ext_pillar
2022-09-15 14:51:41,416 [salt.loaded.ext.pillar.suma_minion:116 ][DEBUG ][500] Getting pillar data for the minion "HOST.DOMAIN.TLD"
2022-09-15 14:51:41,417 [salt.loaded.ext.pillar.suma_minion:90 ][CRITICAL][500] OPTS DEBUG: {'db': 'DBNAME_FROM_CONFIG', 'host': 'localhost', 'pass': 'CORRECT_PASSWD', 'port': 5432, 'user': 'CORRECT_USER'}
2022-09-15 14:51:41,449 [salt.loaded.ext.pillar.suma_minion:103 ][DEBUG ][500] Connected to DB
2022-09-15 14:51:41,449 [salt.loaded.ext.pillar.suma_minion:159 ][DEBUG ][500] Loading global pillars from db
2022-09-15 14:51:41,469 [salt.utils.lazy :99 ][DEBUG ][500] LazyLoaded state.apply
2022-09-15 14:51:41,474 [salt.state :750 ][DEBUG ][500] Gathering pillar data for state run
2022-09-15 14:51:41,474 [salt.pillar :70 ][DEBUG ][500] Determining pillar cache
2022-09-15 14:51:41,510 [salt.utils.lazy :99 ][DEBUG ][500] LazyLoaded jinja.render
2022-09-15 14:51:41,510 [salt.utils.lazy :99 ][DEBUG ][500] LazyLoaded yaml.render
2022-09-15 14:51:41,525 [salt.utils.lazy :99 ][DEBUG ][500] LazyLoaded suma_minion.ext_pillar
2022-09-15 14:51:41,526 [salt.loaded.ext.pillar.suma_minion:116 ][DEBUG ][500] Getting pillar data for the minion "HOST.DOMAIN.TLD"
2022-09-15 14:51:41,526 [salt.loaded.ext.pillar.suma_minion:90 ][CRITICAL][500] OPTS DEBUG: {}
2022-09-15 14:51:41,564 [salt.pillar :1169][ERROR ][500] Exception caught loading ext_pillar 'suma_minion':
File "/usr/lib/python3.6/site-packages/salt/pillar/__init__.py", line 1158, in ext_pillar
ext = self._external_pillar_data(pillar, val, key)
File "/usr/lib/python3.6/site-packages/salt/pillar/__init__.py", line 1098, in _external_pillar_data
ext = self.ext_pillars[key](self.minion_id, pillar, val)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1203, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
return callable(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1218, in _run_as
return _func_or_method(*args, **kwargs)
File "/usr/share/susemanager/modules/pillar/suma_minion.py", line 123, in ext_pillar
with _get_cursor() as cursor:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/usr/share/susemanager/modules/pillar/suma_minion.py", line 100, in _get_cursor
port=options['port'])
File "/usr/lib64/python3.6/site-packages/psycopg2/__init__.py", line 127, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
2022-09-15 14:51:41,565 [salt.pillar :1224][CRITICAL][500] Pillar render error: Failed to load ext_pillar suma_minion: connection to server at "localhost" (::1), port 5432 failed: FATAL: Ident authentication failed for user "spacewalk"
connection to server at "localhost" (::1), port 5432 failed: FATAL: Ident authentication failed for user "spacewalk"
2022-09-15 14:51:41,566 [salt.state :765 ][DEBUG ][500] Finished gathering pillar data for state run
2022-09-15 14:51:41,625 [salt.utils.lazy :99 ][DEBUG ][500] LazyLoaded jinja.render
2022-09-15 14:51:41,625 [salt.utils.lazy :99 ][DEBUG ][500] LazyLoaded yaml.render
2022-09-15 14:51:41,629 [salt.fileserver :149 ][DEBUG ][500] Returning file list from cache: age=6 cache_time=20 /var/cache/salt/master/file_lists/roots/base.p
2022-09-15 14:51:41,642 [salt.fileclient :1136][DEBUG ][500] Could not find file 'salt://certs.sls' in saltenv 'base'
2022-09-15 14:51:41,644 [salt.fileclient :1159][DEBUG ][500] In saltenv 'base', looking at rel_path 'certs/init.sls' to resolve 'salt://certs/init.sls'
2022-09-15 14:51:41,645 [salt.fileclient :1168][DEBUG ][500] In saltenv 'base', ** considering ** path '/var/tmp/.root_d6d5aa_salt/running_data/var/cache/venv-salt-minion/minion/files/base/certs/init.sls' to resolve 'salt://certs/init.sls'
2022-09-15 14:51:41,645 [salt.template :53 ][DEBUG ][500] compile template: /var/tmp/.root_d6d5aa_salt/running_data/var/cache/venv-salt-minion/minion/files/base/certs/init.sls
2022-09-15 14:51:41,647 [salt.utils.jinja :92 ][DEBUG ][500] Jinja search path: ['/var/tmp/.root_d6d5aa_salt/running_data/var/cache/venv-salt-minion/minion/files/base']
2022-09-15 14:51:41,658 [salt.fileclient :1159][DEBUG ][500] In saltenv 'base', looking at rel_path 'certs/suse.sls' to resolve 'salt://certs/suse.sls'
2022-09-15 14:51:41,659 [salt.fileclient :1168][DEBUG ][500] In saltenv 'base', ** considering ** path '/var/tmp/.root_d6d5aa_salt/running_data/var/cache/venv-salt-minion/minion/files/base/certs/suse.sls' to resolve 'salt://certs/suse.sls'
2022-09-15 14:51:41,661 [salt.template :28 ][PROFILE ][500] Time (in seconds) to render '/var/tmp/.root_d6d5aa_salt/running_data/var/cache/venv-salt-minion/minion/files/base/certs/init.sls' using 'jinja' renderer: 0.014566421508789062
2022-09-15 14:51:41,661 [salt.template :120 ][DEBUG ][500] Rendered data from file: /var/tmp/.root_d6d5aa_salt/running_data/var/cache/venv-salt-minion/minion/files/base/certs/init.sls:
mgr_absent_ca_package:
pkg.removed:
- name: rhn-org-trusted-ssl-cert
mgr_ca_cert:
file.managed:
- name: /etc/pki/trust/anchors/RHN-ORG-TRUSTED-SSL-CERT
- source: salt://certs/RHN-ORG-TRUSTED-SSL-CERTHmmmm, really strange. On the next shot the opts is empty:
2022-09-15 14:51:41,526 [salt.loaded.ext.pillar.suma_minion:90 ][CRITICAL][500] OPTS DEBUG: {}@ftigges-comdev thanks. Will debug it, how it could be possible.
vzhestkov
commented
2 years ago @ftigges-comdev I have to disturb you one more time. Sorry, but hope it could help to identify the root cause faster. Here is one more patch to try:
--- a/salt/loader/__init__.py
+++ b/salt/loader/__init__.py
@@ -479,7 +479,7 @@
_module_dirs(opts, "pillar"),
opts,
tag="pillar",
- pack={"__salt__": functions, "__context__": context, "__utils__": _utils},
+ pack={"__salt__": functions, "__context__": context, "__utils__": _utils, "__opts__": opts},
extra_module_dirs=_utils.module_dirs,
pack_self="__ext_pillar__",
)apply it with cd /usr/lib/python3.6/site-packages ; patch -p1 < /PATH/TO/PATCHFILE.
And restart salt-api service after.
Thanks in advance.
vzhestkov
commented
2 years ago @ftigges-comdev I found a way to reproduce the issue. Looking for a solution.
vzhestkov
commented
2 years ago @ftigges-comdev @phibid could you please check if the fix in https://github.com/uyuni-project/uyuni/pull/5949 works fine for you?
It's a workaround actually, but it takes some time to create a proper fix for salt.
With Uyuni 2022.06, I have spotted this error in log
/var/log/salt/api:Not sure what this pillar is used for, for the moment I have not seen any negative side effect but seems worrying to see that
spacewalkuser can't auth to the psql backend, that's why I am reporting it.