dmurphy18
commented
1 year ago From an artifacts/log/file on Ubuntu 22.04 ARM64 which starts daemons on install
313 15:28:46,615 [pytestshellutils.customtypes :85 ][DEBUG ][MainProcess(43843)] Running Daemon._check_listening_ports(1674599926.5185254)
314 15:28:46,615 [pytestshellutils.shell :1257][DEBUG ][MainProcess(43843)] No ports to check connection to for SaltMaster(salt_pkg_install=SaltPkgInstall(conf_dir=PosixPath('/tmp/pytest-of-root/pytes t-7/salt-tests0/etc/salt'), system_service=False, proc=Subprocess(cwd=PosixPath('/tmp/testing'), slow_stop=True, timeout=None), pkgs=['/tmp/testing/pkg/artifacts/salt-syndic_3005.1+1997.g510d688a18_all.deb', '/t mp/testing/pkg/artifacts/salt-ssh_3005.1+1997.g510d688a18_all.deb', '/tmp/testing/pkg/artifacts/salt-cloud_3005.1+1997.g510d688a18_all.deb', '/tmp/testing/pkg/artifacts/salt-master_3005.1+1997.g510d688a18_all.de b', '/tmp/testing/pkg/artifacts/salt-api_3005.1+1997.g510d688a18_all.deb', '/tmp/testing/pkg/artifacts/salt-minion_3005.1+1997.g510d688a18_all.deb', '/tmp/testing/pkg/artifacts/salt-common_3005.1+1997.g510d688a1 8_all.deb'], onedir=False, singlebin=False, compressed=False, hashes={'BLAKE2B': {'file': None, 'tool': '-blake2b512'}, 'SHA3_512': {'file': None, 'tool': '-sha3-512'}, 'SHA512': {'file': None, 'tool': '-sha512' }}, root=None, run_root=None, ssm_bin=None, bin_dir=None, installer_pkg=True, upgrade=False, no_install=False, no_uninstall=False, distro_id='ubuntu', pkg_mngr='apt-get', rm_pkg='purge', salt_pkgs=['salt-api', ' salt-syndic', 'salt-ssh', 'salt-master', 'salt-cloud', 'salt-minion', 'salt-common'], install_dir=PosixPath('/opt/saltstack/salt'), binary_paths={'salt': ['salt'], 'api': ['salt-api'], 'call': ['salt-call'], 'cl oud': ['salt-cloud'], 'cp': ['salt-cp'], 'key': ['salt-key'], 'master': ['salt-master'], 'minion': ['salt-minion'], 'proxy': ['salt-proxy'], 'run': ['salt-run'], 'ssh': ['salt-ssh'], 'syndic': ['salt-syndic'], ' spm': ['spm'], 'pip': ['salt-pip'], 'python': [PosixPath('/opt/saltstack/salt/bin/python3')]}, classic=False), id='master-SrvSzS', config_dir='/tmp/pytest-of-root/pytest-7/salt-tests0/etc/salt', config_file='/tm p/pytest-of-root/pytest-7/salt-tests0/etc/salt/master', python_executable=None, cwd=PosixPath('/tmp/testing'), slow_stop=True, timeout=None, script_name='salt-master', base_script_args=[], check_ports=[], extra_ cli_arguments_after_first_start_failure=[], display_name=None)
315 15:28:46,616 [pytestshellutils.customtypes :85 ][DEBUG ][MainProcess(43843)] Running SaltDaemon._check_start_events(1674599926.5185254)
316 15:28:46,618 [saltfactories.bases :699 ][DEBUG ][MainProcess(43843)] Events to check for SaltMaster(salt_pkg_install=SaltPkgInstall(conf_dir=PosixPath('/tmp/pytest-of-root/pytest-7/salt-tests0/ etc/salt'), system_service=False, proc=Subprocess(cwd=PosixPath('/tmp/testing'), slow_stop=True, timeout=None), pkgs=['/tmp/testing/pkg/artifacts/salt-syndic_3005.1+1997.g510d688a18_all.deb', '/tmp/testing/pkg/a rtifacts/salt-ssh_3005.1+1997.g510d688a18_all.deb', '/tmp/testing/pkg/artifacts/salt-cloud_3005.1+1997.g510d688a18_all.deb', '/tmp/testing/pkg/artifacts/salt-master_3005.1+1997.g510d688a18_all.deb', '/tmp/testin g/pkg/artifacts/salt-api_3005.1+1997.g510d688a18_all.deb', '/tmp/testing/pkg/artifacts/salt-minion_3005.1+1997.g510d688a18_all.deb', '/tmp/testing/pkg/artifacts/salt-common_3005.1+1997.g510d688a18_all.deb'], one dir=False, singlebin=False, compressed=False, hashes={'BLAKE2B': {'file': None, 'tool': '-blake2b512'}, 'SHA3_512': {'file': None, 'tool': '-sha3-512'}, 'SHA512': {'file': None, 'tool': '-sha512'}}, root=None, r un_root=None, ssm_bin=None, bin_dir=None, installer_pkg=True, upgrade=False, no_install=False, no_uninstall=False, distro_id='ubuntu', pkg_mngr='apt-get', rm_pkg='purge', salt_pkgs=['salt-api', 'salt-syndic', 's alt-ssh', 'salt-master', 'salt-cloud', 'salt-minion', 'salt-common'], install_dir=PosixPath('/opt/saltstack/salt'), binary_paths={'salt': ['salt'], 'api': ['salt-api'], 'call': ['salt-call'], 'cloud': ['salt-clo ud'], 'cp': ['salt-cp'], 'key': ['salt-key'], 'master': ['salt-master'], 'minion': ['salt-minion'], 'proxy': ['salt-proxy'], 'run': ['salt-run'], 'ssh': ['salt-ssh'], 'syndic': ['salt-syndic'], 'spm': ['spm'], ' pip': ['salt-pip'], 'python': [PosixPath('/opt/saltstack/salt/bin/python3')]}, classic=False), id='master-SrvSzS', config_dir='/tmp/pytest-of-root/pytest-7/salt-tests0/etc/salt', config_file='/tmp/pytest-of-root /pytest-7/salt-tests0/etc/salt/master', python_executable=None, cwd=PosixPath('/tmp/testing'), slow_stop=True, timeout=None, script_name='salt-master', base_script_args=[], check_ports=[], extra_cli_arguments_af ter_first_start_failure=[], display_name=None): {('master-SrvSzS', 'salt/master/master-SrvSzS/start')}
317 15:28:46,619 [saltfactories.plugins.event_listener:329 ][DEBUG ][MainProcess(43843)] EventListener(timeout=120, address='tcp://127.0.0.1:47139') is checking for event patterns happening after 2023-01-24T22:28: 46.517969: {('master-SrvSzS', 'salt/master/master-SrvSzS/start')}
318 15:28:46,620 [saltfactories.plugins.event_listener:358 ][DEBUG ][MainProcess(43843)] EventListener(timeout=120, address='tcp://127.0.0.1:47139') did not find any matching event patterns happening after 2023-01 -24T22:28:46.517969
319 15:28:47,970 [salt.utils.parsers :222 ][DEBUG ][MainProcess(45690)] [SaltMaster(id='master-SrvSzS')] Configuration file path: /tmp/pytest-of-root/pytest-7/salt-tests0/etc/salt/master
320 15:28:47,984 [salt.cli.daemons :83 ][INFO ][MainProcess(45690)] [SaltMaster(id='master-SrvSzS')] Setting up the Salt Master
321 15:28:47,985 [salt.utils.verify :126 ][WARNING ][MainProcess(45690)] [SaltMaster(id='master-SrvSzS')] Unable to bind socket 127.0.0.1:4505, error: [Errno 98] Address already in use; Is there an other salt-master running?
322 15:28:47,986 [salt.cli.daemons :99 ][INFO ][MainProcess(45690)] [SaltMaster(id='master-SrvSzS')] The Salt Master is shut down
323 15:28:48,122 [saltfactories.plugins.event_listener:329 ][DEBUG ][MainProcess(43843)] EventListener(timeout=120, address='tcp://127.0.0.1:47139') is checking for event patterns happening after 2023-01-24T22:28: 46.517969: {('master-SrvSzS', 'salt/master/master-SrvSzS/start')}
324 15:28:48,123 [saltfactories.plugins.event_listener:358 ][DEBUG ][MainProcess(43843)] EventListener(timeout=120, address='tcp://127.0.0.1:47139') did not find any matching event patterns happening after 2023-01 -24T22:28:46.517969
325 15:28:49,627 [pytestshellutils.shell :1229][INFO ][MainProcess(43843)] Exception raised when running SaltDaemon._check_start_events(): SaltMaster(salt_pkg_install=SaltPkgInstall(conf_dir=PosixPat h('/tmp/pytest-of-root/pytest-7/salt-tests0/etc/salt'), system_service=False, proc=Subprocess(cwd=PosixPath('/tmp/testing'), slow_stop=True, timeout=None), pkgs=['/tmp/testing/pkg/artifacts/salt-syndic_3005.1+19 97.g510d688a18_all.deb', '/tmp/testing/pkg/artifacts/salt-ssh_3005.1+1997.g510d688a18_all.deb', '/tmp/testing/pkg/artifacts/salt-cloud_3005.1+1997.g510d688a18_all.deb', '/tmp/testing/pkg/artifacts/salt-master_30 05.1+1997.g510d688a18_all.deb', '/tmp/testing/pkg/artifacts/salt-api_3005.1+1997.g510d688a18_all.deb', '/tmp/testing/pkg/artifacts/salt-minion_3005.1+1997.g510d688a18_all.deb', '/tmp/testing/pkg/artifacts/salt-c ommon_3005.1+1997.g510d688a18_all.deb'], onedir=False, singlebin=False, compressed=False, hashes={'BLAKE2B': {'file': None, 'tool': '-blake2b512'}, 'SHA3_512': {'file': None, 'tool': '-sha3-512'}, 'SHA512': {'fi le': None, 'tool': '-sha512'}}, root=None, run_root=None, ssm_bin=None, bin_dir=None, installer_pkg=True, upgrade=False, no_install=False, no_uninstall=False, distro_id='ubuntu', pkg_mngr='apt-get', rm_pkg='purg e', salt_pkgs=['salt-api', 'salt-syndic', 'salt-ssh', 'salt-master', 'salt-cloud', 'salt-minion', 'salt-common'], install_dir=PosixPath('/opt/saltstack/salt'), binary_paths={'salt': ['salt'], 'api': ['salt-api'] , 'call': ['salt-call'], 'cloud': ['salt-cloud'], 'cp': ['salt-cp'], 'key': ['salt-key'], 'master': ['salt-master'], 'minion': ['salt-minion'], 'proxy': ['salt-proxy'], 'run': ['salt-run'], 'ssh': ['salt-ssh'], 'syndic': ['salt-syndic'], 'spm': ['spm'], 'pip': ['salt-pip'], 'python': [PosixPath('/opt/saltstack/salt/bin/python3')]}, classic=False), id='master-SrvSzS', config_dir='/tmp/pytest-of-root/pytest-7/salt-tests0 /etc/salt', config_file='/tmp/pytest-of-root/pytest-7/salt-tests0/etc/salt/master', python_executable=None, cwd=PosixPath('/tmp/testing'), slow_stop=True, timeout=None, script_name='salt-master', base_script_arg s=[], check_ports=[], extra_cli_arguments_after_first_start_failure=[], display_name=None) is no longer running
326 Traceback (most recent call last):
327 File "/tmp/testing/.nox/test-pkgs-3-coverage-false/lib/python3.10/site-packages/pytestshellutils/shell.py", line 1225, in run_start_checks
328 ret = start_check(timeout_at)
329 File "/tmp/testing/.nox/test-pkgs-3-coverage-false/lib/python3.10/site-packages/pytestshellutils/customtypes.py", line 86, in __call__
330 return self.func(*_args, **_kwargs)
331 File "/tmp/testing/.nox/test-pkgs-3-coverage-false/lib/python3.10/site-packages/saltfactories/bases.py", line 703, in _check_start_events
332 raise FactoryNotStarted("{} is no longer running".format(self))
333 pytestshellutils.exceptions.FactoryNotStarted: SaltMaster(salt_pkg_install=SaltPkgInstall(conf_dir=PosixPath('/tmp/pytest-of-root/pytest-7/salt-tests0/etc/salt'), system_service=False, proc=Subprocess(cwd=PosixP ath('/tmp/testing'), slow_stop=True, timeout=None), pkgs=['/tmp/testing/pkg/artifacts/salt-syndic_3005.1+1997.g510d688a18_all.deb', '/tmp/testing/pkg/artifacts/salt-ssh_3005.1+1997.g510d688a18_all.deb', '/tmp/te sting/pkg/artifacts/salt-cloud_3005.1+1997.g510d688a18_all.deb', '/tmp/testing/pkg/artifacts/salt-master_3005.1+1997.g510d688a18_all.deb', '/tmp/testing/pkg/artifacts/salt-api_3005.1+1997.g510d688a18_all.deb', ' /tmp/testing/pkg/artifacts/salt-minion_3005.1+1997.g510d688a18_all.deb', '/tmp/testing/pkg/artifacts/salt-common_3005.1+1997.g510d688a18_all.deb'], onedir=False, singlebin=False, compressed=False, hashes={'BLAKE 2B': {'file': None, 'tool': '-blake2b512'}, 'SHA3_512': {'file': None, 'tool': '-sha3-512'}, 'SHA512': {'file': None, 'tool': '-sha512'}}, root=None, run_root=None, ssm_bin=None, bin_dir=None, installer_pkg=True , upgrade=False, no_install=False, no_uninstall=False, distro_id='ubuntu', pkg_mngr='apt-get', rm_pkg='purge', salt_pkgs=['salt-api', 'salt-syndic', 'salt-ssh', 'salt-master', 'salt-cloud', 'salt-minion', 'salt- common'], install_dir=PosixPath('/opt/saltstack/salt'), binary_paths={'salt': ['salt'], 'api': ['salt-api'], 'call': ['salt-call'], 'cloud': ['salt-cloud'], 'cp': ['salt-cp'], 'key': ['salt-key'], 'master': ['sa lt-master'], 'minion': ['salt-minion'], 'proxy': ['salt-proxy'], 'run': ['salt-run'], 'ssh': ['salt-ssh'], 'syndic': ['salt-syndic'], 'spm': ['spm'], 'pip': ['salt-pip'], 'python': [PosixPath('/opt/saltstack/sal t/bin/python3')]}, classic=False), id='master-SrvSzS', config_dir='/tmp/pytest-of-root/pytest-7/salt-tests0/etc/salt', config_file='/tmp/pytest-of-root/pytest-7/salt-tests0/etc/salt/master', python_executable=No ne, cwd=PosixPath('/tmp/testing'), slow_stop=True, timeout=None, script_name='salt-master', base_script_args=[], check_ports=[], extra_cli_arguments_after_first_start_failure=[], display_name=None) is no longer running
The crucial Unable to bind socket 127.0.0.1:4505, error: [Errno 98] Address already in use; Is there an other salt-master running?
And of course there is since install will also start the daemon.
Description The new package tests attempt to install DEB packages for our smoke testing when run in deb-based distributions. By default, on these distributions, these services start automatically. Because of this, when the tests try to spin up a salt master service, the ports are unavailable to bind.
In previous iterations of these tests (i.e. those currently run on salt-pkg), the tests were run in a docker container under privileged mode. The quirks of this setup are believed to be the cause of why they are failing now when being run in a golden image VM.
Setup
Steps to Reproduce the behavior Run the salt package tests in
pkg/tests/against salt deb packages. It should fail with aFactoryNotStarted.Expected behavior The salt factories should spin up and work just fine.
Additional context From looking into this a while ago, I remember this article (https://major.io/2016/05/05/preventing-ubuntu-16-04-starting-daemons-package-installed/) along with some minor code changes (pretty sure just an additional
daemon-reloadinpkg/tests/support/helpers.py::write_systemd_conf) fixed the problem.