search

rpm-software-management / mock

Mock is a tool for a reproducible build of RPM packages.
GNU General Public License v2.0
376 stars 219 forks source link

`mock --init` occasinally fails with `ERROR: stat: path should be string, bytes, os.PathLike or integer, not NoneType` #1351

Open kdudka opened 3 months ago

kdudka commented 3 months ago

Short description of the problem

mock --init occasionally fails with a traceback. mock --scrub=root-cache does not fix it:

>>> 2024-03-27 01:33:04 "/usr/bin/mock" "-r" "rhel-9-x86_64" "--plugin-option=tmpfs:keep_mounted=True" "--config-opts=print_main_output=True" "--init"
INFO: mock.py version 5.2 starting (python version = 3.9.16, NVR = mock-5.2-1.kdudka.5.el9), args: /usr/libexec/mock/mock -r rhel-9-x86_64 --plugin-option=tmpfs:keep_mounted=True --config-opts=print_main_output=True --init
Start(bootstrap): init plugins
INFO: selinux enabled
Finish(bootstrap): init plugins
Start: init plugins
INFO: selinux enabled
Finish: init plugins
INFO: Signal handler active
Start: run
Start: clean chroot
Finish: clean chroot
Mock Version: 5.2
INFO: Mock Version: 5.2
INFO: Package manager dnf detected and used (fallback)
Start: chroot init
ERROR: stat: path should be string, bytes, os.PathLike or integer, not NoneType
Traceback (most recent call last):
  File "/usr/libexec/mock/mock", line 1079, in <module>
    exitStatus = main()
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 836, in main
    result = run_command(options, args, config_opts, commands, buildroot)
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 867, in run_command
    commands.init()
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.9/site-packages/mockbuild/backend.py", line 167, in init
    self.buildroot.initialize(**kwargs)
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.9/site-packages/mockbuild/buildroot.py", line 198, in initialize
    self._init(prebuild=prebuild)
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.9/site-packages/mockbuild/buildroot.py", line 281, in _init
    self._setup_nosync()
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.9/site-packages/mockbuild/buildroot.py", line 882, in _setup_nosync
    if not os.path.isdir(self.tmpdir):
  File "/usr/lib64/python3.9/genericpath.py", line 42, in isdir
    st = os.stat(s)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

!!! 2024-03-27 01:33:05 warning: failed to init mock profile (rhel-9-x86_64), trying to scrub cache...

>>> 2024-03-27 01:33:05 "/usr/bin/mock" "-r" "rhel-9-x86_64" "--plugin-option=tmpfs:keep_mounted=True" "--config-opts=print_main_output=True" "--scrub=root-cache"
INFO: mock.py version 5.2 starting (python version = 3.9.16, NVR = mock-5.2-1.kdudka.5.el9), args: /usr/libexec/mock/mock -r rhel-9-x86_64 --plugin-option=tmpfs:keep_mounted=True --config-opts=print_main_output=True --scrub=root-cache
Start(bootstrap): init plugins
INFO: selinux enabled
Finish(bootstrap): init plugins
Start: init plugins
INFO: selinux enabled
Finish: init plugins
INFO: Signal handler active
Start: run
Start: scrub ['root-cache']
INFO: scrubbing root-cache for rhel-9-x86_64
Finish: scrub ['root-cache']
Finish: run

>>> 2024-03-27 01:33:07 "/usr/bin/mock" "-r" "rhel-9-x86_64" "--plugin-option=tmpfs:keep_mounted=True" "--config-opts=print_main_output=True" "--init"
INFO: mock.py version 5.2 starting (python version = 3.9.16, NVR = mock-5.2-1.kdudka.5.el9), args: /usr/libexec/mock/mock -r rhel-9-x86_64 --plugin-option=tmpfs:keep_mounted=True --config-opts=print_main_output=True --init
Start(bootstrap): init plugins
INFO: selinux enabled
Finish(bootstrap): init plugins
Start: init plugins
INFO: selinux enabled
Finish: init plugins
INFO: Signal handler active
Start: run
Start: clean chroot
Finish: clean chroot
Mock Version: 5.2
INFO: Mock Version: 5.2
INFO: Package manager dnf detected and used (fallback)
Start: chroot init
ERROR: stat: path should be string, bytes, os.PathLike or integer, not NoneType
Traceback (most recent call last):
  File "/usr/libexec/mock/mock", line 1079, in <module>
    exitStatus = main()
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 836, in main
    result = run_command(options, args, config_opts, commands, buildroot)
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 867, in run_command
    commands.init()
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.9/site-packages/mockbuild/backend.py", line 167, in init
    self.buildroot.initialize(**kwargs)
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.9/site-packages/mockbuild/buildroot.py", line 198, in initialize
    self._init(prebuild=prebuild)
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.9/site-packages/mockbuild/buildroot.py", line 281, in _init
    self._setup_nosync()
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.9/site-packages/mockbuild/buildroot.py", line 882, in _setup_nosync
    if not os.path.isdir(self.tmpdir):
  File "/usr/lib64/python3.9/genericpath.py", line 42, in isdir
    st = os.stat(s)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

!!! 2024-03-27 01:33:08 fatal error: failed to init mock profile (rhel-9-x86_64)

Output of rpm -q mock

mock-5.2-1.kdudka.5.el9.noarch

Any additional notes

Output of `mock --debug-config`

 $ mock -r rhel-9-x86_64 --debug-config INFO: mock.py version 5.2 starting (python version = 3.9.16, NVR = mock-5.2-1.kdudka.5.el9), args: /usr/libexec/mock/mock -r rhel-9-x86_64 --debug-config Start(bootstrap): init plugins INFO: selinux enabled Finish(bootstrap): init plugins Start: init plugins INFO: selinux enabled Finish: init plugins INFO: Signal handler active Start: run config_opts['additional_packages'] = [] config_opts['bootstrap_image'] = 'registry.access.redhat.com/ubi9/ubi' config_opts['build_log_fmt_str'] = '%(message)s' config_opts['cache_alterations'] = False config_opts['chroot_name'] = 'rhel-9-x86_64' config_opts['chroot_setup_cmd'] = 'groupinstall build' config_opts['chrootuid'] = 1000 config_opts['cleanup_on_failure'] = False config_opts['cleanup_on_success'] = False config_opts['config_file'] = '/etc/mock/rhel-9-x86_64.cfg' config_opts['config_path'] = '/etc/mock' config_opts['dist'] = 'el9' config_opts['dnf.conf'] = ('\n' '[main]\n' 'cachedir=/var/cache/dnf\n' 'debuglevel=1\n' 'logfile=/var/log/yum.log\n' 'reposdir=/dev/null\n' 'retries=20\n' 'obsoletes=1\n' 'gpgcheck=0\n' 'assumeyes=1\n' 'keepcache=1\n' 'install_weak_deps=0\n' 'best=1\n' '\n' '# repos\n' '\n' '[rhel-9.4.0-build]\n' 'name=rhel-9.4.0-build\n' 'baseurl=https://download.eng.bos.redhat.com/brewroot/repos/rhel-9.4.0-build/latest/x86_64\n' 'module_hotfixes=True\n' '\n' '[rhel-9.4.0-appstream]\n' 'name=rhel-9.4.0-appstream\n' 'baseurl=https://download.eng.bos.redhat.com/rhel-9/nightly/RHEL-9/latest-RHEL-9.4.0/compose/AppStream/x86_64/os/\n' 'priority=254\n' 'module_hotfixes=True\n' 'skip_if_unavailable=True\n' '\n' '# needed for builds that have not passed gating yet\n' '[rhel-9.4.0-devtest-build]\n' 'name=rhel-9.4.0-devtest-build\n' 'baseurl=https://download.eng.bos.redhat.com/brewroot/repos/rhel-9.4.0-devtest-build/latest/x86_64\n' 'module_hotfixes=True\n' 'skip_if_unavailable=True\n' '\n' '[epel9]\n' 'name=epel9\n' 'gpgcheck=0\n' 'baseurl=https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/\n' 'priority=255\n' 'module_hotfixes=True\n' 'skip_if_unavailable=True\n' '\n' '[cov06-rhel-9]\n' 'name=cov06-rhel-9\n' 'gpgcheck=0\n' 'baseurl=https://cov06.lab.eng.brq2.redhat.com/repos/rhel-9-x86_64/\n' 'metadata_expire=60\n' 'priority=8\n' 'module_hotfixes=True\n' 'skip_if_unavailable=True\n' '\n' '[kdudka-shellcheck]\n' 'name=kdudka-shellcheck\n' 'baseurl=https://coprbe.devel.redhat.com/results/kdudka/shellcheck/epel-9-$basearch/\n' 'type=rpm-md\n' 'gpgcheck=0\n' 'gpgkey=https://coprbe.devel.redhat.com/results/kdudka/shellcheck/pubkey.gpg\n' 'repo_gpgcheck=0\n' 'enabled=1\n' 'enabled_metadata=1\n' 'cachedir=/var/cache/dnf\n' 'keepcache=1\n' 'module_hotfixes=True\n' 'skip_if_unavailable=True\n' '\n' '[kdudka-csdiff]\n' 'name=kdudka-csdiff\n' 'baseurl=https://coprbe.devel.redhat.com/results/kdudka/csdiff/epel-9-$basearch/\n' 'type=rpm-md\n' 'gpgcheck=0\n' 'gpgkey=https://coprbe.devel.redhat.com/results/kdudka/csdiff/pubkey.gpg\n' 'repo_gpgcheck=0\n' 'enabled=1\n' 'enabled_metadata=1\n' 'cachedir=/var/cache/dnf\n' 'keepcache=1\n' 'module_hotfixes=True\n' 'skip_if_unavailable=True\n' '\n' '[kdudka-cppcheck]\n' 'name=kdudka-cppcheck\n' 'baseurl=https://coprbe.devel.redhat.com/results/kdudka/cppcheck/rhel-9.dev-$basearch/\n' 'type=rpm-md\n' 'gpgcheck=0\n' 'gpgkey=https://coprbe.devel.redhat.com/results/kdudka/cppcheck/pubkey.gpg\n' 'repo_gpgcheck=0\n' 'enabled=1\n' 'enabled_metadata=1\n' 'cachedir=/var/cache/dnf\n' 'keepcache=1\n' 'module_hotfixes=True\n' 'skip_if_unavailable=True\n' '\n' '[kdudka-pylint]\n' 'name=kdudka-pylint\n' 'baseurl=https://coprbe.devel.redhat.com/results/kdudka/pylint/rhel-9.dev-$basearch/\n' 'type=rpm-md\n' 'gpgcheck=0\n' 'gpgkey=https://coprbe.devel.redhat.com/results/kdudka/pylint/pubkey.gpg\n' 'repo_gpgcheck=0\n' 'enabled=1\n' 'enabled_metadata=1\n' 'cachedir=/var/cache/dnf\n' 'keepcache=1\n' 'module_hotfixes=True\n' 'skip_if_unavailable=True\n') config_opts['dnf_warning'] = False config_opts['enable_disable_repos'] = [] config_opts['files'] = {'etc/hosts': '127.0.0.1 localhost localhost.localdomain\n' '::1 localhost localhost.localdomain localhost6 ' 'localhost6.localdomain6\n'} config_opts['isolation'] = 'nspawn' config_opts['macros'] = {'%_buildhost': 'cov03.lab.eng.brq.redhat.com', '%_host': 'x86_64-redhat-linux-gnu', '%_host_cpu': 'x86_64', '%_rpmfilename': '%%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm', '%_topdir': '/builddir/build', '%dist': '.el9', '%distribution': 'Red Hat, Inc.', '%packager': 'Red Hat, Inc.', '%vendor': 'Red Hat, Inc.'} config_opts['nspawn_args'] = ['--capability=cap_ipc_lock', '--bind=/dev/mapper/control', '--bind=/dev/fuse', '--bind=/dev/loop-control', '--bind=/dev/loop0', '--bind=/dev/loop1', '--bind=/dev/loop2', '--bind=/dev/loop3', '--bind=/dev/loop4', '--bind=/dev/loop5', '--bind=/dev/loop6', '--bind=/dev/loop7', '--bind=/dev/loop8', '--bind=/dev/loop9', '--bind=/dev/loop10', '--bind=/dev/loop11'] config_opts['plugin_conf'] = {'bind_mount_enable': True, 'bind_mount_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/rhel-9-x86_64', 'create_dirs': False, 'dirs': [], 'resultdir': '/var/lib/mock/rhel-9-x86_64/result', 'root': 'rhel-9-x86_64'}, 'ccache_enable': False, 'ccache_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/rhel-9-x86_64', 'compress': None, 'dir': '/var/cache/mock/rhel-9-x86_64/ccache/u1000/', 'max_cache_size': '4G', 'resultdir': '/var/lib/mock/rhel-9-x86_64/result', 'root': 'rhel-9-x86_64'}, 'chroot_scan_enable': False, 'chroot_scan_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/rhel-9-x86_64', 'only_failed': True, 'regexes': ['^[^k]?core(\\.\\d+)?$', '\\.log$'], 'resultdir': '/var/lib/mock/rhel-9-x86_64/result', 'root': 'rhel-9-x86_64'}, 'compress_logs_enable': False, 'compress_logs_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/rhel-9-x86_64', 'command': 'gzip', 'resultdir': '/var/lib/mock/rhel-9-x86_64/result', 'root': 'rhel-9-x86_64'}, 'hw_info_enable': True, 'hw_info_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/rhel-9-x86_64', 'resultdir': '/var/lib/mock/rhel-9-x86_64/result', 'root': 'rhel-9-x86_64'}, 'lvm_root_enable': False, 'lvm_root_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/rhel-9-x86_64', 'pool_name': 'mockbuild', 'resultdir': '/var/lib/mock/rhel-9-x86_64/result', 'root': 'rhel-9-x86_64'}, 'mount_enable': True, 'mount_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/rhel-9-x86_64', 'dirs': [], 'resultdir': '/var/lib/mock/rhel-9-x86_64/result', 'root': 'rhel-9-x86_64'}, 'overlayfs_enable': False, 'overlayfs_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/rhel-9-x86_64', 'resultdir': '/var/lib/mock/rhel-9-x86_64/result', 'root': 'rhel-9-x86_64'}, 'package_state_enable': False, 'package_state_opts': {'available_pkgs': False, 'basedir': '/var/lib/mock/rhel-9-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/rhel-9-x86_64', 'installed_pkgs': True, 'resultdir': '/var/lib/mock/rhel-9-x86_64/result', 'root': 'rhel-9-x86_64'}, 'pm_request_enable': False, 'pm_request_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/rhel-9-x86_64', 'resultdir': '/var/lib/mock/rhel-9-x86_64/result', 'root': 'rhel-9-x86_64'}, 'procenv_enable': False, 'procenv_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/rhel-9-x86_64', 'resultdir': '/var/lib/mock/rhel-9-x86_64/result', 'root': 'rhel-9-x86_64'}, 'root_cache_enable': True, 'root_cache_opts': {'age_check': True, 'basedir': '/var/lib/mock/rhel-9-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/rhel-9-x86_64', 'compress_program': 'pigz', 'decompress_program': None, 'dir': '/var/cache/mock/rhel-9-x86_64/root_cache/', 'exclude_dirs': ['./proc', './sys', './dev', './tmp/ccache', './var/cache/yum', './var/cache/dnf', './var/log'], 'extension': '.gz', 'max_age_days': 15, 'resultdir': '/var/lib/mock/rhel-9-x86_64/result', 'root': 'rhel-9-x86_64'}, 'rpkg_preprocessor_enable': False, 'rpkg_preprocessor_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/rhel-9-x86_64', 'cmd': '/usr/bin/preproc-rpmspec %(source_spec)s ' '--output %(target_spec)s', 'requires': ['preproc-rpmspec'], 'resultdir': '/var/lib/mock/rhel-9-x86_64/result', 'root': 'rhel-9-x86_64'}, 'selinux_enable': True, 'selinux_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/rhel-9-x86_64', 'resultdir': '/var/lib/mock/rhel-9-x86_64/result', 'root': 'rhel-9-x86_64'}, 'showrc_enable': False, 'showrc_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/rhel-9-x86_64', 'resultdir': '/var/lib/mock/rhel-9-x86_64/result', 'root': 'rhel-9-x86_64'}, 'sign_enable': False, 'sign_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/rhel-9-x86_64', 'cmd': 'rpmsign', 'opts': '--addsign %(rpms)s', 'resultdir': '/var/lib/mock/rhel-9-x86_64/result', 'root': 'rhel-9-x86_64'}, 'tmpfs_enable': False, 'tmpfs_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/rhel-9-x86_64', 'keep_mounted': False, 'max_fs_size': None, 'mode': '0755', 'required_ram_mb': 900, 'resultdir': '/var/lib/mock/rhel-9-x86_64/result', 'root': 'rhel-9-x86_64'}, 'yum_cache_enable': True, 'yum_cache_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/rhel-9-x86_64', 'max_age_days': 30, 'max_metadata_age_days': 30, 'online': True, 'resultdir': '/var/lib/mock/rhel-9-x86_64/result', 'root': 'rhel-9-x86_64'}} config_opts['print_main_output'] = True config_opts['releasever'] = 'el9' config_opts['root'] = 'rhel-9-x86_64' config_opts['root_log_fmt_str'] = '%(levelname)s %(filename)s:%(lineno)d: %(message)s' config_opts['rpmbuild_arch'] = 'x86_64' config_opts['rpmbuild_networking'] = True config_opts['rpmbuild_timeout'] = 86400 config_opts['state_log_fmt_str'] = '%(asctime)s - %(message)s' config_opts['target_arch'] = 'x86_64' config_opts['verbose'] = 1 Finish: run
praiskup commented 1 month ago

The buggy logic has been deactivated if config_opts['nosync'] = False (will be in Mock 5.6+) which we think is configured mostly everywhere because nosync is broken. If this is not the case, please tell us.

The problem with the nosync logic is triggered by Mock's locking, and concurrent Mock processes running against the same chroot (e.g. rhel-9-x86_64). When one Mock process performs --init, the other fails to acquire the lock and expects the chroot has already been initialized.. I'd be surprised if nosync was the only symptom of this concurrency bug.

praiskup commented 1 month ago

Thank you for reporting the problem!

kdudka commented 1 month ago

@praiskup Thanks for checking! The captured output is from csmock, which never runs mock in parallel on the same chroot. csmock implements its own locking for the whole scans (because csmock invokes mock repeatedly for each scan). Anyway, I do not remember when I saw this traceback the last time. And we have not updated mock on OSH workers since I reported this. So it might have been a bug somewhere else, which has been fixed meanwhile.

kdudka commented 1 month ago

An OSH scan of kernel-5.14.0-70.103.1.el9_0 failed with the same traceback yesterday, internally tracked as https://issues.redhat.com/browse/OSH-665. I will try to update mock to 5.6 in the hope that it will mitigate this issue.