search

rpm-software-management / mock

Mock is a tool for a reproducible build of RPM packages.
GNU General Public License v2.0
386 stars 235 forks source link

RFE: improve ccache plugin usability by enabling cache hits across package NEVRs #1395

Closed belegdol closed 4 months ago

belegdol commented 5 months ago

Short description of the problem

This is a continuation of a discussion started on fedora-devel ML. Mock ccache plugin, as configured currently, only speeds up re-builds of exactly the same NEVR, due to the majority of packages including NEVR in the builddir. By default, the current working dir is included in the ccache hash, resulting in no cache hits. I have experimented (somewhat crudely) with hash_dir and base_dir options, and was successfully able to get ccache to find hits across builds. Here is the patch I used:

From 43ea7be73b257db0eea2597ae961d520186d45dc Mon Sep 17 00:00:00 2001
From: rpm-build <rpm-build>
Date: Sat, 29 Jun 2024 14:25:01 +0200
Subject: [PATCH] Enable ccache debugging

---
 py/mockbuild/plugins/ccache.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/py/mockbuild/plugins/ccache.py b/py/mockbuild/plugins/ccache.py
index 1a20846..e1058f2 100644
--- a/py/mockbuild/plugins/ccache.py
+++ b/py/mockbuild/plugins/ccache.py
@@ -57,7 +57,7 @@ class CCache(object):
     @traceLog()
     def _ccachePreInitHook(self):
         getLog().info("enabled ccache")
-        envupd = {"CCACHE_DIR": "/var/tmp/ccache", "CCACHE_UMASK": "002"}
+        envupd = {"CCACHE_DIR": "/var/tmp/ccache", "CCACHE_UMASK": "002", "CCACHE_DEBUG": "1", "CCACHE_DEBUGDIR": "/var/tmp/ccache", "CCACHE_BASEDIR": "/builddir/build/BUILD/kernel-6.9.7/linux-6.9.7-200.ds01.fc40.x86_64", "CCACHE_NOHASHDIR": "1"}
         if self.ccache_opts.get('compress') is not None:
             envupd["CCACHE_COMPRESS"] = str(self.ccache_opts['compress'])
         self.buildroot.env.update(envupd)
-- 
2.45.2

There are two issues with these options:

  1. hash_dir can apparently cause issues with debuginfo:

The reason for including the CWD in the hash by default is to prevent a problem with the storage of the current working directory in the debug info of an object file, which can lead ccache to return a cached object file that has the working directory in the debug info set incorrectly.

Not sure if it a problem for rpm builds or if the debug info files even include the working dir

  1. base_dir is package dependent and mock would somehow need to wait until rpmbuild creates the builddir and only then add the value to the environmental variable

Output of rpm -q mock

mock-5.6-3.fc40.test.noarch

Steps to reproduce issue

  1. Build a package in mock with ccache plugin enabled
  2. Update a single file in the package
  3. Bump RPM release
  4. Rebuild the second RPM
  5. See almost no ccache hits

Command line used:

mock --chain --localrepo /mnt/openmediavault/kernel/ -r fedora-40-x86_64 --disable-plugin=tmpfs --enable-plugin=yum_cache --enable-plugin=ccache --plugin-option=ccache:max_cache_size='40G' --isolation=simple redhat/rpm/SRPMS/kernel-6.9.7-200.test.fc40.src.rpm --with baseonly

Any additional notes

Output of `mock --debug-config`

 ``` INFO: mock.py version 5.6 starting (python version = 3.12.3, NVR = mock-5.6-3.fc40.test), args: /usr/libexec/mock/mock --debug-config Start(bootstrap): init plugins INFO: tmpfs initialized INFO: selinux enabled Finish(bootstrap): init plugins Start: init plugins INFO: tmpfs initialized INFO: selinux enabled Finish: init plugins INFO: Signal handler active Start: run config_opts['additional_packages'] = [] config_opts['bootstrap_image'] = 'registry.fedoraproject.org/fedora:{{ releasever }}' config_opts['build_log_fmt_str'] = '%(message)s' config_opts['cache_alterations'] = False config_opts['chroot_name'] = 'default' config_opts['chroot_setup_cmd'] = 'install @{% if mirrored %}buildsys-{% endif %}build' config_opts['chrootuid'] = 1000 config_opts['cleanup_on_failure'] = False config_opts['cleanup_on_success'] = False config_opts['config_file'] = '/etc/mock/default.cfg' config_opts['config_path'] = '/etc/mock' config_opts['config_paths'] = ['/etc/mock/fedora-40-x86_64.cfg', '/etc/mock/templates/fedora-branched.tpl', '/etc/mock/chroot-aliases.cfg', '/etc/mock/site-defaults.cfg'] config_opts['copy_host_users'] = ['pesign'] config_opts['createrepo_on_rpms'] = True config_opts['description'] = 'Fedora {{ releasever }}' config_opts['dist'] = 'fc{{ releasever }}' config_opts['dnf.conf'] = ('\n' '[main]\n' 'keepcache=1\n' 'system_cachedir=/var/cache/dnf\n' 'debuglevel=2\n' 'reposdir=/dev/null\n' 'logfile=/var/log/yum.log\n' 'retries=20\n' 'obsoletes=1\n' 'gpgcheck=0\n' 'assumeyes=1\n' 'syslog_ident=mock\n' 'syslog_device=\n' 'install_weak_deps=0\n' 'metadata_expire=0\n' 'best=1\n' 'module_platform_id=platform:f{{ releasever }}\n' 'protected_packages=\n' 'user_agent={{ user_agent }}\n' '\n' '# repos\n' '\n' '[local]\n' 'name=local\n' 'baseurl=https://kojipkgs.fedoraproject.org/repos/f{{ releasever ' '}}-build/latest/$basearch/\n' 'cost=2000\n' 'enabled={{ not mirrored }}\n' 'skip_if_unavailable=False\n' '\n' '{% if mirrored %}\n' '[fedora]\n' 'name=fedora\n' 'metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch\n' 'gpgkey=file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-{{ ' 'releasever }}-primary\n' 'gpgcheck=1\n' 'skip_if_unavailable=False\n' '\n' '[updates]\n' 'name=updates\n' 'metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-f$releasever&arch=$basearch\n' 'gpgkey=file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-{{ ' 'releasever }}-primary\n' 'gpgcheck=1\n' 'skip_if_unavailable=False\n' '\n' '[updates-testing]\n' 'name=updates-testing\n' 'metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-testing-f$releasever&arch=$basearch\n' 'enabled=0\n' 'gpgkey=file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-{{ ' 'releasever }}-primary\n' 'gpgcheck=1\n' 'skip_if_unavailable=False\n' '\n' '[fedora-debuginfo]\n' 'name=fedora-debuginfo\n' 'metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-debug-$releasever&arch=$basearch\n' 'enabled=0\n' 'gpgkey=file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-{{ ' 'releasever }}-primary\n' 'gpgcheck=1\n' 'skip_if_unavailable=False\n' '\n' '[updates-debuginfo]\n' 'name=updates-debuginfo\n' 'metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-debug-f$releasever&arch=$basearch\n' 'enabled=0\n' 'gpgkey=file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-{{ ' 'releasever }}-primary\n' 'gpgcheck=1\n' 'skip_if_unavailable=False\n' '\n' '[updates-testing-debuginfo]\n' 'name=updates-testing-debuginfo\n' 'metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-testing-debug-f$releasever&arch=$basearch\n' 'enabled=0\n' 'gpgkey=file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-{{ ' 'releasever }}-primary\n' 'gpgcheck=1\n' 'skip_if_unavailable=False\n' '\n' '[fedora-source]\n' 'name=fedora-source\n' 'metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-source-$releasever&arch=$basearch\n' 'gpgkey=file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-{{ ' 'releasever }}-primary\n' 'gpgcheck=1\n' 'enabled=0\n' 'skip_if_unavailable=False\n' '\n' '[updates-source]\n' 'name=updates-source\n' 'metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-source-f$releasever&arch=$basearch\n' 'gpgkey=file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-{{ ' 'releasever }}-primary\n' 'gpgcheck=1\n' 'enabled=0\n' 'skip_if_unavailable=False\n' '{% endif %}\n') config_opts['enable_disable_repos'] = [] config_opts['extra_chroot_dirs'] = ['/run/lock'] config_opts['files'] = {'etc/hosts': '127.0.0.1 localhost localhost.localdomain\n' '::1 localhost localhost.localdomain localhost6 ' 'localhost6.localdomain6\n'} config_opts['legal_host_arches'] = ('x86_64',) config_opts['macros'] = {'%_buildhost': 'napoleon2.local', '%_platform_multiplier': 1, '%_rpmfilename': '%%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm', '%_topdir': '/builddir/build'} config_opts['mirrored'] = True config_opts['no-config'] = {'epel-8': {'alternatives': {'alma+epel-8': {'description': ['Builds against ' 'AlmaLinux 8 ' 'repositories, ' 'together with ' 'the official ' 'EPEL ' 'repositories.', 'Project page: ' 'https://almalinux.org/']}, 'centos-stream+epel-8': {'description': ['Builds ' 'against ' 'CentOS ' 'Stream ' '8 ' 'repositories ' '(some ' 'packages ' 'may be ' 'a bit ' 'ahead ' 'the Red ' 'Hat ' 'Enterprise ' 'Linux ' '8) ' 'together ' 'with ' 'the ' 'official ' 'EPEL 8 ' 'repositories.', 'Project ' 'page: ' 'https://www.centos.org/centos-stream/']}, 'circlelinux+epel-8': {'description': ['Builds ' 'against ' 'Circle ' 'Linux 8 ' 'repositories, ' 'together ' 'with the ' 'official ' 'EPEL ' 'repositories.', 'Project ' 'page: ' 'https://cclinux.org/']}, 'rhel+epel-8': {'description': ['Builds against ' 'Red Hat ' 'Enterprise Linux ' '8 repositories, ' 'together with ' 'the official ' 'EPEL ' 'repositories.', 'This mimics what ' 'is done in the ' 'official EPEL ' 'build system, ' 'but you need a ' 'Red Hat ' 'subscription:', 'https://rpm-software-management.github.io/mock/Feature-rhelchroots']}, 'rocky+epel-8': {'description': ['Builds against ' 'Rocky Linux 8 ' 'repositories, ' 'together with ' 'the official ' 'EPEL ' 'repositories.', 'Project page: ' 'https://rockylinux.org/']}}}, 'epel-9': {'alternatives': {'alma+epel-9': {'description': ['Builds against ' 'AlmaLinux 9 ' 'repositories, ' 'together with ' 'the official ' 'EPEL ' 'repositories.', 'Project page: ' 'https://almalinux.org/']}, 'centos-stream+epel-9': {'description': ['Builds ' 'against ' 'CentOS ' 'Stream ' '9 ' 'repositories ' '(some ' 'packages ' 'may be ' 'a bit ' 'ahead ' 'the Red ' 'Hat ' 'Enterprise ' 'Linux ' '9) ' 'together ' 'with ' 'the ' 'official ' 'EPEL ' 'repositories.', 'Project ' 'page: ' 'https://www.centos.org/centos-stream/']}, 'rhel+epel-9': {'description': ['Builds against ' 'Red Hat ' 'Enterprise Linux ' '9 repositories, ' 'together with ' 'the official ' 'EPEL ' 'repositories.', 'This mimics what ' 'is done in the ' 'official EPEL ' 'build system, ' 'but you need a ' 'Red Hat ' 'subscription:', 'https://rpm-software-management.github.io/mock/Feature-rhelchroots']}, 'rocky+epel-9': {'description': ['Builds against ' 'Rocky Linux 9 ' 'repositories, ' 'together with ' 'the official ' 'EPEL ' 'repositories.', 'Project page: ' 'https://rockylinux.org/']}}}} config_opts['nspawn_args'] = ['--capability=cap_ipc_lock', '--bind=/tmp/mock-resolv.3cqlops4:/etc/resolv.conf', '--bind=/dev/btrfs-control', '--bind=/dev/mapper/control', '--bind=/dev/fuse', '--bind=/dev/loop-control', '--bind=/dev/loop0', '--bind=/dev/loop1', '--bind=/dev/loop2', '--bind=/dev/loop3', '--bind=/dev/loop4', '--bind=/dev/loop5', '--bind=/dev/loop6', '--bind=/dev/loop7', '--bind=/dev/loop8', '--bind=/dev/loop9', '--bind=/dev/loop10', '--bind=/dev/loop11'] config_opts['package_manager'] = '{% if releasever|int >= 40 %}dnf5{% else %}dnf{% endif %}' config_opts['plugin_conf'] = {'__init___enable': False, '__init___opts': {'basedir': '/var/lib/mock/fedora-40-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-40-x86_64', 'resultdir': '/var/lib/mock/fedora-40-x86_64/result', 'root': 'fedora-40-x86_64'}, 'bind_mount_enable': True, 'bind_mount_opts': {'basedir': '/var/lib/mock/fedora-40-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-40-x86_64', 'create_dirs': False, 'dirs': [('/var/run/pesign', '/var/run/pesign')], 'resultdir': '/var/lib/mock/fedora-40-x86_64/result', 'root': 'fedora-40-x86_64'}, 'ccache_enable': False, 'ccache_opts': {'basedir': '/var/lib/mock/fedora-40-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-40-x86_64', 'compress': None, 'dir': '/var/cache/mock/fedora-40-x86_64/ccache/u1000/', 'max_cache_size': '4G', 'resultdir': '/var/lib/mock/fedora-40-x86_64/result', 'root': 'fedora-40-x86_64'}, 'chroot_scan_enable': False, 'chroot_scan_opts': {'basedir': '/var/lib/mock/fedora-40-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-40-x86_64', 'only_failed': True, 'regexes': ['^[^k]?core(\\.\\d+)?$', '\\.log$'], 'resultdir': '/var/lib/mock/fedora-40-x86_64/result', 'root': 'fedora-40-x86_64', 'write_tar': False}, 'compress_logs_enable': False, 'compress_logs_opts': {'basedir': '/var/lib/mock/fedora-40-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-40-x86_64', 'command': 'gzip', 'resultdir': '/var/lib/mock/fedora-40-x86_64/result', 'root': 'fedora-40-x86_64'}, 'hw_info_enable': True, 'hw_info_opts': {'basedir': '/var/lib/mock/fedora-40-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-40-x86_64', 'resultdir': '/var/lib/mock/fedora-40-x86_64/result', 'root': 'fedora-40-x86_64'}, 'lvm_root_enable': False, 'lvm_root_opts': {'basedir': '/var/lib/mock/fedora-40-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-40-x86_64', 'pool_name': 'mockbuild', 'resultdir': '/var/lib/mock/fedora-40-x86_64/result', 'root': 'fedora-40-x86_64'}, 'mount_enable': True, 'mount_opts': {'basedir': '/var/lib/mock/fedora-40-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-40-x86_64', 'dirs': [], 'resultdir': '/var/lib/mock/fedora-40-x86_64/result', 'root': 'fedora-40-x86_64'}, 'overlayfs_enable': False, 'overlayfs_opts': {'basedir': '/var/lib/mock/fedora-40-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-40-x86_64', 'resultdir': '/var/lib/mock/fedora-40-x86_64/result', 'root': 'fedora-40-x86_64'}, 'package_state_enable': True, 'package_state_opts': {'available_pkgs': False, 'basedir': '/var/lib/mock/fedora-40-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-40-x86_64', 'installed_pkgs': True, 'resultdir': '/var/lib/mock/fedora-40-x86_64/result', 'root': 'fedora-40-x86_64'}, 'pesign_enable': False, 'pesign_opts': {'basedir': '/var/lib/mock/fedora-40-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-40-x86_64', 'resultdir': '/var/lib/mock/fedora-40-x86_64/result', 'root': 'fedora-40-x86_64'}, 'pm_request_enable': False, 'pm_request_opts': {'basedir': '/var/lib/mock/fedora-40-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-40-x86_64', 'resultdir': '/var/lib/mock/fedora-40-x86_64/result', 'root': 'fedora-40-x86_64'}, 'procenv_enable': False, 'procenv_opts': {'basedir': '/var/lib/mock/fedora-40-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-40-x86_64', 'resultdir': '/var/lib/mock/fedora-40-x86_64/result', 'root': 'fedora-40-x86_64'}, 'root_cache_enable': True, 'root_cache_opts': {'age_check': True, 'basedir': '/var/lib/mock/fedora-40-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-40-x86_64', 'compress_program': 'pigz', 'decompress_program': None, 'dir': '/var/cache/mock/fedora-40-x86_64/root_cache/', 'exclude_dirs': ['./proc', './sys', './dev', './tmp/ccache', './var/cache/yum', './var/cache/dnf', './var/log'], 'extension': '.gz', 'max_age_days': 15, 'resultdir': '/var/lib/mock/fedora-40-x86_64/result', 'root': 'fedora-40-x86_64'}, 'rpkg_preprocessor_enable': False, 'rpkg_preprocessor_opts': {'basedir': '/var/lib/mock/fedora-40-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-40-x86_64', 'cmd': '/usr/bin/preproc-rpmspec %(source_spec)s ' '--output %(target_spec)s', 'requires': ['preproc-rpmspec'], 'resultdir': '/var/lib/mock/fedora-40-x86_64/result', 'root': 'fedora-40-x86_64'}, 'rpmautospec_enable': False, 'rpmautospec_opts': {'basedir': '/var/lib/mock/fedora-40-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-40-x86_64', 'cmd_base': ['/usr/bin/rpmautospec', 'process-distgit'], 'requires': ['rpmautospec'], 'resultdir': '/var/lib/mock/fedora-40-x86_64/result', 'root': 'fedora-40-x86_64'}, 'selinux_enable': True, 'selinux_opts': {'basedir': '/var/lib/mock/fedora-40-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-40-x86_64', 'resultdir': '/var/lib/mock/fedora-40-x86_64/result', 'root': 'fedora-40-x86_64'}, 'showrc_enable': False, 'showrc_opts': {'basedir': '/var/lib/mock/fedora-40-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-40-x86_64', 'resultdir': '/var/lib/mock/fedora-40-x86_64/result', 'root': 'fedora-40-x86_64'}, 'sign_enable': False, 'sign_opts': {'basedir': '/var/lib/mock/fedora-40-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-40-x86_64', 'cmd': 'rpmsign', 'opts': '--addsign %(rpms)s', 'resultdir': '/var/lib/mock/fedora-40-x86_64/result', 'root': 'fedora-40-x86_64'}, 'tmpfs_enable': True, 'tmpfs_opts': {'basedir': '/var/lib/mock/fedora-40-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-40-x86_64', 'keep_mounted': False, 'max_fs_size': '16G', 'mode': '0755', 'required_ram_mb': 900, 'resultdir': '/var/lib/mock/fedora-40-x86_64/result', 'root': 'fedora-40-x86_64'}, 'yum_cache_enable': False, 'yum_cache_opts': {'basedir': '/var/lib/mock/fedora-40-x86_64', 'cache_topdir': '/var/cache/mock', 'cachedir': '/var/cache/mock/fedora-40-x86_64', 'max_age_days': 30, 'max_metadata_age_days': 30, 'online': True, 'resultdir': '/var/lib/mock/fedora-40-x86_64/result', 'root': 'fedora-40-x86_64'}} config_opts['print_main_output'] = True config_opts['releasever'] = '40' config_opts['repo_arch'] = 'x86_64' config_opts['root'] = 'fedora-{{ releasever }}-{{ target_arch }}' config_opts['root_log_fmt_str'] = '%(levelname)s %(filename)s:%(lineno)d: %(message)s' config_opts['rpmbuild_arch'] = 'x86_64' config_opts['state_log_fmt_str'] = '%(asctime)s - %(message)s' config_opts['target_arch'] = 'x86_64' config_opts['verbose'] = 1 Finish: run ```
belegdol commented 5 months ago

Looks like "CCACHE_NOHASHDIR": "1" is sufficient, which at least makes implementation simple. What remains to be clarified is whether there is any impact on debug info files.

xsuchy commented 5 months ago

CCache is not used in any build system Fedora use, so this is very low priority for us. If you submit PR that set this variable (even for all builds) we will review and accept it. You can take example from https://github.com/rpm-software-management/mock/blob/main/mock/py/mockbuild/plugins/ccache.py#L56 and test if this helps.