Closed brianjmurrell closed 4 years ago
praiskup
commented
4 years ago Can you please retry with mock-core-configs 32.3? You seem to have
configured yum.conf but mock 2.0 requires dnf.conf. Or maybe you want
to try pre-release mock from dnf copr enable @mock/mock (where dnf.conf
is equivalent to yum.conf).
brianjmurrell
commented
4 years ago Unfortunately, this is a "production" builder which has no ability to pull in non-production RPMs.
But your explanation:
You seem to have configured yum.conf but mock 2.0 requires dnf.conf
seems to indicate that I can resolve this through configuration by simply using dnf.conf instead of yum.conf. Is that correct? But I don't think I touch either of those files. That is driven by /etc/mock/opensuse-leap-15.1-x86_64.cfg isn't it? Should I s/yum.conf/dnf.conf/g in /etc/mock/opensuse-leap-15.1-x86_64.cfg?
Or am I completely misunderstanding your suggestion?
brianjmurrell
commented
4 years ago FWIW, replacing yum.conf with dnf.conf in /etc/mock/opensuse-leap-15.1-x86_64.cfg does not resolve the issue.
$ mock -r opensuse-leap-15.1-x86_64 --debug-config
/bin/sh: warning: setlocale: LC_ALL: cannot change locale (en_US.utf8): No such file or directory
INFO: mock.py version 2.0 starting (python version = 3.7.4)...
Start(bootstrap): init plugins
INFO: selinux disabled
Finish(bootstrap): init plugins
Start: init plugins
INFO: selinux disabled
Finish: init plugins
INFO: Signal handler active
Start: run
config_opts['build_log_fmt_str'] = '%(message)s'
config_opts['cache_alterations'] = False
config_opts['chroot_name'] = 'opensuse-leap-15.1-x86_64'
config_opts['chroot_setup_cmd'] = 'install patterns-devel-base-devel_rpm_build'
config_opts['cleanup_on_failure'] = False
config_opts['cleanup_on_success'] = False
config_opts['config_file'] = '/etc/mock/opensuse-leap-15.1-x86_64.cfg'
config_opts['config_path'] = '/etc/mock'
config_opts['config_paths'] = ['/etc/mock/site-defaults.cfg', '/etc/mock/opensuse-leap-15.1-x86_64.cfg']
config_opts['dist'] = 'suse.lp151'
config_opts['dnf.conf'] = ('\n'
'[main]\n'
'keepcache=1\n'
'debuglevel=2\n'
'reposdir=/dev/null\n'
'logfile=/var/log/yum.log\n'
'retries=20\n'
'obsoletes=1\n'
'gpgcheck=0\n'
'assumeyes=1\n'
'syslog_ident=mock\n'
'syslog_device=\n'
'install_weak_deps=0\n'
'metadata_expire=0\n'
'best=1\n'
'excludepkgs=*.i586,*.i686\n'
'protected_packages=\n'
'\n'
'# repos\n'
'\n'
'[opensuse-leap-oss]\n'
'name=openSUSE Leap $releasever - x86_64 - OSS\n'
'#baseurl=http://download.opensuse.org/distribution/leap/$releasever/repo/oss/\n'
'metalink=http://download.opensuse.org/distribution/leap/$releasever/repo/oss/repodata/repomd.xml.metalink\n'
'gpgkey=file:///usr/share/distribution-gpg-keys/opensuse/RPM-GPG-KEY-openSUSE\n'
'gpgcheck=1\n'
'\n'
'[updates-oss]\n'
'name=openSUSE Leap $releasever - x86_64 - Updates - OSS\n'
'#baseurl=http://download.opensuse.org/update/leap/$releasever/oss/\n'
'metalink=http://download.opensuse.org/update/leap/$releasever/oss/repodata/repomd.xml.metalink\n'
'gpgkey=file:///usr/share/distribution-gpg-keys/opensuse/RPM-GPG-KEY-openSUSE\n'
'gpgcheck=1\n'
'\n'
'\n'
'\n'
'[repo.dc.hpdd.intel.com_repository_daos-stack-leap-15-x86_64-stable-local_]\n'
'name=repo.dc.hpdd.intel.com_repository_daos-stack-leap-15-x86_64-stable-local_\n'
'baseurl=https://repo.dc.hpdd.intel.com/repository/daos-stack-leap-15-x86_64-stable-local/\n'
'enabled=1\n'
'\n'
'[repo.dc.hpdd.intel.com_repository_daos-stack-external-leap-15-x86_64-stable-group_]\n'
'name=repo.dc.hpdd.intel.com_repository_daos-stack-external-leap-15-x86_64-stable-group_\n'
'baseurl=https://repo.dc.hpdd.intel.com/repository/daos-stack-external-leap-15-x86_64-stable-group/\n'
'enabled=1\n'
'\n')
config_opts['enable_disable_repos'] = []
config_opts['extra_chroot_dirs'] = ['/run/lock']
config_opts['files'] = {'etc/hosts': '127.0.0.1 localhost localhost.localdomain\n'
'::1 localhost localhost.localdomain localhost6 '
'localhost6.localdomain6'}
config_opts['legal_host_arches'] = ('x86_64',)
config_opts['macros'] = {'%_buildhost': 'f24fd1d6271e',
'%_rpmfilename': '%%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm',
'%_topdir': '/builddir/build',
'%dist': '.suse.lp151'}
config_opts['plugin_conf'] = {'bind_mount_enable': True,
'bind_mount_opts': {'basedir': '/var/lib/mock/opensuse-leap-15.1-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/opensuse-leap-15.1-x86_64',
'create_dirs': False,
'dirs': [],
'resultdir': '/var/lib/mock/opensuse-leap-15.1-x86_64/result',
'root': 'opensuse-leap-15.1-x86_64'},
'ccache_enable': False,
'ccache_opts': {'basedir': '/var/lib/mock/opensuse-leap-15.1-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/opensuse-leap-15.1-x86_64',
'compress': None,
'dir': '/var/cache/mock/opensuse-leap-15.1-x86_64/ccache/u1101/',
'max_cache_size': '4G',
'resultdir': '/var/lib/mock/opensuse-leap-15.1-x86_64/result',
'root': 'opensuse-leap-15.1-x86_64'},
'chroot_scan_enable': False,
'chroot_scan_opts': {'basedir': '/var/lib/mock/opensuse-leap-15.1-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/opensuse-leap-15.1-x86_64',
'only_failed': True,
'regexes': ['^[^k]?core(\\.\\d+)?$', '\\.log$'],
'resultdir': '/var/lib/mock/opensuse-leap-15.1-x86_64/result',
'root': 'opensuse-leap-15.1-x86_64'},
'compress_logs_enable': False,
'compress_logs_opts': {'basedir': '/var/lib/mock/opensuse-leap-15.1-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/opensuse-leap-15.1-x86_64',
'command': 'gzip',
'resultdir': '/var/lib/mock/opensuse-leap-15.1-x86_64/result',
'root': 'opensuse-leap-15.1-x86_64'},
'hw_info_enable': True,
'hw_info_opts': {'basedir': '/var/lib/mock/opensuse-leap-15.1-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/opensuse-leap-15.1-x86_64',
'resultdir': '/var/lib/mock/opensuse-leap-15.1-x86_64/result',
'root': 'opensuse-leap-15.1-x86_64'},
'lvm_root_enable': False,
'lvm_root_opts': {'basedir': '/var/lib/mock/opensuse-leap-15.1-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/opensuse-leap-15.1-x86_64',
'pool_name': 'mockbuild',
'resultdir': '/var/lib/mock/opensuse-leap-15.1-x86_64/result',
'root': 'opensuse-leap-15.1-x86_64'},
'mount_enable': False,
'mount_opts': {'basedir': '/var/lib/mock/opensuse-leap-15.1-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/opensuse-leap-15.1-x86_64',
'resultdir': '/var/lib/mock/opensuse-leap-15.1-x86_64/result',
'root': 'opensuse-leap-15.1-x86_64'},
'overlayfs_enable': False,
'overlayfs_opts': {'basedir': '/var/lib/mock/opensuse-leap-15.1-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/opensuse-leap-15.1-x86_64',
'resultdir': '/var/lib/mock/opensuse-leap-15.1-x86_64/result',
'root': 'opensuse-leap-15.1-x86_64'},
'package_state_enable': True,
'package_state_opts': {'available_pkgs': False,
'basedir': '/var/lib/mock/opensuse-leap-15.1-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/opensuse-leap-15.1-x86_64',
'installed_pkgs': True,
'resultdir': '/var/lib/mock/opensuse-leap-15.1-x86_64/result',
'root': 'opensuse-leap-15.1-x86_64'},
'pm_request_enable': False,
'pm_request_opts': {'basedir': '/var/lib/mock/opensuse-leap-15.1-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/opensuse-leap-15.1-x86_64',
'resultdir': '/var/lib/mock/opensuse-leap-15.1-x86_64/result',
'root': 'opensuse-leap-15.1-x86_64'},
'procenv_enable': False,
'procenv_opts': {'basedir': '/var/lib/mock/opensuse-leap-15.1-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/opensuse-leap-15.1-x86_64',
'resultdir': '/var/lib/mock/opensuse-leap-15.1-x86_64/result',
'root': 'opensuse-leap-15.1-x86_64'},
'root_cache_enable': True,
'root_cache_opts': {'age_check': True,
'basedir': '/var/lib/mock/opensuse-leap-15.1-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/opensuse-leap-15.1-x86_64',
'compress_program': 'pigz',
'decompress_program': None,
'dir': '/var/cache/mock/opensuse-leap-15.1-x86_64/root_cache/',
'exclude_dirs': ['./proc',
'./sys',
'./dev',
'./tmp/ccache',
'./var/cache/yum',
'./var/cache/dnf',
'./var/log'],
'extension': '.gz',
'max_age_days': 15,
'resultdir': '/var/lib/mock/opensuse-leap-15.1-x86_64/result',
'root': 'opensuse-leap-15.1-x86_64',
'tar': 'gnutar'},
'selinux_enable': True,
'selinux_opts': {'basedir': '/var/lib/mock/opensuse-leap-15.1-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/opensuse-leap-15.1-x86_64',
'resultdir': '/var/lib/mock/opensuse-leap-15.1-x86_64/result',
'root': 'opensuse-leap-15.1-x86_64'},
'sign_enable': False,
'sign_opts': {'basedir': '/var/lib/mock/opensuse-leap-15.1-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/opensuse-leap-15.1-x86_64',
'cmd': 'rpmsign',
'opts': '--addsign ',
'resultdir': '/var/lib/mock/opensuse-leap-15.1-x86_64/result',
'root': 'opensuse-leap-15.1-x86_64'},
'tmpfs_enable': False,
'tmpfs_opts': {'basedir': '/var/lib/mock/opensuse-leap-15.1-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/opensuse-leap-15.1-x86_64',
'keep_mounted': False,
'max_fs_size': None,
'mode': '0755',
'required_ram_mb': 900,
'resultdir': '/var/lib/mock/opensuse-leap-15.1-x86_64/result',
'root': 'opensuse-leap-15.1-x86_64'},
'yum_cache_enable': True,
'yum_cache_opts': {'basedir': '/var/lib/mock/opensuse-leap-15.1-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/opensuse-leap-15.1-x86_64',
'max_age_days': 30,
'max_metadata_age_days': 30,
'online': True,
'resultdir': '/var/lib/mock/opensuse-leap-15.1-x86_64/result',
'root': 'opensuse-leap-15.1-x86_64'}}Finish: run
config_opts['print_main_output'] = False
config_opts['releasever'] = '15.1'
config_opts['root'] = 'opensuse-leap-15.1-x86_64'
config_opts['root_log_fmt_str'] = '%(levelname)s %(filename)s:%(lineno)d: %(message)s'
config_opts['rpmbuild_arch'] = 'x86_64'
config_opts['state_log_fmt_str'] = '%(asctime)s - %(message)s'
config_opts['target_arch'] = 'x86_64'
config_opts['use_nspawn'] = False
config_opts['useradd'] = ('/usr/sbin/useradd -o -m -u {{chrootuid}} -g {{chrootgid}} -d {{chroothome}} '
'{{chrootuser}}')
config_opts['verbose'] = 1Can you please retry with mock-core-configs 32.3?
That is effectively what I have. I have diffed my /etc/mock/opensuse-leap-15.1-x86_64.cfg with the mock-core-configs 32.3-2 opensuse-leap-15.1-x86_64.cfg and have confirmed the only difference is the:
config_opts['dnf.conf'] += """
[repo.dc.hpdd.intel.com_repository_daos-stack-leap-15-x86_64-stable-local_]
name=repo.dc.hpdd.intel.com_repository_daos-stack-leap-15-x86_64-stable-local_
baseurl=https://repo.dc.hpdd.intel.com/repository/daos-stack-leap-15-x86_64-stable-local/
enabled=1
[repo.dc.hpdd.intel.com_repository_daos-stack-external-leap-15-x86_64-stable-group_]
name=repo.dc.hpdd.intel.com_repository_daos-stack-external-leap-15-x86_64-stable-group_
baseurl=https://repo.dc.hpdd.intel.com/repository/daos-stack-external-leap-15-x86_64-stable-group/
enabled=1
"""I have at the end of mine to add the repos that DNF/Curl are having issues with.
brianjmurrell
commented
4 years ago Any more thoughts here? I've actually rolled my production builders back to mock < 2 but this needs to be resolved.
praiskup
commented
4 years ago Do you have the CA in trusted certificates on your host? We likely need to copy the certificate
bundle into bootstrap chroot (the stuff from /etc/pki/ca-trust/source/anchors gets compiled
into /etc/pki/tls/certs/ca-bundle.crt with updates-ca-trust).
brianjmurrell
commented
4 years ago Do you have the CA in trusted certificates on your host?
The server is signed by globally known CAs per the tests in the first comment.
We likely need to copy the certificate bundle into bootstrap chroot
But the dnf that is failing here, is running outside of the chroot with --installroot and is the dnf being used to build the initial chroot. Is that dnf really going to use the certificate bundle in the --installroot? It's not even there at this point is it?
Even if it does, surely the standard bundle in the chroot has the globally known CAs my repo server is signed with.
This theory can certainly be tested. I could, in the same way as I do in the original comment, use curl to the repo server but instead of using the default bundle, point curlat the bundle inside the chroot with--cacert or the environment variable namedCURL_CA_BUNDLE`.
I'm just not sure what file I should be pointing curl at to get the bundle inside the chroot.
praiskup
commented
4 years ago The dnf which is failing is calling dnf --installroot /var/lib/mock/opensuse-leap-15.1-x86_64/root,
it means that it is dnf from bootstrap chroot.
Btw., I was wrong with the ca-bundle; /etc/pki/tls/certs/ca-bundle.crt is just symlink to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem - and that is already copied into bootstrap. There must
be some other problem.
Can you please try curl -v ..../repomd.xml >/dev/null from within bootstrap chroot, like:
sudo chroot /var/lib/mock/opensuse-leap-15.1-x86_64-bootstrap/root?
praiskup
commented
4 years ago FWIW, it is hard to reproduce for me because I can not reach the hostname from my box: Could not resolve host: repo.dc.hpdd.intel.com
brianjmurrell
commented
4 years ago Could not resolve host: repo.dc.hpdd.intel.com
Yes, that is available to our own internal network only.
praiskup
commented
4 years ago I got an idea ... is curl in opensuse bootstrap chroot patched (as in fedora) so it accepts the bundle?
Conan-Kudo
commented
4 years ago @praiskup It accepts a ca bundle like Fedora does, but the path used in /etc/ssl/certs instead of /etc/pki/tls/certs.
brianjmurrell
commented
4 years ago But why is using the bootstrap for opensuse all of a sudden a problem? It was being used in core-configs-31.7-1 (comparison) wasn't it?
praiskup
commented
4 years ago We turned use_bootstrap=True in mock.rpm, not mock-core-configs.rpm.
praiskup
commented
4 years ago @Conan-Kudo can you more precisely define what files need to be copied where?
Conan-Kudo
commented
4 years ago Hmm, actually, according to openSUSE:Factory/openssl, the cert bundle needs to be installed as /var/lib/ca-certificates/ca-bundle.pem.
Is there anything else you need?
praiskup
commented
4 years ago Not yet ... well, we could have bootstra_ca_bundle_path = None set by
default, and for OpenSUSE chroots we would set
bootstra_ca_bundle_path = '/var/lib/ca-certificates/ca-bundle.pem';
when set mock would copy the ca-bundle.crt contents into the desired
path.
The question is whether Fedora's /etc/pki/tls/certs/ca-bundle.crt is compatible
with /var/lib/ca-certificates/ca-bundle.pem.
praiskup
commented
4 years ago I tested with podman, and it should work.
praiskup
commented
4 years ago The other question is whether it is enough; i.e. @brianjmurrell, is your CA in the crt bundle, actually?
brianjmurrell
commented
4 years ago The other question is whether it is enough; i.e. @brianjmurrell, is your CA in the crt bundle, actually?
Yes, per the original problem description, both openssl s_client and curl, when run on host file system, verify the web server just fine.
Just so I am clear here, the bootstrap dnf (that runs outside of the chroot, correct?) with --installroot to create the initial chroot, is getting it's CA certificate bundle from inside the chroot, and is not using the CA certificate bundle from the host file system, outside of the chroot?
Conan-Kudo
commented
4 years ago @brianjmurrell bootstrap uses host certs, everything after will use the bootstrap root certs.
brianjmurrell
commented
4 years ago @Conan-Kudo As I would expect.
Is this dnf command the bootstrap then?
15:57:52 2020-02-28T15:55:42Z DDEBUG Command: dnf --installroot /var/lib/mock/opensuse-leap-15.1-x86_64/root/ --releasever 15.1 --setopt=deltarpm=False --allowerasing --disableplugin=local --disableplugin=spacewalk install patterns-devel-base-devel_rpm_build But maybe more to the point, would that dnf command be using the chroot cert bundle or the the host's cert bundle?
Conan-Kudo
commented
4 years ago @brianjmurrell That is post-bootstrap. Mock actually classifies the commands in its logs which ones are bootstrap.
praiskup
commented
4 years ago Is this dnf command the bootstrap then? ... dnf --installroot /var/lib/mock/opensuse-leap-15.1-x86_64/root ...
Since you install into normal root (not to opensuse-leap-15.1-x86_64-bootstrap),
it means that you install by dnf from bootstrap into normal chroot.
brianjmurrell
commented
4 years ago
sudo chroot /var/lib/mock/opensuse-leap-15.1-x86_64-bootstrap/root?
chroot: failed to run command 'curl': No such file or directoryTo make things easier to test, you can try dnf copr enable @mock/mock. It should just work.
If you want to have curl availalble in the bootstrap chroot, put it into dnf_install_command option.
Short description of the problem
dnfis unable to verify certificate chain when creating installroot.Output of
rpm -q mockmock-2.0-2.fc31.noarch
Steps to reproduce issue
mock -r opensuse-leap-15.1-x86_64 --define "%relval .606.g456cf07" example.src.rpmAny additional notes
Output of
mock --debug-configHere is what is in
/var/lib/mock/opensuse-leap-15.1-x86_64/root/var/log/dnf.logafter this happens. Notice thePeer certificate cannot be authenticatederrors and resulting repos being ignored.curlandopenssl s_clientboth have no issues with the cert chain:dnf --installrootis even successful in the same root after this fails:So what's the problem with
mock's use ofdnf --installrootthat it has certificate problems?