search

Linutronix / elbe

Embedded Linux Build Environment
https://elbe-rfs.org
GNU General Public License v3.0
163 stars 59 forks source link

pbuilder and https connection #350

Closed debdanfli closed 1 year ago

debdanfli commented 1 year ago

Hi Team, i’m tried to add the microsoft debian mirror as a second mirror to the elbe pbuilder xml configuration.

https://packages.microsoft.com/debian/11/prod bullseye main

I wanna use the elbe-pbuilder for debianize a project with some dotnet build-depends. After some tries to add the public-key to the pbuilder, I got this:

The pbuilder --create cmd had some Warnings that the ca-certificates pkg could help.
` "CMD] pbuilder --create --configfile "/var/cache/elbe/741c8dbc-7b65-4f2d-b419-67810ff1938d/pbuilderrc" --aptconfdir "/var/cache/elbe/741c8dbc-7b65-4f2d-b419-67810ff1938d/aptconfdir" --debootstrapopts --include="git,gnupg" --debootstrapopts --no-check-gpg" `

`W: No local /etc/mailname to copy, relying on /var/cache/pbuilder/build/559/etc/mailname to be correct
I: Installing apt-lines
I: Copy  /var/cache/elbe/741c8dbc-7b65-4f2d-b419-67810ff1938d/aptconfdir/apt.conf.d  to chroot
I: user script /var/cache/pbuilder/build/559/tmp/hooks/G10elbe_apt_sources starting
W: --force-yes is deprecated, use one of the options starting with --allow instead.
W: --force-yes is deprecated, use one of the options starting with --allow instead.
W: --force-yes is deprecated, use one of the options starting with --allow instead.
W: --force-yes is deprecated, use one of the options starting with --allow instead.
W: --force-yes is deprecated, use one of the options starting with --allow instead.
W: --force-yes is deprecated, use one of the options starting with --allow instead.
W: --force-yes is deprecated, use one of the options starting with --allow instead.
E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation
Get:2 http://127.0.0.1:8080/var/cache/elbe/741c8dbc-7b65-4f2d-b419-67810ff1938d/repo bullseye InRelease [2647 B]
Ign:2 http://127.0.0.1:8080/var/cache/elbe/741c8dbc-7b65-4f2d-b419-67810ff1938d/repo bullseye InRelease
Err:5 https://packages.microsoft.com/debian/11/prod bullseye InRelease
  Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error in the certificate verification. [IP: 13.80.99.124 443]
Fetched 6448 B in 5s (1266 B/s)
Reading package lists...
W: --force-yes is deprecated, use one of the options starting with --allow instead.
W: GPG error: http://127.0.0.1:8080/var/cache/elbe/741c8dbc-7b65-4f2d-b419-67810ff1938d/repo bullseye InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F809F0C5CCE4B11F
W: The repository 'http://127.0.0.1:8080/var/cache/elbe/741c8dbc-7b65-4f2d-b419-67810ff1938d/repo bullseye InRelease' is not signed.
W: https://packages.microsoft.com/debian/11/prod/dists/bullseye/InRelease: No system certificates available. Try installing ca-certificates.
W: Failed to fetch https://packages.microsoft.com/debian/11/prod/dists/bullseye/InRelease  Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error in the certificate verification. [IP: 13.80.99.124 443]
W: Some index files failed to download. They have been ignored, or old ones used instead.
I: user script /var/cache/pbuilder/build/559/tmp/hooks/G10elbe_apt_sources finished
I: Refreshing the base.tgz

Some special with the ms debian mirror is in the source.list: 

deb **[trusted=true]** https://packages.microsoft.com/debian/11/prod bullseye main
If the **[trusted=true]** sign in the line, than it failed with following:
E: Conflicting values set for option Trusted regarding source https://packages.microsoft.com/debian/11/prod/ bullseye
E: The list of sources could not be read.

So, than I tried to add the ca-certificates packages to the pbuilder chroot environment. After more tries to add this package with the debootstrap-variants element, it failed with the same error. It seems to me, that the package couldn't add to the chroot env. (But as package build-depends it works to install the ca-certificates ,but not with debootstrap).

Now, I'm using the http:// ms debian repo, that I got from the warnings (13.80.99.124).

It is possible to use the https connection to the debian ms repo with the elbe-pbuilder? If yes, please please explain to me. Much thanks.

bgermann commented 1 year ago

Please give the XML mirror section to show how you include the repo and keyring.

debdanfli commented 1 year ago

For the https, I tried with this mirror section: 

<mirror>
    <primary_host><meip>/mirror</primary_host>
    <primary_path>/debian</primary_path>
    <primary_proto>http</primary_proto>
    <url-list>
        <url>
            <binary>http://meip/lrrepo bullseye main</binary>
            <key>http://meip/lrrepo.key</key>
            <options>
                <option>trusted=yes</option>
            </options>
        </url>
        <url>
            <binary>https://packages.microsoft.com/debian/11/prod bullseye main</binary>
            <key>https://packages.microsoft.com/keys/microsoft.asc</key>
            <options>
                <option>trusted=yes</option>
            </options>
        </url>
    </url-list>
</mirror>

For http like this: 

<mirror>
        <url>
            <binary>http://13.80.99.124/debian/11/prod/ bullseye main</binary>
        </url>
...
bgermann commented 1 year ago

Just to be sure: Have you tried the following?

<target>
                <debootstrap>
                        <include>ca-certificates</include>
                </debootstrap>
</target>
bgermann commented 1 year ago

If that does not work, just stick with HTTP. You should use raw-key with the microsoft.asc content to have a trust anchor.

debdanfli commented 1 year ago

Some debugging history:

        <debootstrap>
            <include>ca-certificates</include>
        </debootstrap>

The debootstrap xml section seems to me useless, because the "pbuilder --create " commad: 

pbuilder --create --configfile "/var/cache/elbe/2c4ed163-960a-494a-9c42-e46ccd2072d0/pbuilderrc" --aptconfdir "/var/cache/elbe/2c4ed163-960a-494a-9c42-e46ccd2072d0/aptconfdir" --debootstrapopts --include="git,gnupg" --debootstrapopts --no-check-gpg

doesn't install the ca-certificates in the pbuilder build chroot env. The is no package ca-certificates in the pbuilder/aptcache/:

root@elbe-daemon:/var/cache/elbe/2c4ed163-960a-494a-9c42-e46ccd2072d0/pbuilder/aptcache# ls -l | grep "ca-*"
-rw-r--r-- 1 root root   445628 Aug 18 09:42 ccache_4.2-1_amd64.deb
-rw-r--r-- 1 root root   472168 Aug 18 09:37 libboost-locale1.74.0_1.74.0-9_amd64.deb
-rw-r--r-- 1 root root   572844 Aug 18 09:37 libboost-locale1.74-dev_1.74.0-9_amd64.deb
-rw-r--r-- 1 root root     4660 Aug 18 09:37 libboost-locale-dev_1.74.0.3_amd64.deb
-rw-r--r-- 1 root root    14764 Aug 18 12:01 libcap-ng0_0.7.9-2.2+b1_amd64.deb

There is some other special with the ms-mirror in the hooks.d/G10elbe_apt_source file the dobbel trusted=yes: 

root@elbe-daemon:/var/cache/elbe/2c4ed163-960a-494a-9c42-e46ccd2072d0/pbuilder/hooks.d# cat G10elbe_apt_sources 
#!/bin/sh
cat -> /etc/apt/sources.list <<EOF
deb http://127.0.0.1:8080/var/cache/elbe/2c4ed163-960a-494a-9c42-e46ccd2072d0/repo bullseye main
deb [trusted=yes arch=amd64] http://meip/mirror/debian bullseye main
deb [trusted=yes] http://meip/lrrepo bullseye main
deb [trusted=yes trusted=yes] https://packages.microsoft.com/debian/11/prod bullseye main
EOF
cat << EOF | apt-key add -
-----BEGIN PGP PUBLIC KEY BLOCK-----

Without the option "trusted=yes" in the xml section there is only one sign.

After the try with the raw key, there is still the same problem: 

.
.
.
All packages are up to date.
W: --force-yes is deprecated, use one of the options starting with --allow instead.
W: GPG error: http://172.16.40.5/lrrepo bullseye InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 24EBADBC001DD4B0
W: GPG error: http://127.0.0.1:8080/var/cache/elbe/c1a640cd-7af9-45ff-8e8f-95497dd64efc/repo bullseye InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY C1754212B98094E7
W: The repository 'http://127.0.0.1:8080/var/cache/elbe/c1a640cd-7af9-45ff-8e8f-95497dd64efc/repo bullseye InRelease' is not signed.
W: https://packages.microsoft.com/debian/11/prod/dists/bullseye/InRelease: No system certificates available. Try installing ca-certificates.
W: Failed to fetch https://packages.microsoft.com/debian/11/prod/dists/bullseye/InRelease  Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error in the certificate verification. [IP: 104.214.230.139 443]
W: Some index files failed to download. They have been ignored, or old ones used instead.
I: user script /var/cache/pbuilder/build/13370/tmp/hooks/H10elbe_apt_update finished
I: Obtaining the cached apt archive contents
Reading package lists...
.
.
.
Setting up pbuilder (0.231) ...
Processing triggers for libc-bin (2.31-13+deb11u4) ...
W: --force-yes is deprecated, use one of the options starting with --allow instead.
I: Setting DEBBUILDOPTS=
I: Setting DEBBUILDOPTS= -jauto -sa
 -> Attempting to parse the build-deps
 -> Considering build-depgit
   -> Trying to add git
 -> Considering build-dep cmake
   -> Trying to add cmake
 -> Considering build-dep ninja-build
   -> Trying to add ninja-build
 -> Considering build-dep ccache
   -> Trying to add ccache
 -> Considering build-dep unzip
   -> Trying to add unzip
 -> Considering build-dep rsync
   -> Trying to add rsync
 -> Considering build-dep ca-certificates
   -> Trying to add ca-certificates
 -> Considering build-dep dotnet-sdk-6.0
   -> Trying to add dotnet-sdk-6.0
   -> Loop detected, last APT error was: ======
Reading package lists...
Building dependency tree...
Reading state information...
W: --force-yes is deprecated, use one of the options starting with --allow instead.
E: Unable to locate package dotnet-sdk-6.0
E: Couldn't find any package by glob 'dotnet-sdk-6.0'
E: Couldn't find any package by regex 'dotnet-sdk-6.0'
   -> =========================================
   -> (not adding  to dotnet-sdk-6.0)
       -> Cannot install dotnet-sdk-6.0; apt errors follow:
Reading package lists...
Building dependency tree...
Reading state information...
W: --force-yes is deprecated, use one of the options starting with --allow instead.
E: Unable to locate package dotnet-sdk-6.0
E: Couldn't find any package by glob 'dotnet-sdk-6.0'
E: Couldn't find any package by regex 'dotnet-sdk-6.0'
W: --force-yes is deprecated, use one of the options starting with --allow instead.
E: Could not satisfy build-dependency.
I: Copying back the cached apt archive contents
I: unmounting /var/cache/elbe/c1a640cd-7af9-45ff-8e8f-95497dd64efc/pdebuilder filesystem
I: unmounting /var/cache/elbe/c1a640cd-7af9-45ff-8e8f-95497dd64efc/ccache filesystem
I: unmounting dev/ptmx filesystem
I: unmounting dev/pts filesystem
I: unmounting dev/shm filesystem
I: unmounting proc filesystem
I: unmounting sys filesystem
I: cleaning the build env 
I: removing directory /var/cache/pbuilder/build/13370 and its subdirectories
[ERROR] Package fails to build.
Please make sure, that the submitted package builds in pbuilder
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/elbepack/elbeproject.py", line 804, in pdebuild_build
    do('cd "%s"; %s pdebuild --debbuildopts "-j%s -sa" '
  File "/usr/lib/python3/dist-packages/elbepack/shellhelper.py", line 258, in do
    raise CommandError(cmd, p.returncode)
elbepack.shellhelper.CommandError: Error: 1 returned from Command cd "/var/cache/elbe/c1a640cd-7af9-45ff-8e

Some other trying in the initvm: adding the following lines to the "pbuilder/hooks.d/H10elbe_apt_update" and the pdebuild-cmd in the initvm running successfully. 

apt install ca-certificates -y 
apt update
EOF
cat << EOF | apt-key add -
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.7 (GNU/Linux)

mQENBFYxWIwBCADAKoZhZlJxGNGWzqV+1OG1xiQeoowKhssGAKvd+buXCGISZJwT
LXZqIcIiLP7pqdcZWtE9bSc7yBY2MalDp9Liu0KekywQ6VVX1T72NPf5Ev6x6DLV
7aVWsCzUAF+eb7DC9fPuFLEdxmOEYoPjzrQ7cCnSV4JQxAqhU4T6OjbvRazGl3ag
OeizPXmRljMtUUttHQZnRhtlzkmwIrUivbfFPD+fEoHJ1+uIdfOzZX8/oKHKLe2j
H632kvsNzJFlROVvGLYAk2WRcLu+RjjggixhwiB+Mu/A8Tf4V6b+YppS44q8EvVr
M+QvY7LNSOffSO6Slsy9oisGTdfE39nC7pVRABEBAAG0N01pY3Jvc29mdCAoUmVs
ZWFzZSBzaWduaW5nKSA8Z3Bnc2VjdXJpdHlAbWljcm9zb2Z0LmNvbT6JATUEEwEC
AB8FAlYxWIwCGwMGCwkIBwMCBBUCCAMDFgIBAh4BAheAAAoJEOs+lK2+EinPGpsH
/32vKy29Hg51H9dfFJMx0/a/F+5vKeCeVqimvyTM04C+XENNuSbYZ3eRPHGHFLqe
MNGxsfb7C7ZxEeW7J/vSzRgHxm7ZvESisUYRFq2sgkJ+HFERNrqfci45bdhmrUsy
7SWw9ybxdFOkuQoyKD3tBmiGfONQMlBaOMWdAsic965rvJsd5zYaZZFI1UwTkFXV
KJt3bp3Ngn1vEYXwijGTa+FXz6GLHueJwF0I7ug34DgUkAFvAs8Hacr2DRYxL5RJ
XdNgj4Jd2/g6T9InmWT0hASljur+dJnzNiNCkbn9KbX7J/qK1IbR8y560yRmFsU+
NdCFTW7wY0Fb1fWJ+/KTsC4=
=J6gs
-----END PGP PUBLIC KEY BLOCK-----
EOF

I will use the http connection.

Thanks for helping.