search

QubesOS / qubes-issues

The Qubes OS Project issue tracker
https://www.qubes-os.org/doc/issue-tracking/
543 stars 48 forks source link

IPv6 enabled in sys-net results in debian-based TemplateVM update failure #9376

Open BetoHydroxyButyrate opened 4 months ago

BetoHydroxyButyrate commented 4 months ago

How to file a helpful issue

Qubes OS release

4.2.2

Brief summary

TemplateVMs, debian-based, fail to update when ipv6 is enabled in sys-net

Steps to reproduce

enable IPv6 in sys-net qubes-update-gui --log DEBUG --target debian-12-xcfe

Expected behavior

I expect it to succeed.

Actual behavior

It fails.

Updating debian-12-xfce
Refreshing package info
Refreshing packages.
Fail to refresh InRelease: https://deb.qubes-os.org/r4.2/vm bookworm InRelease from https://deb.qubes-os.org/r4.2/vm/dists/bookworm/InRelease
Fail to refresh InRelease: https://deb.debian.org/debian bookworm InRelease from https://deb.debian.org/debian/dists/bookworm/InRelease
Fail to refresh InRelease: https://deb.debian.org/debian-security bookworm-security InRelease from https://deb.debian.org/debian-security/dists/bookworm-security/InRelease
Fail to refresh InRelease: https://deb.qubes-os.org/r4.2/vm bookworm InRelease from https://deb.qubes-os.org/r4.2/vm/dists/bookworm/InRelease
Fail to refresh InRelease: https://deb.debian.org/debian bookworm InRelease from https://deb.debian.org/debian/dists/bookworm/InRelease
Fail to refresh InRelease: https://deb.qubes-os.org/r4.2/vm bookworm InRelease from https://deb.qubes-os.org/r4.2/vm/dists/bookworm/InRelease
Fail to refresh InRelease: https://deb.debian.org/debian-security bookworm-security InRelease from https://deb.debian.org/debian-security/dists/bookworm-security/InRelease
Fail to refresh InRelease: https://deb.qubes-os.org/r4.2/vm bookworm InRelease from https://deb.qubes-os.org/r4.2/vm/dists/bookworm/InRelease
Fail to refresh InRelease: https://deb.debian.org/debian bookworm InRelease from https://deb.debian.org/debian/dists/bookworm/InRelease
Fail to refresh InRelease: https://deb.debian.org/debian-security bookworm-security InRelease from https://deb.debian.org/debian-security/dists/bookworm-security/InRelease
Fail to refresh InRelease: https://deb.debian.org/debian bookworm InRelease from https://deb.debian.org/debian/dists/bookworm/InRelease
Refreshed.
Fail to refresh InRelease: https://deb.debian.org/debian-security bookworm-security InRelease from https://deb.debian.org/debian-security/dists/bookworm-security/InRelease
E:Failed to fetch https://deb.debian.org/debian/dists/bookworm/InRelease  Reading from proxy failed - select (115: Operation now in progress) [IP: 127.0.0.1 8082], E:Failed to fetch https://deb.debian.org/debian-security/dists/bookworm-security/InRelease  Reading from proxy failed - select (115: Operation now in progress) [IP: 127.0.0.1 8082], E:Failed to fetch https://deb.qubes-os.org/r4.2/vm/dists/bookworm/InRelease  Reading from proxy failed - select (115: Operation now in progress) [IP: 127.0.0.1 8082], E:Some index files failed to download. They have been ignored, or old ones used instead.

If I qvm-run -u root debian-12-xfce xterm and then manually update:

root@debian-12-xfce:~# apt update
0% [Connecting to HTTP proxy (http://127.0.0.1:8082)] [Connecting to HTTP proxy (http://127.0.0.1:8082)]
Ign:1 https://deb.qubes-os.org/r4.2/vm bookworm InRelease
Ign:2 https://deb.debian.org/debian bookworm InRelease
0% [Connecting to HTTP proxy (http://127.0.0.1:8082)] [Connecting to HTTP proxy (http://127.0.0.1:8082)]
Ign:3 https://deb.debian.org/debian-security bookworm-security InRelease
Ign:1 https://deb.qubes-os.org/r4.2/vm bookworm InRelease
0% [Connecting to HTTP proxy (http://127.0.0.1:8082)] [Connecting to HTTP proxy (http://127.0.0.1:8082)]
Ign:2 https://deb.debian.org/debian bookworm InRelease
Ign:1 https://deb.qubes-os.org/r4.2/vm bookworm InRelease
0% [Connecting to HTTP proxy (http://127.0.0.1:8082)] [Connecting to HTTP proxy (http://127.0.0.1:8082)]
Ign:3 https://deb.debian.org/debian-security bookworm-security InRelease
0% [Connecting to HTTP proxy (http://127.0.0.1:8082)] [Connecting to HTTP proxy (http://127.0.0.1:8082)]
Err:1 https://deb.qubes-os.org/r4.2/vm bookworm InRelease
  Reading from proxy failed - select (115: Operation now in progress) [IP: 127.0.0.1 8082]
Ign:2 https://deb.debian.org/debian bookworm InRelease
Ign:3 https://deb.debian.org/debian-security bookworm-security InRelease
Err:2 https://deb.debian.org/debian bookworm InRelease
  Reading from proxy failed - select (115: Operation now in progress) [IP: 127.0.0.1 8082]
Err:3 https://deb.debian.org/debian-security bookworm-security InRelease
  Reading from proxy failed - select (115: Operation now in progress) [IP: 127.0.0.1 8082]
Reading package lists... Done
E: Failed to fetch https://deb.debian.org/debian/dists/bookworm/InRelease  Reading from proxy failed - select (115: Operation now in progress) [IP: 127.0.0.1 8082]
E: Failed to fetch https://deb.debian.org/debian-security/dists/bookworm-security/InRelease  Reading from proxy failed - select (115: Operation now in progress) [IP: 127.0.0.1 8082]
E: Failed to fetch https://deb.qubes-os.org/r4.2/vm/dists/bookworm/InRelease  Reading from proxy failed - select (115: Operation now in progress) [IP: 127.0.0.1 8082]
E: Some index files failed to download. They have been ignored, or old ones used instead.
root@debian-12-xfce:~#

If I open a terminal on sys-net:

root@nuc13:/home/user# systemctl status qubes-updates-proxy
● qubes-updates-proxy.service - Qubes updates proxy (tinyproxy)
     Loaded: loaded (/lib/systemd/system/qubes-updates-proxy.service; enabled; preset: enabled)
     Active: active (running) since Thu 2024-07-25 13:20:56 AEST; 2s ago
   Main PID: 2177 (tinyproxy)
      Tasks: 3 (limit: 381)
     Memory: 1.2M
        CPU: 7ms
     CGroup: /system.slice/qubes-updates-proxy.service
             └─2177 /usr/bin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf

Jul 25 13:20:56 nuc13.wtf.com systemd[1]: Started qubes-updates-proxy.service - Qubes updates proxy (tinyproxy).
Jul 25 13:20:56 nuc13.wtf.com tinyproxy-wrapper[2177]: Found tinyproxy at /usr/bin/tinyproxy
Jul 25 13:20:56 nuc13.wtf.com tinyproxy[2177]: Initializing tinyproxy ...
Jul 25 13:20:56 nuc13.wtf.com tinyproxy[2177]: Reloading config file
Jul 25 13:20:56 nuc13.wtf.com tinyproxy[2177]: Reloading config file finished
root@nuc13:/home/user# vi /etc/tinyproxy/tinyproxy-updates.conf

If I attempt to interact directly with the proxy:

root@debian-12-xfce:~# time curl -v --proxy http://127.1:8082/ https://www.example.com/
*   Trying 127.0.0.1:8082...
* Connected to 127.0.0.1 (127.0.0.1) port 8082 (#0)
* allocate connect buffer
* Establish HTTP proxy tunnel to www.example.com:443
> CONNECT www.example.com:443 HTTP/1.1
> Host: www.example.com:443
> User-Agent: curl/7.88.1
> Proxy-Connection: Keep-Alive
>

It outputs the above promptly, and then hangs.

< HTTP/1.0 200 Connection established
< Proxy-agent: tinyproxy/1.11.1
<
* CONNECT phase completed
* CONNECT tunnel established, response 200
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=example.com
*  start date: Jun 11 02:44:21 2024 GMT
*  expire date: Sep  9 02:44:20 2024 GMT
*  subjectAltName: host "www.example.com" matched cert's "*.example.com"
*  issuer: C=US; O=Google Trust Services; CN=WE1
*  SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: www.example.com]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x59751e801460)
> GET / HTTP/2
> Host: www.example.com
> user-agent: curl/7.88.1
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 301
< date: Thu, 25 Jul 2024 03:31:16 GMT
< content-type: text/html
< content-length: 167
< location: https://example.com
< cache-control: max-age=3600
< expires: Thu, 25 Jul 2024 04:31:16 GMT
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=Msa1JxOFTYzmlkwVlgo%2BtRg%2BaRdWLA1e0HKVHTG02W%2FeMhEExiUSTabAg0g6BOBxWj%2Fnl2WKm%2BkfQZXjldLVqHVABKieKlVHBJ4S7sG4KvyC7m8WCZcSbnV%2FlV%2B95wiV"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 8a89137e2f1355b1-SYD
< alt-svc: h3=":443"; ma=86400
<
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>cloudflare</center>
</body>
</html>
* Connection #0 to host 127.0.0.1 left intact

real    2m5.478s
user    0m0.051s
sys     0m0.023s
root@debian-12-xfce:~#

No matter what I try, it connects to the proxy promptly, and then it takes about 3 minutes to complete. Could be the apt gives up before 3 minutes. I note that deb.debian.org returns some stuff which is actually a re-direct, but if I edit the file in /etc in the template to explicitly reference the fastly redirect, it still fails.

I just this morning accepted a dom0 update, rebooted, and then attempted to apply the rest of the pending updates.

apparatius commented 4 months ago

Try to change your template updates qube from sys-net to sys-firewall. There were similar issues presumably because of IPv6 in sys-net: https://forum.qubes-os.org/t/cant-get-templatevm-update-proxy-to-work-with-firewall-or-vpn-netvms/20268 https://forum.qubes-os.org/t/qubes-update-tool-fails/27423 https://forum.qubes-os.org/t/failed-to-download-metadata-repo-qubes-vm-r4-2-current/27592

BetoHydroxyButyrate commented 4 months ago

Try to change your template updates qube from sys-net to sys-firewall. There were similar issues presumably because of IPv6 in sys-net: https://forum.qubes-os.org/t/cant-get-templatevm-update-proxy-to-work-with-firewall-or-vpn-netvms/20268 https://forum.qubes-os.org/t/qubes-update-tool-fails/27423 https://forum.qubes-os.org/t/failed-to-download-metadata-repo-qubes-vm-r4-2-current/27592

I tried both sys-firewall and sys-whonix prior to opening the default, but I am still assuming it is a hot-swap. IE, I do not need to reboot the templateVM after making the change. I did see some differences in the failure modes depending on the update qubes selection, so felt comfortable enough with my assumption to not drill down.

I'll check out those links and see....

BetoHydroxyButyrate commented 4 months ago

https://forum.qubes-os.org/t/cant-get-templatevm-update-proxy-to-work-with-firewall-or-vpn-netvms/20268

I do have a WireGuard VPN service VM but it only spins up if I also start the appVM which requires it. Not running since reboot this morning.

https://forum.qubes-os.org/t/qubes-update-tool-fails/27423

I do have IPv6 networking enabled. Disabled it, and the update proceeds.

If there is not currently a specific bug report for this issue, perhaps this can be used to track it?

BetoHydroxyButyrate commented 4 months ago

@apparatius: thanks for that!

BetoHydroxyButyrate commented 4 months ago

sys-net was restarted after upgrade, and IPv6 came back, as the default appears to be automatic, which is actually what I wanted, but I know I will forget all about this in a day or so.... better if someone fixes it before my memory fades.