search

QubesOS / qubes-issues

The Qubes OS Project issue tracker
https://www.qubes-os.org/doc/issue-tracking/
541 stars 48 forks source link

Can't update TemplateVMs, errors in UpdateVM #5187

Open 1dnrr opened 5 years ago

1dnrr commented 5 years ago

Qubes OS version R4.0

Affected component(s) or functionality Updates proxy

Brief summary No connection between TemplateVMs and UpdateVM. Unable to update TemplateVMs.

To Reproduce [0. Upgrade TemplateVMs]

  1. Install new qubes-template-debian-9 (or fedora-30, ...), qubes-template-whonix-gw-15.
  2. Create sys-whonix AppVM and Debian-9 AppVM (based on just installed qubes-template-whonix-gw-15 and qubes-template-debian-9).
  3. Setup UpdateVM: qubes-prefs updatevm sys-whonix
  4. Run in Debian-9: sudo apt-get update

Expected behavior A distributive repository cache synchronization.

Actual behavior In Debian-9 TemplateVM: sudo apt-get update causes an error 500 Unable to connect. In Whonix-gw TemplateVM sudo apt-get update causes: Err:4 tor+https://deb.debian.org/debian buster InRelease Invalid response from proxy: HTTP/1.0 500 Unable to connect Server: tinyproxy/1.10.0 Content-Type: text/html Connection: close [IP: 127.0.0.1 8082]

Additional context I still can update dom0. And can run sudo apt-get update inside UpdateVM (sys-whonix AppVM) without the connection error.

When I run sudo apt-get update in AppVMs or TemplateVMs, my UpdateVM does not start automatically.

I see errors inside UpdateVM qubes-updates-proxy.service:

user@host:~$ sudo systemctl status qubes-updates-proxy.service 
● qubes-updates-proxy.service - Qubes updates proxy (tinyproxy)
   Loaded: loaded (/lib/systemd/system/qubes-updates-proxy.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/qubes-updates-proxy.service.d
           └─40_qubes-whonix.conf
   Active: active (running) since Sat 2019-07-20 04:18:57 UTC; 15min ago
  Process: 2577 ExecStartPre=/usr/bin/install -d --owner tinyproxy --group tinyproxy /var/run/tinyproxy (code=exited, status=0/SUCCESS)
 Main PID: 2578 (tinyproxy)
    Tasks: 3 (limit: 4667)
   Memory: 3.9M
   CGroup: /system.slice/qubes-updates-proxy.service
           ├─2578 /usr/bin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
           ├─2580 /usr/bin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
           └─2581 /usr/bin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf

Jul 20 04:33:53 host tinyproxy[2580]: opensock: Could not retrieve info for deb.debian.org
Jul 20 04:33:53 host tinyproxy[2580]: opensock: Could not retrieve info for deb.whonix.org
Jul 20 04:33:53 host tinyproxy[2581]: opensock: Could not retrieve info for deb.qubes-os.org
Jul 20 04:33:53 host tinyproxy[2581]: opensock: Could not retrieve info for sdscoq7snqtznauu.onion
Jul 20 04:33:53 host tinyproxy[2580]: opensock: Could not retrieve info for deb.debian.org
Jul 20 04:33:53 host tinyproxy[2581]: opensock: Could not retrieve info for deb.whonix.org
Jul 20 04:33:53 host tinyproxy[2580]: opensock: Could not retrieve info for deb.debian.org
Jul 20 04:33:53 host tinyproxy[2581]: opensock: Could not retrieve info for deb.debian.org
Jul 20 04:33:53 host tinyproxy[2581]: opensock: Could not retrieve info for deb.debian.org
Jul 20 04:33:54 host tinyproxy[2581]: opensock: Could not retrieve info for deb.debian.org

Solutions you've tried

  1. I tried to restart machines.

  2. to change UpdateVM to non whonix-gw-15 one.

  3. to use proxy in Debian-9 AppVM with wget:

    user@host:~$ export http_proxy=127.0.0.1:8082
user@host:~$ wget ipecho.net/plain
--2019-07-21 11:31:28--  http://ipecho.net/plain
Connecting to 127.0.0.1:8082... connected.
Proxy request sent, awaiting response... 500 Unable to connect
2019-07-21 11:31:28 ERROR 500: Unable to connect.

  4. I tried to check proxy config file (I did not change the file):

    
user@dom0 ~ % sudo qubesctl state.sls qvm.updates-via-whonix
local:                  
----------
      ID: default-update-policy-whonix
Function: file.prepend
    Name: /etc/qubes-rpc/policy/qubes.UpdatesProxy
  Result: True
 Comment: File /etc/qubes-rpc/policy/qubes.UpdatesProxy is in correct state
 Started: 08:10:33.373253
Duration: 5.195 ms
 Changes:

Summary for local

Succeeded: 1 Failed: 0

Total states run: 1 Total run time: 5.195 ms

andrewdavidwong commented 5 years ago

Could you provide the output of cat /etc/qubes-rpc/policy/qubes.UpdatesProxy in dom0?

1dnrr commented 5 years ago

Yes of course.

user@dom0 ~ % cat /etc/qubes-rpc/policy/qubes.UpdatesProxy       
$type:TemplateVM $default allow,target=sys-whonix
$tag:whonix-updatevm $default allow,target=sys-whonix
$tag:whonix-updatevm $anyvm deny
## Note that policy parsing stops at the first match,
## so adding anything below "$anyvm $anyvm action" line will have no effect

## Please use a single # to start your custom comments

# Default rule for all TemplateVMs - direct the connection to sys-net
$type:TemplateVM $default allow,target=sys-net

$anyvm $anyvm deny
andrewdavidwong commented 5 years ago

Thanks, the content of qubes.UpdatesProxy looks normal to me.

One thing to note: I believe the UpdateVM is only for dom0, not TemplateVMs.

Do Fedora TemplateVMs exhibit the same problems as Debian and Whonix?

1dnrr commented 5 years ago

Yes. Fedora (just installed and my old one template) exhibit the same problem.

[user@fedora-30 ~]$ sudo dnf update
Fedora Modular 30 - x86_64                                                                                                                                                                                                                                                                                                          0.0  B/s |   0  B     00:00    
Failed to synchronize cache for repo 'fedora-modular'
Error: Failed to synchronize cache for repo 'fedora-modular'
andrewdavidwong commented 5 years ago

If you comment out the first line of /etc/qubes-rpc/policy/qubes.UpdatesProxy, can you update TemplateVMs via sys-net?

(Note: Only try this if you're willing to update TemplateVMs over clearnet. For most people, this should be fine, but if you have especially stringent privacy needs, it might not be.)

1dnrr commented 5 years ago

(Fortunately, I have a VPN profile in network manager at sys-net.)

I try to comment the first line, and run update with sys-net as updateVM, and the problem persists.

andrewdavidwong commented 5 years ago

I try to comment the first line, and run update with sys-net as updateVM, and the problem persists.

Again, I'm pretty sure the UpdateVM is just for dom0, not TemplateVMs. For example, if $type:TemplateVM $default allow,target=sys-net is the first applicable rule in /etc/qubes-rpc/policy/qubes.UpdatesProxy, then TemplateVMs should use sys-net for updates, even if the UpdateVM is different (e.g., sys-firewall).

In any case, I'm out of troubleshooting ideas, so I'll leave it this one to the other devs.

hexagonrecursion commented 5 years ago

From man qubes-prefs (emphasis mine):

updatevm Qube used to download dom0 updates

1dnrr, You posted that you got this error in sys-whonix:

Jul 20 04:33:53 host tinyproxy[2580]: opensock: Could not retrieve info for deb.debian.org

It seams to imply that

  1. the updates-proxy-setup service works fine in the template
  2. the policy for qubes.UpdatesProxy routs the rpc call to the correct qube
  3. the qubes-updates-proxy service in sys-whonix is active
  4. tinyproxy in sys-whonix is unable to connect to dns

I think that apt-get update succeeding in sys-whonix, rules out issues like not having an internet connection at all as well as tor-related issues.

apt-get manages to resolve domain names just fine, but tinyproxy in the same qube does not. Very strange.

hexagonrecursion commented 5 years ago

Note: updating through tor appears to work fine on my machine.

1dnrr commented 5 years ago

With the help of multiple reinstallations and settings changes, I fixed the updates in whonix-ws-15 and whonix-gw-15 TemplateVMs. But updates in debian-9 and fedora-30 are continue continue to be faulty.

It seems that the problem is in an assignment of network addresses of the machines: the ProxyVM works fine, but the TemplateVMs (fedora and debian) are addressed to a different address.

There are no errors in sudo systemctl status qubes-updates-proxy.service output in ProxyVM (sys-whonix) now.

OklahomaSpider commented 4 years ago

Does anyone know if this was resolved? I just installed today and am having the same issues... I am unable to update fedora-30 or debian-9. I will be trying to update the whonix VMs separately based on the above.

andrewdavidwong commented 4 years ago

Does anyone know if this was resolved? I just installed today and am having the same issues... I am unable to update fedora-30 or debian-9. I will be trying to update the whonix VMs separately based on the above.

I'm not aware of any resolution, but I also haven't seen any other recent reports of this problem.

Could you provide a bit more detail about how exactly you're trying to update them and the behavior you're observing?

OklahomaSpider commented 4 years ago

This is a fresh installation of 4.0.3 on a Dell Optiplex 990 SFF, using stock Qubes configurations/templates. I have not changed any settings from the stock installation standard. I am not using a proxy external to the machine.

I have attempted using the QubesUpdate tool as well as starting a terminal within the template VM to run "sudo apt-get update" or "sudo dnf update" for debian or fedora respectively. Principle symptom using the QubesUpdate tool is that it will sit at the first selected template VM and nothing will occur - no progress is achieved. I have let the system sit for >12hrs with no noticable increase in processor, hard drive or network utilization and no progress completed on update. Running the commands from the terminal I get the same "Error 500" referenced above.

I have not dug too deeply into the status of the proxy service or its logs (yet).

On Tue, Apr 21, 2020 at 5:29 AM Andrew David Wong notifications@github.com wrote:

Does anyone know if this was resolved? I just installed today and am having the same issues... I am unable to update fedora-30 or debian-9. I will be trying to update the whonix VMs separately based on the above.

I'm not aware of any resolution, but I also haven't seen any other recent reports of this problem.

Could you provide a bit more detail about how exactly you're trying to update them and the behavior you're observing?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/QubesOS/qubes-issues/issues/5187#issuecomment-617065414, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANVNSGGIOKHOHJPHL36D2NTRNVRN3ANCNFSM4IFYETKA .

OklahomaSpider commented 4 years ago

Go figure. I booted back up this evening, and updating via console commands seems to be working for Debian, Fedora, and both iterations of Whonix. Will try QubesUpdate again once I'm patched to the current baseline.

iuriguilherme commented 4 years ago

Steps to reproduce:

Install a debian buster template from offical repositories:

[user@dom0 ~]$ sudo qubes-dom0-update qubes-template-debian-10

The debian-10 TemplateVM won't update. AppVMs based on it are working fine.

user@debian-10:~$ sudo apt update
Err:1 https://deb.qubes-os.org/r4.0/vm buster InRelease                                                                                                                     
  Invalid response from proxy: /usr/lib/qubes/qubes-rpc-multiplexer: 14: /etc/profile.d/20_power_savings_disable_in_vms.sh: shopt: not found HTTP/1.0 200 Connection established  Proxy-agent: tinyproxy/1.8.3     [IP: 127.0.0.1 8082]
Err:2 https://deb.debian.org/debian buster InRelease                                                                                                                        
  Invalid response from proxy: /usr/lib/qubes/qubes-rpc-multiplexer: 14: /etc/profile.d/20_power_savings_disable_in_vms.sh: shopt: not found HTTP/1.0 200 Connection established  Proxy-agent: tinyproxy/1.8.3     [IP: 127.0.0.1 8082]
Err:3 https://deb.debian.org/debian-security buster/updates InRelease
  Invalid response from proxy: /usr/lib/qubes/qubes-rpc-multiplexer: 14: /etc/profile.d/20_power_savings_disable_in_vms.sh: shopt: not found HTTP/1.0 200 Connection established  Proxy-agent: tinyproxy/1.8.3     [IP: 127.0.0.1 8082]
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
W: Failed to fetch https://deb.debian.org/debian/dists/buster/InRelease  Invalid response from proxy: /usr/lib/qubes/qubes-rpc-multiplexer: 14: /etc/profile.d/20_power_savings_disable_in_vms.sh: shopt: not found HTTP/1.0 200 Connection established  Proxy-agent: tinyproxy/1.8.3     [IP: 127.0.0.1 8082]                                           
W: Failed to fetch https://deb.debian.org/debian-security/dists/buster/updates/InRelease  Invalid response from proxy: /usr/lib/qubes/qubes-rpc-multiplexer: 14: /etc/profile.d/20_power_savings_disable_in_vms.sh: shopt: not found HTTP/1.0 200 Connection established  Proxy-agent: tinyproxy/1.8.3     [IP: 127.0.0.1 8082]                          
W: Failed to fetch https://deb.qubes-os.org/r4.0/vm/dists/buster/InRelease  Invalid response from proxy: /usr/lib/qubes/qubes-rpc-multiplexer: 14: /etc/profile.d/20_power_savings_disable_in_vms.sh: shopt: not found HTTP/1.0 200 Connection established  Proxy-agent: tinyproxy/1.8.3     [IP: 127.0.0.1 8082]                                        
W: Some index files failed to download. They have been ignored, or old ones used instead.

sys-net and sys-firewall use fedora-32 TemplateVM.

iuriguilherme commented 4 years ago

I have commented out the following line from /etc/qubes-rpc/policy/qubes.UpdateProxy at dom0:

$type:TemplateVM $default allow,target=sys-whonix

It seems that my Whonix is broken. Will fix it.

adrelanos commented 3 years ago

To debug, try these commands in sys-whonix. Then post the output here.

Just to run a connectivity test (doesn't wait for time synchronization to be done):

whonixcheck --verbose --leak-tests --function check_tor_socks_port

Verbose run of whonixcheck including leak tests (just as a connectivity test. I don't think here is any leak issue.):

whonixcheck --verbose --leak-tests

Running APT inside sys-whonix. If even that fails, it's a "lower level" issue, connectivity issue, no need to look into tinyproxy.

sudo apt update

Today I experienced this issue:

W: Failed to fetch tor+https://deb.debian.org/debian-security/dists/buster/updates/InRelease Invalid response from proxy: HTTP/1.0 500 Unable to connect Server: tinyproxy/1.10.0 Content-Type: text/html Connection: close [IP: 127.0.0.1 8082]

opensock: Could not retrieve info for deb.debian.org

I don't know if it was a similar issue as in this Qubes issue.

Reason was an outdated Tor consensus due to the current DoS issues of the Tor network. References:

As a workaround I stopped Tor, deleted Tor consensus and restarted Tor.

sudo systemctl stop tor@default
sudo su
cd /var/lib/tor
rm /var/lib/tor/cached-*
sudo systemctl restart tor@default

That resolved the issue for me for now.

(I might add a script that users could manually run to Whonix to simplify this process as a usability feature.)

Do you know other Qubes tickets showing HTTP/1.0 500 Unable to connect Server: tinyproxy or systemd journal log containing opensock: Could not retrieve info for deb.debian.org? Please post links to these tickets here.

Did you ever have this issue without any of Whonix being involved? If so, please check if there is an existing Qubes issue ticket for this and if not create one. Mention that Whonix isn't involved and share the link here so I can follow there too.

iuriguilherme commented 3 years ago

Almost everyone here knows, but for reference, the updated list of debian onion services:

https://onion.debian.org/

blobless commented 3 years ago

I have experienced this behavior on Qubes Release 4.0.4-rc1 and on 4.1.

Looks like I'm not alone: https://github.com/QubesOS/qubes-issues/issues/4977 https://github.com/QubesOS/qubes-issues/issues/6283

adrelanos commented 3 years ago

I have experienced this behavior on Qubes Release 4.0.4-rc1 and on 4.1.

Please provide debug information as requested in https://github.com/QubesOS/qubes-issues/issues/5187#issuecomment-762190926.

Looks like I'm not alone: #4977 #6283

Thanks for pointing me to these tickets. Replied there just now.

interact0r commented 2 years ago

Hi all, I'm experiencing this particular issue as well. Dom0, Whonix ws and gw updates successfully, however both fedora and debian template shows the following error:

Debian-10: Failed to fetch https://deb.debian.org/debian/dists/buster/InRelease Invalid response from proxy: HTTP/1.0 500 Unable to connect Server: tinyproxy/1.10.0 Content-Type: text/html Connection: close [IP: 127.0.0.1 8082]

Fedora-33: Failed to download metadata for repo 'updates': Cannot prepare internal mirrorlist: Curl error (56): Failure when receiving data from the peer for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f33&arch=x86_64 [Received HTTP code 500 from proxy after CONNECT]

The only change I did was the Qubes is now connected to internet via my router. Previously connection to internet was through usb modem. Could it be that the router is somehow blocking the templateVM update? How can I troubleshoot the tinyproxy, if this has something to do with it at all?

Note 0: Any VM created based on fedora-33 and debian-10 connects to the internet just fine. Note 1: everything was working fine previously (before using the router). Note 2: content of qubes.UpdatesProxy


$tag:whonix-updatevm $default allow,target=sys-whonix
$tag:whonix-updatevm $anyvm deny

$type:TemplateVM $default allow,target=sys-net

$anyvm $anyvm deny

Thanks

adrelanos commented 2 years ago

Still happening in Qubes R4.1?

andrewdavidwong commented 2 years ago

I'm also seeing the "invalid response from proxy" lines in some of my update logs, but it seems to be an intermittent or temporary problem. Doesn't happen every time for me.

adrelanos commented 1 year ago

I'm also seeing the "invalid response from proxy" lines in some of my update logs, but it seems to be an intermittent or temporary problem. Doesn't happen every time for me.

This seems to be temporary Tor network issues which resolve themselves without user action.

What other users reported here however are permanent update issues. Are these still reproducible?