search

QubesOS / qubes-issues

The Qubes OS Project issue tracker
https://www.qubes-os.org/doc/issue-tracking/
541 stars 48 forks source link

Fedora templates do not update the Qubes' repos through Whonix update proxy (R4.0-RC5) : " Error: Failed to synchronize cache for repo 'qubes-vm-r4.0-current' " #3737

Closed n1m1 closed 6 years ago

n1m1 commented 6 years ago

Qubes OS version:

R4.0-rc5

Affected component(s):

Fedora templates.

Steps to reproduce the behavior:

In a Fedora template type: sudo dnf update

Qubes is configured in order to update the templates through sys-whonix-14. Below the conf file in Dom0' (/etc/qubes-rpc/policy/qubes.UpdatesProxy)


$type:TemplateVM $default allow,target=sys-whonix-14
$tag:whonix-updatevm $default allow,target=sys-whonix-14
$tag:whonix-updatevm $anyvm deny
## Note that policy parsing stops at the first match,
## so adding anything below "$anyvm $anyvm action" line will have no effect

## Please use a single # to start your custom comments

# Default rule for all TemplateVMs - direct the connection to sys-net
$type:TemplateVM $default allow,target=sys-net

$anyvm $anyvm deny

Expected behavior:

The Fedora template is updated

Actual behavior:

The Fedora repos properly work, but the Qubes' ones do not. Here is the output I get.

Fedora 26 - x86_64 - Updates                    561 kB/s |  21 MB     00:38    
Fedora 26 - x86_64                              916 kB/s |  53 MB     00:59    
Error: Failed to synchronize cache for repo 'qubes-vm-r4.0-current'

General notes:

● qubes-updates-proxy.service - Qubes updates proxy (tinyproxy)
   Loaded: loaded (/lib/systemd/system/qubes-updates-proxy.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/qubes-updates-proxy.service.d
           └─40_qubes-whonix.conf
   Active: active (running) since Fri 2018-03-23 14:37:20 UTC; 2h 10min ago
  Process: 887 ExecStartPre=/usr/bin/install -d --owner tinyproxy --group tinyproxy /var/run/tinyproxy (code=exited, status=0/SUCCESS)
 Main PID: 892 (tinyproxy)
    Tasks: 8 (limit: 4915)
   CGroup: /system.slice/qubes-updates-proxy.service
           ├─ 892 /usr/sbin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
           ├─ 898 /usr/sbin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
           ├─ 899 /usr/sbin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
           ├─3367 /usr/sbin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
           ├─3396 /usr/sbin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
           ├─3455 /usr/sbin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
           ├─8643 /usr/sbin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
           └─9233 /usr/sbin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf

Mar 23 14:36:56 host tinyproxy[892]: Waiting servers (0) is less than MinSpareServers (2). Creating new child.
Mar 23 14:37:01 host tinyproxy[892]: Waiting servers (0) is less than MinSpareServers (2). Creating new child.                                                                                                                                                
Mar 23 14:37:06 host tinyproxy[892]: Waiting servers (0) is less than MinSpareServers (2). Creating new child.                                                                                                                                                
Mar 23 14:38:05 host tinyproxy[898]: Proxying refused on filtered domain "127.0.0.1"                                                                                                                                                                          
Mar 23 16:39:03 host tinyproxy[892]: Waiting servers (1) is less than MinSpareServers (2). Creating new child.                                                                                                                                                
Mar 23 16:39:35 host tinyproxy[3455]: Error reading readable client_fd 10                                                                                                                                                                                     
Mar 23 16:39:35 host tinyproxy[3455]: Could not retrieve request entity                                                                                                                                                                                       
Mar 23 16:39:35 host tinyproxy[8643]: Error reading readable client_fd 10                                                                                                                                                                                     
Mar 23 16:39:35 host tinyproxy[8643]: Could not retrieve request entity                                                                                                                                                                                       
Mar 23 16:42:33 host tinyproxy[892]: Waiting servers (0) is less than MinSpareServers (2). Creating new child.

Related issues:

3557 and #3135 ?

mirrorway commented 6 years ago

Same issue here with jessie-based sys-whonix and fedora-26 current-testing.

$ sudo dnf update
Fedora 26 - x86_64 - Updates                    644 kB/s |  21 MB     00:33    
Error: Failed to synchronize cache for repo 'qubes-vm-r4.0-current-testing'

debian, dom0 updates not affected. fedora-26 updates properly if changed to use sys-net in /etc/qubes-rpc/policy/qubes.UpdatesProxy.

andrewdavidwong commented 6 years ago

Duplicate of https://github.com/QubesOS/qubes-issues/issues/1352#issuecomment-375499955

andrewdavidwong commented 6 years ago

This appears to be a duplicate of an existing issue. If you believe this is not really a duplicate, please leave a comment briefly explaining why. We'll be happy to take another look and, if appropriate, reopen this issue. Thank you.

marmarek commented 6 years ago

I'm not so sure about this being duplicate - fedora repositories do not use onion service by default.

n1m1 commented 6 years ago

Thanks @andrewdavidwong and @marmarek for your kind and prompt answers. I am not using onion services on Qubes 4.0: the problem is not related to them.

P.s: just a note, not related with this issue. I use onion services on a different laptop where I run Qubes 3.2 and, until few days ago, I've never ever experienced a problem with them.

marmarek commented 6 years ago

Does dnf update --refresh change anything?

n1m1 commented 6 years ago

Unfortunately it does not. 

[user@fedora-26 ~]$ sudo dnf update --refresh
Fedora 26 - x86_64 - Updates                    265 kB/s |  21 MB     01:20    
Error: Failed to synchronize cache for repo 'qubes-vm-r4.0-current'

I tried also with:


[user@fedora-26 ~]$ sudo dnf clean all
18 files removed
[user@fedora-26 ~]$ sudo dnf update
Fedora 26 - x86_64 - Updates                    172 kB/s |  21 MB     02:04    
Fedora 26 - x86_64                              351 kB/s |  53 MB     02:35    
Error: Failed to synchronize cache for repo 'qubes-vm-r4.0-current'

Some more information here: 

[user@fedora-26 ~]$ sudo dnf update -v
Loaded plugins: builddep, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, qubes-hooks, repoclosure, repograph, repomanage, reposync
DNF version: 2.7.5
cachedir: /var/cache/dnf
repo: using cache for: updates
updates: using metadata from Sat Mar 24 04:21:18 2018.
repo: using cache for: fedora
not found deltainfo for: Fedora 26 - x86_64
not found updateinfo for: Fedora 26 - x86_64
fedora: using metadata from Wed Jul  5 21:31:38 2017.
Cannot download 'http://yum.qubes-os.org/r4.0/current/vm/fc26': repomd.xml parser error: Parse error at line: 1 (not well-formed (invalid token)).
Error: Failed to synchronize cache for repo 'qubes-vm-r4.0-current'

May be it is not necessary, but let me emphasize this one more time: this issue occurs only with the Qubes' repos on Fedora templates when updating through sys-whonix. With Qubes' repos on Debian-based templates (8,9.Whonix 13 and 14) I do not have this problem.

marmarek commented 6 years ago

Can you access that http://yum.qubes-os.org/r4.0/current/vm/fc26/repodata/repomd.xml from sys-whonix (using wget or curl)?

n1m1 commented 6 years ago

Yes, I can.

user@host:~$ wget http://yum.qubes-os.org/r4.0/current/vm/fc26/repodata/repomd.xml
--2018-03-24 14:47:49--  http://yum.qubes-os.org/r4.0/current/vm/fc26/repodata/repomd.xml
Resolving yum.qubes-os.org (yum.qubes-os.org)... 82.94.215.165
Connecting to yum.qubes-os.org (yum.qubes-os.org)|82.94.215.165|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3689 (3.6K) [application/xml]
Saving to: 'repomd.xml'

repomd.xml                                                      100%[=====================================================================================================================================================>]   3.60K  --.-KB/s    in 0.02s   

2018-03-24 14:47:50 (219 KB/s) - 'repomd.xml' saved [3689/3689]
n1m1 commented 6 years ago

Have you modified something on your side? I've done several tests on my Fedora templates and now everything works as expected.

Here, I just run in Dom0:

sudo qubesctl state.sls qvm.updates-via-whonix

However, the configuration in /etc/qubes-rpc/policy/qubes.UpdatesProxy has not changed.

@mirrorway ?

marmarek commented 6 years ago

No changes from me.

SSPJ commented 6 years ago

I am also experiencing this on Qubes 3.2 trying to update Fedora 23 template. Error: Failed to synchronize cache for repo 'qubes-vm-r3.2-current' I am using whatever the default updating mechanism on 3.2 is.

mirrorway commented 6 years ago

The wget from sys-whonix also works for me. One tweak with my sys-whonix is, the firewall tab has traffic restricted to my tor entry nodes. Not sure if disabling those rules would help, but I'm a little scared to try...

Caesarwm1 commented 6 years ago

Hello!

Here, it's nearly the same as with user @n1m1. I'm quite sure I checked (since many, many days) every single dnf possibilitiy like --refresh --clean all, etc., as well as issues #3135, #1352, and setting RestartSec=0s in etc/systemd/system/multi-user.target.wants/qubes-updates-proxy.service.

I can access http://yum.qubes-os.org/r4.0/current/vm/fc26/repodata/repomd.xml from sys-whonix, if I connect sys-whonix directly to Chris Laprise'/tasket's OpenVPN setup, not also to sys-firewall. But the fedora-26 template dnf issue (Qubes repos only; Fedora updates properly if I set Qubes repos enabled=0) existed before this VPN setup, so I don't think it's related.

sudo qubesctl state.sls qvm.updates-via-whonix in dom0 succeeds.

caesarwm1_qubes_ctl_issue3737

More information:

[user@fedora-26 ~]$ sudo dnf update -v
Loaded plugins: builddep, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, qubes-hooks, repoclosure, repograph, repomanage, reposync
DNF version: 2.7.5
cachedir: /var/cache/dnf
repo: using cache for: updates
updates: using metadata from Mon Mar 26 19:23:26 2018.
repo: using cache for: fedora
not found deltainfo for: Fedora 26 - x86_64
not found updateinfo for: Fedora 26 - x86_64
fedora: using metadata from Wed Jul  5 22:31:38 2017.
Cannot download 'http://yum.qubes-os.org/r4.0/current/vm/fc26': repomd.xml parser error: Parse error at line: 1 (not well-formed (invalid token)).
Error: Failed to synchronize cache for repo 'qubes-vm-r4.0-current'

I tried every other possibility here (sys-net and/or sys-firewall):

caesarwm1_updatesproxy_issue3737

Also, I tested both onion/hidden service addresses; at the moment I use the clearnet addresses.

Best regards!

marmarek commented 6 years ago

Try to modify /etc/yum.repos.d/qubes-r4.repo to use https instead of http.

Caesarwm1 commented 6 years ago

Oh my … That – dead ;) – simple. I didnt' read about this nowhere. And I read a lot. Thank you so much, marmarek! Best regards.

screenshot_2018-03-27_03-07-24

mirrorway commented 6 years ago

Change to https fixes it for me too :)

awokd commented 6 years ago

This is an issue on a fresh install of 4.0 final as well, with the default fedora-26 template. The https fix works, although those checksum errors are worrisome. Probably shouldn't expect users to edit files out of the box to get updates working. Is the http -> https redirect breaking it? Can 4.0 final be updated to include the https fix?

titxz commented 6 years ago

I'm using Qubes os r4.0 and i have this problem, but i can not find "qubes-r4.repo" file from the "/etc/yum.repos.d/" location.. But i found "fedora.repo" , "fedora-updates.repo" , "qubes-dom0.repo" , "qubes-templates.repo" files in this location. Please help me...!!!

andrewdavidwong commented 6 years ago

I'm using Qubes os r4.0 and i have this problem, but i can not find "qubes-r4.repo" file from the "/etc/yum.repos.d/" location.. But i found "fedora.repo" , "fedora-updates.repo" , "qubes-dom0.repo" , "qubes-templates.repo" files in this location.

It sounds like you're looking in dom0 when you should be looking in the desired Fedora TemplateVM.

carnak commented 6 years ago

Problem: Fedora 26 template repo update fails with error message: "Error: failed to synchronize cache for repo 'qubes-vm-r4.0-current'. Solution: From dom0 terminal window, I run following 5 commands: 1) qvm-service -l fedora-26 qubes-yum-proxy 2) qvm-service -e fedora-26 qubes-yum-proxy 3) qvm-service -l fedora-26 qubes-updates-proxy 4) qvm-service -e fedora-26 qubes-updates-proxy 5) reboot dom0

ymmv, though this has solved the problem for me repeatedly.

awokd commented 6 years ago

This is an (out of the box) issue with dom0 updates as well as Fedora templates. To summarize, after installing 4.0 final with the box checked to use Whonix for updates, neither dom0 or Fedora template updates work. I had to manually edit dom0's /etc/yum.repos.d/qubes-dom0.repo and fedora-26's /etc/yum.repos.d/qubes-r4.repo and replace http with https. This was not an issue until somewhere around the release of rc5. Was http://yum.qubes-os.org changed around that time to redirect to https? Should the redirect be turned off so updates will work over http again?

andrewdavidwong commented 6 years ago

Upgrading priority to critical. Rationale:

Even though the workaround is "easy," most users won't know to do it. They'll simply stop getting updates.

marmarek commented 6 years ago

Solution: From dom0 terminal window, I run following 5 commands:

qvm-service -l fedora-26 qubes-yum-proxy
qvm-service -e fedora-26 qubes-yum-proxy
qvm-service -l fedora-26 qubes-updates-proxy
qvm-service -e fedora-26 qubes-updates-proxy
reboot dom0

This is wrong. qubes-updates-proxy (and its deprecated name qubes-yum-proxy) is a service providing a proxy for templates, which should be running in whatever VM you use to download updates (sys-net or sys-whonix in default configuration, depending on initial setup choice). Running it in the template itself will (depending on start order) either do nothing, or nullify updates proxy mechanism and use plain network - which is not enabled for templates in R4 by default. So, I guess it worked for you @carnak because you additionally set some netvm for this template.

I have no idea why updates over http stopped working when using whonix (there is no redirection @awokd ; web browser use https by default because qubes-os.org is on hsts preload list). But the correct solution is to switch to https. I wanted to wait until we'll have proper metalinks in place (#1254) to avoid changing repository definitions multiple times, but it looks like it will take more time than expected. So, I'll switch to https right away. For this to be applied, the update will need to be installed (so, working update mechanism is required...).

qubesos-bot commented 6 years ago

Automated announcement from builder-github

The component core-agent-linux (including package python2-dnf-plugins-qubes-hooks-4.0.26-1.fc26) has been pushed to the r4.0 testing repository for the Fedora template. To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

awokd commented 6 years ago

Good point on HSTS, I didn't think that one through. I wonder if a Debian update broke it. I seem to recall seeing a curl update go through lately. I'll do another fresh install and try to down/upgrade packages.

qubesos-bot commented 6 years ago

Automated announcement from builder-github

The package qubes-core-agent_4.0.26-1+deb9u1 has been pushed to the r4.0 testing repository for the Debian template. To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing stretch-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

qubesos-bot commented 6 years ago

Automated announcement from builder-github

The package pykickstart-2.32-4.fc25 has been pushed to the r4.0 testing repository for dom0. To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

awokd commented 6 years ago

dom0 updates work sometimes, after all. I have a couple systems with 4.0 final installed 2-3 weeks ago that received no dom0 updates and were showing that Failed to synchronize cache for repo error until I edited the dom0 yum.repos to HTTPS. I did a fresh install a different system and when I did dom0 updates first, it worked (but still no Fedora template updates). Going to try again but attempt a Fedora template update first, in case it's caching a bad file somewhere. I found a similar bug report here https://bugzilla.redhat.com/show_bug.cgi?id=1207685 when HTTP headers got included in the body. Trying to work through https://access.redhat.com/solutions/189533. It seems to show up on other http only repos as well; for example, if I enable adobe-linux-x86_64.repo in the Fedora template.

For this to be applied, the update will need to be installed (so, working update mechanism is required...)

@andrewdavidwong, should an announcement go out with steps (like https://www.mail-archive.com/qubes-users@googlegroups.com/msg21002.html) on how to edit these files? Should a replacement ISO be generated or install documentation/release notes updated to mention it?

andrewdavidwong commented 6 years ago

For this to be applied, the update will need to be installed (so, working update mechanism is required...)

@andrewdavidwong, should an announcement go out with steps (like https://www.mail-archive.com/qubes-users@googlegroups.com/msg21002.html) on how to edit these files?

Yes. Please confirm the exact steps required, and we will make an announcement notifying current Qubes 4.0 users to perform them. (This is another case where we need #3430.)

Should a replacement ISO be generated or install documentation/release notes updated to mention it?

Yes, we should do one of these two things. I don't know whether @marmarek wants to issue a new ISO for this. If not, we'll have to mention it prominently in the installation guide and release notes so that users perform the steps right after they install.

andrewdavidwong commented 6 years ago

We should also update this FAQ entry:

https://www.qubes-os.org/faq/#i-keep-getting-failed-to-synchronize-cache-for-repo-errors-when-trying-to-update-my-fedora-templates

awokd commented 6 years ago

Edit Fedora-26 Update Location

  1. Click on the Qubes menu (Q in top left)
  2. Then, Template: fedora-26 > fedora-26: Terminal
  3. In the window that opens, enter sudo gedit /etc/yum.repos.d/qubes-r4.repo
  4. Change all instances of http to https (there are four)
  5. Click the Save button in the top right
  6. Close gedit and sudo poweroff the template

Edit Dom0 Update Location

  1. Click on the Qubes menu (Q in top left)
  2. Then, Terminal Emulator
  3. In the window that opens, enter sudo nano /etc/yum.repos.d/qubes-dom0.repo
  4. Change all instances of http to https (there are four)
  5. Once done, hit CTRL-x, y, and then ENTER to save changes and exit
  6. Type sudo qubes-dom0-update to check for updates

I'll take a look at that FAQ entry too.

andrewdavidwong commented 6 years ago

@awokd: Thank you for providing these steps. Before I send an edited version of these steps to the MLs and post it on the website, I want to confirm with @marmarek and/or @rootkovska on a couple of points:

  1. Can you confirm that these are actually steps that all current 4.0 users should perform?
  2. Should this be a QSB? It is highly unusual for us to post an important announcement that requires user action that is not a QSB. (I can't currently recall another case.) For example, it seems like this announcement should go to qubes-announce, even if it's not a QSB.
awokd commented 6 years ago

I'm having difficulty recreating the dom0 update issue on my test install- they've been working pretty reliably there. My other systems might have been operating against an old cache, but neither of them showed updates until yesterday when I changed them to https.

andrewdavidwong commented 6 years ago

It looks like @marmarek and @rootkovska are too busy to respond to this, so I'm just going to opt for a conservative course of action by sending the announcement to qubes-users and qubes-devel (but not qubes-announce or posting it on the website). If, for some reason, we later come to believe that sending it to qubes-users and qubes-devel was insufficient, we can always send or post it in additional places.

andrewdavidwong commented 6 years ago

Done: https://groups.google.com/d/topic/qubes-users/IJ3bpMhd8oY/discussion https://groups.google.com/d/topic/qubes-devel/VDKe6CWj9iA/discussion

andrewdavidwong commented 6 years ago

We should also update this FAQ entry:

After thinking about this some more, I don't think it's necessary to update the FAQ entry. The current content still applies in the general case, and the proposed addition wouldn't apply after this fix has been implemented. Stating the fix in the announcement, installation guide, and release notes should be sufficient to spread awareness to users.

adrelanos commented 6 years ago

I think the only realistic way to have most users who have this issue having it fixed for them is a dom0 upgrade which then uses salt to make the change for the user inside the template.

(But I am aware that the general development time scarcity as well.)

qubesos-bot commented 6 years ago

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.0 testing repository for the CentOS centos7 template. To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

qubesos-bot commented 6 years ago

Automated announcement from builder-github

The package pykickstart-2.32-4.fc25 has been pushed to the r4.0 stable repository for dom0. To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

qubesos-bot commented 6 years ago

Automated announcement from builder-github

The component core-agent-linux (including package python2-dnf-plugins-qubes-hooks-4.0.28-1.fc26) has been pushed to the r4.0 stable repository for the Fedora template. To install this update, please use the standard update command:

sudo yum update

Changes included in this update

qubesos-bot commented 6 years ago

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.0 stable repository for the Fedora centos7 template. To install this update, please use the standard update command:

sudo yum update

Changes included in this update

qubesos-bot commented 6 years ago

Automated announcement from builder-github

The package qubes-core-agent_4.0.28-1+deb9u1 has been pushed to the r4.0 stable repository for the Debian template. To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

ahimta commented 5 years ago

I had a similar issue with the error message Error: Failed to synchronize cache for repo 'qubes-vm-r4.0-current' for Fedora 29. I fixed it by switching from sys-firewall to sys-whonix and uncommenting onion URLs lines in /etc/yum.repos.d/qubes-r4.repo.

The exact error as I remember (can't produce it now with sys-firewall because it doesn't happen anymore) was a timeout and that all mirrors were tried.

I thought of trying to change from http to https as @marmarek suggested but the URLs were already using https. I then tested the URLs using wget (both HTTP and HTTPS) with sys-firewall and directly in Firefox using both sys-firewall and sys-whonix and both timedout.

Finally, I tried the commented onion URLs in anon-whonix and it worked.

Long story short, it seems that there are some DNS or server-side errors that are making this harder for both users and developers. For example, my issue was a timeout and someone here mentioned the XML file was invalid:sweat_smile:.