Make sys-cacher apply fix for URLs dynamically

[tlaurion]

Commitment

I confirm that I have read the following resources:

• How to ask questions The Smart Way
• Writing the perfect question
• Question checklist
• Could you please make my preference the default?

Current problem (if any)

Proposed solution

The value to a user, and who that user might be

As if today, sys-cacher is static and doesn't apply changes to repositories deployed after cacher deployment.

The ideal would be to have fixes for url automatically applied. Suggestion applied upstream bit not yet applied: https://github.com/unman/shaker/issues/17

.

[ben-grande]

As if today, sys-cacher is static and doesn't apply changes to repositories deployed after cacher deployment.

True, that is why unfortunately the external repositories such as signal, opentofu, syncthing etc are defaults to http:// instead of https://, because http works on both cases, with cacher or without and the authentication comes from the PGP verification anyway.

I was thinking of doing a script to automatically rewrite the repositories instead of using regexes with Salt, the downside is that changes won't be shown with qubesctl --show-output. The script is primarily intended for non templates that you have cacher configured, so you can rewrite their definitions at will. This script can be called by inotifywait to do changes on just certain files.

The ideal would be to have fixes for url automatically applied.

Downsides:

• Requires to install a package to the client, the upside is that it is very small;
• Dupliction of code, regexes present in the salt state and in the script, so I am leaning towards removing regexes transformations from the salt state and make the salt state call the script instead.

[ben-grande]

With the commit https://github.com/ben-grande/qusal/commit/7faf944964489980f159a5e27b4f96dd64347d43, you can rewrite the definitions inside the qube.

It does not do what you asked for because it doesn't monitor for files changes. But it is a step away from being able to do that more easily. It does not support arbitary files given as parameters though, it focuses on doing changes just to the files allowed by the distribution.

Apparently, inotify-tools has a good track record, with the last CVE from 15 years ago.

The following command:

sudo extrepo --offlinedata enable signal && sudo apt update && sudo apt install signal-desktop

Would become:

sudo extrepo --offlinedata enable signal && sudo apt-cacher-ng-repo && sudo apt update && sudo apt install signal-desktop

if not using inotiy, which doesn't seem a big deal.

I am still thinking if inotify is worth it.

[tlaurion]

With the commit 7faf944, you can rewrite the definitions inside the qube.

It does not do what you asked for because it doesn't monitor for files changes. But it is a step away from being able to do that more easily. It does not support arbitary files given as parameters though, it focuses on doing changes just to the files allowed by the distribution.

Apparently, inotify-tools has a good track record, with the last CVE from 15 years ago.

The following command:

sudo extrepo --offlinedata enable signal && sudo apt update && sudo apt install signal-desktop

Would become:

sudo extrepo --offlinedata enable signal && sudo apt-cacher-ng-repo && sudo apt update && sudo apt install signal-desktop

if not using inotiy, which doesn't seem a big deal.

Hmmm. Another long term debate without consensus.... Tackled that issue a while back https://forum.qubes-os.org/t/curl-proxy-wget-proxy-scripts-in-templates-so-users-can-add-gpg-distro-keys-linked-to-added-external-repositories/10935/96

Conclusion was to go in same direction you suggested above with extrepo which is nice, but yet again... There is no one size fits all as always. The whole point here, with users having deployed sys-cacher and having modified templates once (now with an helper script: great! but another additional step) is that users normally expect/trust upstream instructions to work as advertised (works TM). And those imply following, if desired, deploying Signal per upstream instructions. qusal resolves some default installation, but users still expect to be able to simply follow upstream instructions (main issue still being downloading public keys in template which would fail unless punching holes one way or the other or being advanced users and downloading pubkey in a vm, copying it around importing it blah blah which goes sideways from upstream instructions)...

Referred qubes forum discussion here goes into the more global considerations. extrepo resolves most, but yet another unmaintained properly project (element key rotated, not updated. I contributed there but got bored myself.... might be considered upstream here for the sake of contributing proper qubesos afditional software deployment though...)

TLDR.. inotify would fix in place. It is pretty stable code, doesnt do much. But watch for changes and apply what is needed to be applied in a goal like apt-cacher. I am sold here so not impartial. If goal is to offer transparency to end user and better UX, solution is not to add yet another command between the user and finally running needed app, but make it work out of the box abstracting the complexity to reach the goal of deploying needed app in a template/qube.

Inotifywait comes to the rescue if sys-cacher is deployed. If deployed, all urls should be fixed to work with apt-cacher. Nothing should stand in between and following upstream instructions (outside of deploying pubkey still problematic) and being able to run said needed application in a isolated qube/cloned+specialized template right out of the box.

I am still thinking if inotify is worth it.

It is. Yet again, for one needing to deploy soft with extrepo, one has to punch holes as well. Therefore, parts of the solution here lies in qubes forum post above. UX needs to also guide users on protections in place and how to get the desired outcome without expecting users to be experienced multi-years user of QubesOS. But that is another debate for which outcome has not been implemented nor upstreamed.

[ben-grande]

There are certain cases where the cacher proxy with URL rewriting are used:

1. Templates: URLs works but when adding external repositories, needs intervention;
2. Non-Templates: URLs do not work because most Non-Templates do not have and should not have access to the proxy, URLs can be modified on boot so the update check works (except if the qube has updates proxy service enabled);
3. Using sys-cacher as netvm: URLs do not have to be modified and update check will work.

The number 1 is the most used method (I my own experience) and it is the only method that requires to rewrite the URLs dynamically.

I will think more about inotify.

[ben-grande]

This issue was closed as qubes-update-check is now working. URLs are being rewritten dynamically on boot. URLs are not being automatically rewritten upon addition of new repository definitions, I am letting this be considered in #44.