Failed to return clean data when installing some formulas

[wassp-ds]

Commitment

I confirm that I have read the following resources:

• How to ask questions The Smart Way
• Writing the perfect question
• Question checklist
• Could you please make my preference the default?

Hi again, I initially thought this was true only for sys-ssh-agent formula, but seems it is true for others (true for terraform, sys-print), therefore I am thinking it's an underlying issue somewhere else.

Going the state route for installing the formulas (I found them more successful than the top route), I am getting to a point where the command sudo qubesctl --skip-dom0 --targets=<TPL-*> state.apply <FORMULA>.install returns

ERROR (exit code 20, details in /var/log/qubes/<MGMT-TPL-FORMULA>

Example with sys-ssh-agent:

output: tpl-sys-ssh-agent:
output:       -----------
output:       _error:
output:              Failed to return clean data
output:       retcode:
                          126
output:       stderr:
                          Request refused
output:       stdout:
output:  /usr/lib/python3.12/site-packages/salt/utils/jid.py:19: DeprecationWarning: datetime.datetime.utc.now() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
output:        return datetime.datetime.utcnow()
exit code: 20

---

Things I checked:

• I am connected to the Internet
• I tried to remove the created qubes and start from scratch
• I have rebooted the system for good measure

Any other suggestions?

[ben-grande]

The Request refused smells like a Qrexec policy issue.

On dom0:

sudo journalctl -fu qubes-qrexec-policy-daemon  | cut -d " " -f7- | tee /tmp/qrexec-policy.log

Watch the Qrexec logs and search for the resolution denied. You may look at the file /tmp/qrexec-policy.log for searching.

And run a state again with the --show-output as explained in the Salt troublehshooting section:

sudo qubesctl --show-output --skip-dom0 --targets=<TPL-*>  state.apply <FORMULA>.install

You may even narrow it down to a simple qube update:

sudo qubesctl --show-output --skip-dom0 --targets=<TPL-*>  state.apply update.qubes-vm

You can also try to run apt update inside a template see what happens in the policy and in the qube terminal.

[wassp-ds]

@ben-grande I managed to add a lot of stuff to my debugging arsenal, and your tips worked 👍 thanks!

[ben-grande]

Please share what worked for you so others can learn also. If you share, I can reference this issue later for others to learn. Of course, respecting your privacy, share only what you can.

[wassp-ds]

sudo journalctl -fu qubes-qrexec-policy-daemon | cut -d " " -f7- | tee /tmp/qrexec-policy.log

showcased the misconfigurations in the policies, and I was able to debug this - interestingly, the last pipe (tee /tmp/qrexec-policy.log) only produced an empty file, but after removing it I was able to debug this.

[Q: Do I keep the below here, or a separate Issue?] @ben-grande I am also trying to find the qusal.ConnectTCP policy, as I am getting

kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535
fatal: Could not read from remote repository

error when trying to connect the dev cube with split-ssh (sys-ssh-agent) to a remote private repo? I didn't have this issue before, after updating the sys-net setup I suddenly started getting it.

Thanks so much for your help!