Switching from "prod" to "staging" doesn't work, hits yum repository issue

[nathandyer]

• [x] I have searched for duplicates or related issues

Description

While trying to install securedrop-workstation-dom0-config-1.0.0rc2 for QA testing, I discovered an issue where sdw-admin --apply can get into an inconsistent state where it doesn't pick up on changes to the config.json file.

Specifically, I originally had the environment set to "prod" in config.json. When running sdw-admin --apply it provided an error with the repo (which is expected, since yum.securedrop.org does not have have fc37 packages yet).

After changing the environment to "staging" in config.json and re-running sdw-admin --apply, it produced the exact same error.

Only after manually editing /etc/yum.repos.d/securedrop-workstation-dom0.repo to include yum-test did sdw-admin --apply complete successfully.

Steps to Reproduce

1. Install Qubes 4.2 (I used Qubes-R4.2.2-rc1-x86_64.iso for testing purposes)
2. Set up SDW according to all the standard install instructions (I manually pulled securedrop-workstation-dom0-config-1.0.0rc2-1.fc37.noarch.rpm from yum-test.securedrop.org for QA purposes)
3. Configure config.json, but leave it set to prod for the environment
4. Run sdw-admin --apply and receive an error that it couldn't find the repo (expected)
5. Update config.json and set the environment to staging
6. Run sdw-admin --apply again

Expected Behavior

sdw-admin --apply detects the change and completes successfully

Actual Behavior

sdw-admin --apply gives the exact same error message, immediately

Error:

Error: Failed to download metadata for repo 'securedrop-workstation-dom0': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

Comments

May be an issue in this part of the provisioning script. (hat tip to @cfm for pulling that up so quickly)

[rocodes]

Thank you for filing - I think we do have some undocumented behaviour when orchestrating directly on top of an existing setup but switching environments (staging/prod/dev) like this.

My suspicion is that sudo rm /var/salt/cache, then sudo qubesctl saltutil.sync_all then sdw-admin --apply would resolve this, but not sure if it's worth testing too much since it's a developer-facing problem and right now the best case is to sdw-admin --uninstall then apply if switching between dev/staging/prod

[cfm]

A "fix it with documentation" approach so we don't get bitten by our own tooling: check if config.json has changed[^1] since the last sdw-admin --apply, in which case insist on sdw-admin --uninstall.

[^1]: diff config.json /usr/share/securedrop-workstation-dom0-config/config.json might be sufficient, unless there's some other canonical configuration used by sdw-admin --{validate,apply}.