QubesOS / qubes-issues

The Qubes OS Project issue tracker
https://www.qubes-os.org/doc/issue-tracking/
534 stars 46 forks source link

Distribute the build environment for qubes builder (e.g., template or Salt recipe) #8774

Open ddevz opened 8 months ago

ddevz commented 8 months ago

How to file a helpful issue

The problem you're addressing (if any)

When following the "qubes builder" instructions at https://www.qubes-os.org/doc/qubes-builder/ (and therefore https://www.qubes-os.org/doc/qubes-iso-building/) and it does not work (one example would be aborting the build with make[1]: *** [Makefile:74: linux-6.1.62.tar.UNTRUSTED] Error 1), it is not obvious if it is because the version of fedora you chose is not in sync with the version of qubes-builder you are using. (For example, you could be using a new version of fedora that hasn't been tested with the build process yet). It is also not obvious if the part of the instructions which mentions fedora versions is out of date.

The solution you'd like

Distribute the build environment needed to run qubes builder without error. One option would be to make a template for "qubes builder" which allows people to install with something like:

sudo qubes-dom0-update qubes-template-qubes-builder

The value to a user, and who that user might be

Easier way to figure out the build process for the first time. Easier way for people to build the non-distributed templates (example: ubuntu)

Augsch123 commented 8 months ago

I think distributing a salt file to configure a Qubes OS developing environment is better than maintaining and distributing a full template.

In fact I have made something for myself:

fed-min-qubes-dev.sls:

``` {% set fedoraver = salt['pillar.get']('fedoraver', 'fedora-38') %} {% if grains['id'] == 'dom0' %} ensure-{{ fedoraver }}-minimal-installed: qvm.template_installed: - name: {{ fedoraver }}-minimal - fromrepo: qubes-templates-itl create-{{ fedoraver }}-builder: qvm.clone: - name: {{ fedoraver }}-builder - source: {{ fedoraver }}-minimal - require: - qvm: ensure-{{ fedoraver }}-minimal-installed create-qubes-builder-dvm: qvm.vm: - name: qubes-builder-dvm - present: - template: {{ fedoraver }}-builder - label: red - prefs: - virt_mode: pvh - label: red - template-for-dispvms: true - memory: 800 - maxmem: 8000 - vcpus: 6 - require: - qvm: create-{{ fedoraver }}-builder create-qubes-work: qvm.vm: - name: qubes-work - present: - template: {{ fedoraver }}-builder - label: red - prefs: - virt_mode: pvh - label: red - memory: 400 - maxmem: 1000 - default-dispvm: qubes-builder-dvm - require: - qvm: create-qubes-builder-dvm resize-builder-volume: cmd.run: - name: qvm-volume resize qubes-builder-dvm:private 32212254720 ;qvm-volume resize qubes-work:private 32212254720 builder-policy: file.managed: - name: /etc/qubes/policy.d/30-qubesbuilder.policy - contents: | admin.vm.CreateDisposable * qubes-work qubes-builder-dvm allow target=dom0 admin.vm.Start * qubes-work @tag:disp-created-by-qubes-work allow target=dom0 admin.vm.Kill * qubes-work @tag:disp-created-by-qubes-work allow target=dom0 qubesbuilder.FileCopyIn * qubes-work @tag:disp-created-by-qubes-work allow qubesbuilder.FileCopyOut * qubes-work @tag:disp-created-by-qubes-work allow qubes.Filecopy * qubes-work @tag:disp-created-by-qubes-work allow qubes.WaitForSession * qubes-work @tag:disp-created-by-qubes-work allow qubes.VMShell * qubes-work @tag:disp-created-by-qubes-work allow {% elif grains['id'] == [fedoraver, '-builder']|join %} update: cmd.run: - name: dnf install -y qubes-core-agent-networking qubes-core-agent-passwordless-root qubes-core-agent-thunar xfce4-terminal unzip wqy-microhei-fonts python3-packaging createrepo_c devscripts gpg python3-pyyaml rpm reprepro python3-pathspec mktorrent rpm-sign tree mock openssl python3-jinja2-cli m4 asciidoc rsync qubes-gpg-split dnf-plugins-core debootstrap dpkg-dev git mock pbuilder which perl-Digest-MD5 perl-Digest-SHA python3-sh rpm-build rpmdevtools wget python3-debian python3-lxml systemd-udev python3-click qubes-gpg-split {% elif grains['id'] == 'qubes-builder-dvm' %} builderdirectory: cmd.run: - name: mkdir -p /rw/bind-dirs/builder /rw/config/qubes-bind-dirs.d ; echo "binds+=('/builder')" > /rw/config/qubes-bind-dirs.d/builder.conf ; appendrc: file.append: - name: /rw/config/rc.local - text: mount /builder -o dev,suid,remount {% elif grains['id'] == 'qubes-work' %} git-fetch: cmd.run: - name: cd /home/user ; git clone https://github.com/QubesOS/qubes-infrastructure-mirrors ; git clone /https://github.com/QubesOS/qubes-builderv2 - runas: user install-infrastructure: cmd.run: - name: cd /home/user/qubes-infrastructure-mirrors ; python3 setup.py build ; python3 setup.py install {% endif %} ```

fed-min-qubes-dev.top:

``` {% set fedoraver = salt['pillar.get']('fedoraver', 'fedora-38') %} user: dom0 or {{ fedoraver }}-builder or qubes-builder-dvm or qubes-work: - fed-min-qubes-dev ```

So this basically creates a template fedora-38-builder and configures it to be able to work as both the actual qubes executor and where qubes-builderv2 locates. And it creates qubes-builder-dvm which is the executor, and qubes-work which is where you work with the source code and issue commands to build the packages.

Update: Fixed the top file.

Update2: Fixed the sls file.

Update3: Keep up with builderv2 code changes.

ddevz commented 8 months ago

Neat. Any chance we could get something like this added to the distribution?

Augsch123 commented 8 months ago

Any chance we could get something like this added to the distribution?

That would be nice.

Having many people using the same script would make the process more tested

In fact almost all things that this script does come from the official builderv2 repo's README. I just put them all together to ease my future re-install or something.

If fedora-38 template is in stable, and fedora-40 template is being developed, then things like ('fedoraver', 'fedora-40') could be in the script in the development branch version and ('fedoraver', 'fedora-38) could be in the script in the stable branch )

In my opinion the template chosen is not something critical to make things work, as long as it's a supported template ( i.e. not have reached EOL ). That's because even with qubes executor, the builder still uses mock to be able to build against different target distributions.

People would be more likely to use a salt script in dom0 that has been reviewed and added to the qubes distro, rather then one they download.

I agree. This script was originally just made for myself, but since you brought up this topic, I decided to post it here, thinking it might inspire someone to look into salt and eventually benefit the whole community.

tlaurion commented 7 months ago

@marmarek @fepitre I would strongly advice distributing a qubes-builder salt recipe. To be honest, its the third time I try to follow instructions of README at https://github.com/QubesOS/qubes-builderv2 and i'm still failing. Will give https://github.com/QubesOS/qubes-issues/issues/8774#issuecomment-1867854489 a shot myself.

@ddevz Can you rename this issue to "distribute salt recipe under qubes 4.2" or something similar? @andrewdavidwong ?

Note that @unman did a builder recipe at https://github.com/unman/shaker/tree/main/builder for v1 builder, which is packaged anfd worked for Q4.1. Maybe propose a PR there?

tlaurion commented 7 months ago

So this basically creates a template fedora-38-builder and configures it to be able to work as both the actual qubes executor and where qubes-builderv2 locates. And it creates qubes-builder-dvm which is the executor, and qubes-work which is where you work with the source code and issue commands to build the packages.

@Augsch123 : mind to explain how to use this properly?

andrewdavidwong commented 7 months ago

Can you rename this issue to "distribute salt recipe under qubes 4.2" or something similar? @andrewdavidwong ?

How about the title now? I don't want to put "4.2" in the title, because that's redundant with labels, prone to going out-of-date, and isn't necessarily accurate, as Qubes Builder v2 isn't necessarily specific to 4.2. (For example, it could be used to build 4.3 or 5.0, and presumably people would still want this at that time.)

I also don't want to narrow the title to just "Distribute Salt recipe" because that assumes that Salt is the optimal implementation, which we don't know for certain. The issue title should state the goal you want to achieve, then let the developers decide on the best means to achieve it.

tlaurion commented 7 months ago

On a more general note @marmarek @fepitre @andrewdavidwong @DemiMarie @marmarta : @unman salt recipes under shaker project were spot on for Q4.1, but as for anything not upstreamed, it's lagging behind since the effort of one man army and is lacking eyes of qubesos devs/community to thrive properly and reach their qubesos users.

May I suggest forking shaker under contrib repo and collaborating with @unman, so that qubesos minimally packs shaker salt recipes for users to decide if they want to deploy such from contrib packages manually, but ideally, have rpm packages deployed per rpm spec files already existing and even have the salt installer also forked and contrib package of https://github.com/unman/qubes-task deployed as well?

This has been discussed numerous times in the past (persona related templates, moved to salt recipes to ease qubes customization deployment) going in the idealistic dream of having qubesos installer connected to the internet and permitting users/oem to deploy customized installation without needing to craft a custom qubesos iso for their needs?

Some references:

This is the best example I have for documentation as code. This explains directly how salt works to the curious, is easy to audit and facilitates learning. And permits people to understand easily what is happening (audit), simply looking at the spec files, which does the magic in an explicit fashion. More people would contribute salt recipes. More salt recipes would be installable by qubes-task GUI effortlessly within OS. More eyes. More easyness to contribute. Less documentation. Win win everywhere.

And then at term, qubes-task integration in qubesos installer with network access making the system behave exactly as needed on first boot? Wow. That would be a dream came true.

Augsch123 commented 7 months ago

@Augsch123 : mind to explain how to use this properly?

I followed the steps below:

  1. Set up /srv/user_salt/ by following https://forum.qubes-os.org/t/qubes-salt-beginners-guide/20126
  2. Create /srv/user_salt/fed-min-qubes-dev.sls with content pasted above.
  3. Create /srv/user_salt/fed-min-qubes-dev.top with content pasted above.
  4. Run sudo qubesctl top.enable fed-min-qubes-dev pillarenv=user saltenv=user in dom0.
  5. Run sudo qubesctl state.highstate, and qubes development environment should be ready.
tlaurion commented 7 months ago

@Augsch123 : mind to explain how to use this properly?

I followed the steps below:

1. Set up `/srv/user_salt/` by following https://forum.qubes-os.org/t/qubes-salt-beginners-guide/20126

2. Create `/srv/user_salt/fed-min-qubes-dev.sls` with content pasted above.

3. Create `/srv/user_salt/fed-min-qubes-dev.top` with content pasted above.

4. Run `sudo qubesctl top.enable fed-min-qubes-dev pillarenv=user saltenv=user` in dom0.

5. Run `sudo qubesctl state.highstate`, and qubes development environment should be ready.

Hello @Augsch123 ! Thanks you for sharing and updating your salts with V3.

I tried to replicate.

Replication trace ``` [user@dom0 qubes-builder-v2-salt]$ mv /srv/ formulas/ pillar/ reactor/ salt/ user_salt/ [user@dom0 qubes-builder-v2-salt]$ sudo mv srv/user_salt/fed-min-qubes-dev.* /srv/user_salt/ (failed reverse-i-search)`qubsc': sudo mv srv/user_salt/fed-min-^Cbes-dev.* /srv/user_salt/ [user@dom0 qubes-builder-v2-salt]$ sudo qubesctl top.enable fed-min-qubes-dev pillarenv=user saltenv=user local: ---------- fed-min-qubes-dev.top: ---------- status: unchanged [user@dom0 qubes-builder-v2-salt]$ sudo qubesctl top.enable fed-min-qubes-dev pillarenv=user saltenv=user^C [user@dom0 qubes-builder-v2-salt]$ sudo qubesctl state.highstate [ERROR ] State 'qvm.template_installed' was not found in SLS 'fed-min-qubes-dev' Reason: 'qvm.template_installed' is not available. [ERROR ] Command 'qvm-volume' failed with return code: 1 [ERROR ] stderr: qvm-volume: error: no vm 'qubes-builder-dvm' qvm-volume: error: no vm 'qubes-work' [ERROR ] retcode: 1 [ERROR ] {'pid': 17410, 'retcode': 1, 'stdout': '', 'stderr': "qvm-volume: error: no vm 'qubes-builder-dvm'\nqvm-volume: error: no vm 'qubes-work'"} local: ---------- ID: ensure-fedora-38-minimal-installed Function: qvm.template_installed Name: fedora-38-minimal Result: False Comment: State 'qvm.template_installed' was not found in SLS 'fed-min-qubes-dev' Reason: 'qvm.template_installed' is not available. Changes: ---------- ID: create-fedora-38-builder Function: qvm.clone Name: fedora-38-builder Result: False Comment: One or more requisite failed: fed-min-qubes-dev.ensure-fedora-38-minimal-installed Started: 12:09:46.507544 Duration: 0.01 ms Changes: ---------- ID: create-qubes-builder-dvm Function: qvm.vm Name: qubes-builder-dvm Result: False Comment: One or more requisite failed: fed-min-qubes-dev.create-fedora-38-builder Started: 12:09:46.523374 Duration: 0.01 ms Changes: ---------- ID: create-qubes-work Function: qvm.vm Name: qubes-work Result: False Comment: One or more requisite failed: fed-min-qubes-dev.create-qubes-builder-dvm Started: 12:09:46.537349 Duration: 0.01 ms Changes: ---------- ID: resize-builder-volume Function: cmd.run Name: qvm-volume resize qubes-builder-dvm:private 32212254720 ;qvm-volume resize qubes-work:private 32212254720 Result: False Comment: Command "qvm-volume resize qubes-builder-dvm:private 32212254720 ;qvm-volume resize qubes-work:private 32212254720" run Started: 12:09:46.537698 Duration: 326.819 ms Changes: ---------- pid: 17410 retcode: 1 stderr: qvm-volume: error: no vm 'qubes-builder-dvm' qvm-volume: error: no vm 'qubes-work' stdout: ---------- ID: builder-policy Function: file.managed Name: /etc/qubes/policy.d/30-qubesbuilder.policy Result: True Comment: File /etc/qubes/policy.d/30-qubesbuilder.policy updated Started: 12:09:46.864793 Duration: 5.576 ms Changes: ---------- diff: New file Summary for local ------------ Succeeded: 1 (changed=2) Failed: 5 ------------ Total states run: 6 Total run time: 332.425 ms DOM0 configuration failed, not continuing ```

V3 fed-min-qubes-dev.sls :


[user@dom0 qubes-builder-v2-salt]$ sudo cat /srv/user_salt/fed-min-qubes-dev.sls 
{% set fedoraver = salt['pillar.get']('fedoraver', 'fedora-38') %}

{% if grains['id'] == 'dom0' %}

ensure-{{ fedoraver }}-minimal-installed:
  qvm.template_installed:
    - name: {{ fedoraver }}-minimal
    - fromrepo: qubes-templates-itl

create-{{ fedoraver }}-builder:
  qvm.clone:
    - name: {{ fedoraver }}-builder
    - source: {{ fedoraver }}-minimal
    - require:
      - qvm: ensure-{{ fedoraver }}-minimal-installed

create-qubes-builder-dvm:
  qvm.vm:
    - name: qubes-builder-dvm
    - present:
      - template: {{ fedoraver }}-builder
      - label: red
    - prefs:
      - virt_mode: pvh
      - label: red
      - template-for-dispvms: true
      - memory: 800
      - maxmem: 8000
      - vcpus: 6
    - require:
      - qvm: create-{{ fedoraver }}-builder

create-qubes-work:
  qvm.vm:
    - name: qubes-work
    - present:
      - template: {{ fedoraver }}-builder
      - label: red
    - prefs:
      - virt_mode: pvh
      - label: red
      - memory: 400
      - maxmem: 1000
      - default-dispvm: qubes-builder-dvm
    - require:  
      - qvm: create-qubes-builder-dvm

resize-builder-volume:
  cmd.run:
    - name: qvm-volume resize qubes-builder-dvm:private 32212254720 ;qvm-volume resize qubes-work:private 32212254720

builder-policy:
  file.managed:
    - name: /etc/qubes/policy.d/30-qubesbuilder.policy
    - contents: |
        admin.vm.CreateDisposable * qubes-work qubes-builder-dvm allow target=dom0
        admin.vm.Start * qubes-work @tag:disp-created-by-qubes-work allow target=dom0
        admin.vm.Kill * qubes-work @tag:disp-created-by-qubes-work allow target=dom0
        qubesbuilder.FileCopyIn * qubes-work @tag:disp-created-by-qubes-work allow
        qubesbuilder.FileCopyOut * qubes-work @tag:disp-created-by-qubes-work allow
        qubes.Filecopy * qubes-work @tag:disp-created-by-qubes-work allow
        qubes.WaitForSession * qubes-work @tag:disp-created-by-qubes-work allow
        qubes.VMShell * qubes-work @tag:disp-created-by-qubes-work allow

{% elif grains['id'] == [fedoraver, '-builder']|join %}

update:
  cmd.run:
    - name: dnf install -y qubes-core-agent-networking qubes-core-agent-passwordless-root qubes-core-agent-thunar xfce4-terminal unzip wqy-microhei-fonts python3-packaging createrepo_c devscripts gpg python3-pyyaml rpm reprepro python3-pathspec mktorrent rpm-sign tree mock openssl python3-jinja2-cli m4 asciidoc rsync qubes-gpg-split dnf-plugins-core debootstrap dpkg-dev git mock pbuilder which perl-Digest-MD5 perl-Digest-SHA  python3-sh rpm-build rpmdevtools wget python3-debian python3-lxml systemd-udev python3-click qubes-gpg-split

{% elif grains['id'] == 'qubes-builder-dvm' %}

builderdirectory:
  cmd.run:
    - name: mkdir -p /rw/bind-dirs/builder /rw/config/qubes-bind-dirs.d ; echo "binds+=('/builder')" > /rw/config/qubes-bind-dirs.d/builder.conf ;

appendrc:
  file.append:
    - name: /rw/config/rc.local
    - text: mount /builder -o dev,suid,remount

{% elif grains['id'] == 'qubes-work' %}

git-fetch:
  cmd.run:
    - name: cd /home/user ; git clone https://github.com/QubesOS/qubes-infrastructure-mirrors ; git clone /https://github.com/QubesOS/qubes-builderv2
    - runas: user
install-infrastructure:
  cmd.run:
    - name: cd /home/user/qubes-infrastructure-mirrors ; python3 setup.py build ; python3 setup.py install

{% endif %}
[user@dom0 qubes-builder-v2-salt]$ sudo cat /srv/user_salt/fed-min-qubes-dev.top 
{% set fedoraver = salt['pillar.get']('fedoraver', 'fedora-38') %}

user:
  dom0 or {{ fedoraver }}-builder or qubes-builder-dvm or qubes-work:
    - fed-min-qubes-dev

V3 fed-min-qubes-dev.top:

[user@dom0 qubes-builder-v2-salt]$ sudo cat /srv/user_salt/fed-min-qubes-dev.top
{% set fedoraver = salt['pillar.get']('fedoraver', 'fedora-38') %}

user:
  dom0 or {{ fedoraver }}-builder or qubes-builder-dvm or qubes-work:
    - fed-min-qubes-dev

What am I doing wrong?

Augsch123 commented 7 months ago

@tlaurion Did you manually create /srv/user_salt? Unfortunately that's not what I meant. I think you'll need to run sudo qubesctl state.sls qubes.user-dirs to set up it properly, as mentioned in that forum thread.

We run the command sudo qubesctl state.sls qubes.user-dirs. Salt applies the corresponding state, and tells us that some files and directories were created. Among these directories we can find /srv/user_salt/: this is the main directory where we’ll place our own state configuration files.

Reading the logs, [ERROR ] State 'qvm.template_installed' was not found in SLS 'fed-min-qubes-dev' is strange and the only reason that I can come up with for it not being found, is that you might manually created/srv/user_salt.

tlaurion commented 7 months ago

@tlaurion Did you manually create /srv/user_salt? Unfortunately that's not what I meant. I think you'll need to run sudo qubesctl state.sls qubes.user-dirs to set up it properly, as mentioned in that forum thread.

We run the command sudo qubesctl state.sls qubes.user-dirs. Salt applies the corresponding state, and tells us that some files and directories were created. Among these directories we can find /srv/user_salt/: this is the main directory where we’ll place our own state configuration files.

Reading the logs, [ERROR ] State 'qvm.template_installed' was not found in SLS 'fed-min-qubes-dev' is strange and the only reason that I can come up with for it not being found, is that you might manually created/srv/user_salt.

Right. I didn't understand that part and re-did. On Q4.2, fedora-38-minimal installed from command line with qvm-template.

Replication trace ``` [user@dom0 ~]$ sudo rm -rf /srv/user_salt/ [user@dom0 ~]$ sudo qubesctl saltutil.sync_all [CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable local: ---------- beacons: clouds: engines: executors: grains: - grains.boot_mode - grains.pci_devs - grains.redefined_dom0_grains - grains.whonix log_handlers: matchers: modules: - modules.debug - modules.ext_module_qvm - modules.module_utils - modules.qubes - modules.qubes_dom0_update - modules.topd output: pillar: - pillar.qvm_prefs proxymodules: renderers: returners: sdb: serializers: states: - states.debug - states.ext_state_qvm - states.status thorium: utils: - utils.__init__ - utils.fileinfo - utils.matcher - utils.nulltype - utils.pathinfo - utils.pathutils - utils.qubes_utils - utils.toputils [user@dom0 ~]$ sudo qubesctl state.sls qubes.user-dirs local: ---------- ID: directory_srv_user_salt Function: file.directory Name: /srv/user_salt Result: True Comment: Started: 21:51:31.099879 Duration: 8.813 ms Changes: ---------- /srv/user_salt: ---------- directory: new ---------- ID: directory_srv_user_pillar Function: file.directory Name: /srv/user_pillar Result: True Comment: The directory /srv/user_pillar is in the correct state Started: 21:51:31.108864 Duration: 1.931 ms Changes: ---------- ID: directory_srv_user_formulas Function: file.directory Name: /srv/user_formulas Result: True Comment: The directory /srv/user_formulas is in the correct state Started: 21:51:31.110972 Duration: 2.031 ms Changes: ---------- ID: /srv/user_salt/top.sls Function: file.managed Result: True Comment: File /srv/user_salt/top.sls updated Started: 21:51:31.113821 Duration: 96.33 ms Changes: ---------- diff: New file mode: 0640 ---------- ID: /srv/user_pillar/top.sls Function: file.managed Result: True Comment: File /srv/user_pillar/top.sls exists with proper permissions. No changes made. Started: 21:51:31.210526 Duration: 1.843 ms Changes: ---------- ID: /srv/user_salt/locale Function: file.directory Result: True Comment: Started: 21:51:31.212616 Duration: 2.208 ms Changes: ---------- /srv/user_salt/locale: ---------- directory: new ---------- ID: /srv/user_salt/locale/init.sls Function: file.managed Result: True Comment: File /srv/user_salt/locale/init.sls updated Started: 21:51:31.215139 Duration: 113.841 ms Changes: ---------- diff: New file mode: 0640 Summary for local ------------ Succeeded: 7 (changed=4) Failed: 0 ------------ Total states run: 7 Total run time: 226.997 ms [user@dom0 ~]$ sudo cp ~/Downloads/qubes-builder-v2-salt/srv/user_salt/fed-min-qubes-dev.* /srv/user_salt/ [user@dom0 ~]$ sudo qubesctl state.sls qubes.user-dirs local: ---------- ID: directory_srv_user_salt Function: file.directory Name: /srv/user_salt Result: True Comment: Directory /srv/user_salt updated Started: 21:51:40.501453 Duration: 13.109 ms Changes: ---------- /srv/user_salt/fed-min-qubes-dev.sls: ---------- mode: 0640 /srv/user_salt/fed-min-qubes-dev.top: ---------- mode: 0640 mode: 0640 ---------- ID: directory_srv_user_pillar Function: file.directory Name: /srv/user_pillar Result: True Comment: The directory /srv/user_pillar is in the correct state Started: 21:51:40.514759 Duration: 1.732 ms Changes: ---------- ID: directory_srv_user_formulas Function: file.directory Name: /srv/user_formulas Result: True Comment: The directory /srv/user_formulas is in the correct state Started: 21:51:40.516665 Duration: 2.103 ms Changes: ---------- ID: /srv/user_salt/top.sls Function: file.managed Result: True Comment: File /srv/user_salt/top.sls exists with proper permissions. No changes made. Started: 21:51:40.519191 Duration: 1.896 ms Changes: ---------- ID: /srv/user_pillar/top.sls Function: file.managed Result: True Comment: File /srv/user_pillar/top.sls exists with proper permissions. No changes made. Started: 21:51:40.521442 Duration: 1.677 ms Changes: ---------- ID: /srv/user_salt/locale Function: file.directory Result: True Comment: The directory /srv/user_salt/locale is in the correct state Started: 21:51:40.523270 Duration: 1.693 ms Changes: ---------- ID: /srv/user_salt/locale/init.sls Function: file.managed Result: True Comment: File /srv/user_salt/locale/init.sls exists with proper permissions. No changes made. Started: 21:51:40.525285 Duration: 1.551 ms Changes: Summary for local ------------ Succeeded: 7 (changed=1) Failed: 0 ------------ Total states run: 7 Total run time: 23.761 ms [user@dom0 ~]$ sudo qubesctl saltutil.sync_all [WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order. local: ---------- beacons: clouds: engines: executors: grains: log_handlers: matchers: modules: output: pillar: proxymodules: renderers: returners: sdb: serializers: states: thorium: utils: [user@dom0 ~]$ sudo cp ~/Downloads/qubes-builder-v2-salt/srv/user_salt/fed-min-qubes-dev.* /srv/user_salt/ [user@dom0 ~]$ sudo qubesctl saltutil.sync_all [CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable [WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order. local: ---------- beacons: clouds: engines: executors: grains: - grains.boot_mode - grains.pci_devs - grains.redefined_dom0_grains - grains.whonix log_handlers: matchers: modules: - modules.debug - modules.ext_module_qvm - modules.module_utils - modules.qubes - modules.qubes_dom0_update - modules.topd output: pillar: - pillar.qvm_prefs proxymodules: renderers: returners: sdb: serializers: states: - states.debug - states.ext_state_qvm - states.status thorium: utils: - utils.__init__ - utils.fileinfo - utils.matcher - utils.nulltype - utils.pathinfo - utils.pathutils - utils.qubes_utils - utils.toputils [user@dom0 ~]$ qvm-ls | grep 38-minimal fedora-38-minimal Halted TemplateVM black - - [user@dom0 ~]$ sudo qubesctl state.highstate [WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order. [ERROR ] State 'qvm.template_installed' was not found in SLS 'fed-min-qubes-dev' Reason: 'qvm.template_installed' is not available. [ERROR ] Command 'qvm-volume' failed with return code: 1 [ERROR ] stderr: qvm-volume: error: no vm 'qubes-builder-dvm' qvm-volume: error: no vm 'qubes-work' [ERROR ] retcode: 1 [ERROR ] {'pid': 10868, 'retcode': 1, 'stdout': '', 'stderr': "qvm-volume: error: no vm 'qubes-builder-dvm'\nqvm-volume: error: no vm 'qubes-work'"} local: ---------- ID: ensure-fedora-38-minimal-installed Function: qvm.template_installed Name: fedora-38-minimal Result: False Comment: State 'qvm.template_installed' was not found in SLS 'fed-min-qubes-dev' Reason: 'qvm.template_installed' is not available. Changes: ---------- ID: create-fedora-38-builder Function: qvm.clone Name: fedora-38-builder Result: False Comment: One or more requisite failed: fed-min-qubes-dev.ensure-fedora-38-minimal-installed Started: 21:53:09.729493 Duration: 0.01 ms Changes: ---------- ID: create-qubes-builder-dvm Function: qvm.vm Name: qubes-builder-dvm Result: False Comment: One or more requisite failed: fed-min-qubes-dev.create-fedora-38-builder Started: 21:53:09.744015 Duration: 0.01 ms Changes: ---------- ID: create-qubes-work Function: qvm.vm Name: qubes-work Result: False Comment: One or more requisite failed: fed-min-qubes-dev.create-qubes-builder-dvm Started: 21:53:09.759163 Duration: 0.019 ms Changes: ---------- ID: resize-builder-volume Function: cmd.run Name: qvm-volume resize qubes-builder-dvm:private 32212254720 ;qvm-volume resize qubes-work:private 32212254720 Result: False Comment: Command "qvm-volume resize qubes-builder-dvm:private 32212254720 ;qvm-volume resize qubes-work:private 32212254720" run Started: 21:53:09.759499 Duration: 271.28 ms Changes: ---------- pid: 10868 retcode: 1 stderr: qvm-volume: error: no vm 'qubes-builder-dvm' qvm-volume: error: no vm 'qubes-work' stdout: ---------- ID: builder-policy Function: file.managed Name: /etc/qubes/policy.d/30-qubesbuilder.policy Result: True Comment: File /etc/qubes/policy.d/30-qubesbuilder.policy is in the correct state Started: 21:53:10.031014 Duration: 117.751 ms Changes: Summary for local ------------ Succeeded: 1 (changed=1) Failed: 5 ------------ Total states run: 6 Total run time: 389.070 ms DOM0 configuration failed, not continuing [user@dom0 ~]$ ```

@Augsch123 Thank for your time.

Multiple errors. fedora-38-minimal not found, actions are applied in wrong order... Too new to salt to understand what i'm doing here but unfortunately this is not working out of the box on Q4.2, just like upstream instructions. :/

Any idea for an update 4 at https://github.com/QubesOS/qubes-issues/issues/8774#issuecomment-1867854489 ? @unman (shaker: builderv2?) @marmarek?

tlaurion commented 7 months ago

@Augsch123 Seems like there is a bug on Q4.2 that was unfixed. https://github.com/QubesOS/qubes-issues/issues/8491#issuecomment-1890843137

Basically this worked:

  1. sudo ln -s /srv/salt/qubes/user-dirs.top /srv/salt/_tops/base/user-dirs.top #needs bugfix @marmarek
    1. Set up /srv/user_salt/ by following https://forum.qubes-os.org/t/qubes-salt-beginners-guide/20126

    2. Create /srv/user_salt/fed-min-qubes-dev.sls with content pasted above.

    3. Create /srv/user_salt/fed-min-qubes-dev.top with content pasted above.

    4. Run sudo qubesctl top.enable fed-min-qubes-dev pillarenv=user saltenv=user in dom0.

    5. Run sudo qubesctl state.highstate, and qubes development environment should be ready.

Replication trace ``` [user@dom0 ~]$ sudo ln -s /srv/salt/qubes/user-dirs.top /srv/salt/_tops/base/user-dirs.top [user@dom0 ~]$ sudo qubesctl state.sls qubes.user-dirs local: ---------- ID: directory_srv_user_salt Function: file.directory Name: /srv/user_salt Result: True Comment: The directory /srv/user_salt is in the correct state Started: 23:34:35.662384 Duration: 28.998 ms Changes: ---------- ID: directory_srv_user_pillar Function: file.directory Name: /srv/user_pillar Result: True Comment: The directory /srv/user_pillar is in the correct state Started: 23:34:35.691821 Duration: 5.285 ms Changes: ---------- ID: directory_srv_user_formulas Function: file.directory Name: /srv/user_formulas Result: True Comment: The directory /srv/user_formulas is in the correct state Started: 23:34:35.698360 Duration: 6.51 ms Changes: ---------- ID: /srv/user_salt/top.sls Function: file.managed Result: True Comment: File /srv/user_salt/top.sls exists with proper permissions. No changes made. Started: 23:34:35.705890 Duration: 7.64 ms Changes: ---------- ID: /srv/user_pillar/top.sls Function: file.managed Result: True Comment: File /srv/user_pillar/top.sls exists with proper permissions. No changes made. Started: 23:34:35.714657 Duration: 5.817 ms Changes: ---------- ID: /srv/user_salt/locale Function: file.directory Result: True Comment: The directory /srv/user_salt/locale is in the correct state Started: 23:34:35.720923 Duration: 5.68 ms Changes: ---------- ID: /srv/user_salt/locale/init.sls Function: file.managed Result: True Comment: File /srv/user_salt/locale/init.sls exists with proper permissions. No changes made. Started: 23:34:35.727541 Duration: 6.806 ms Changes: Summary for local ------------ Succeeded: 7 Failed: 0 ------------ Total states run: 7 Total run time: 66.736 ms [user@dom0 ~]$ sudo cp ~/Downloads/qubes-builder-v2-salt/srv/user_salt/fed-min-qubes-dev.* /srv/user_salt/ [user@dom0 ~]$ sudo qubesctl state.sls qubes.user-dirs local: ---------- ID: directory_srv_user_salt Function: file.directory Name: /srv/user_salt Result: True Comment: Directory /srv/user_salt updated Started: 23:35:28.667067 Duration: 36.337 ms Changes: ---------- /srv/user_salt/fed-min-qubes-dev.sls: ---------- mode: 0640 /srv/user_salt/fed-min-qubes-dev.top: ---------- mode: 0640 mode: 0640 ---------- ID: directory_srv_user_pillar Function: file.directory Name: /srv/user_pillar Result: True Comment: The directory /srv/user_pillar is in the correct state Started: 23:35:28.703835 Duration: 4.423 ms Changes: ---------- ID: directory_srv_user_formulas Function: file.directory Name: /srv/user_formulas Result: True Comment: The directory /srv/user_formulas is in the correct state Started: 23:35:28.708650 Duration: 4.41 ms Changes: ---------- ID: /srv/user_salt/top.sls Function: file.managed Result: True Comment: File /srv/user_salt/top.sls exists with proper permissions. No changes made. Started: 23:35:28.713909 Duration: 4.576 ms Changes: ---------- ID: /srv/user_pillar/top.sls Function: file.managed Result: True Comment: File /srv/user_pillar/top.sls exists with proper permissions. No changes made. Started: 23:35:28.719394 Duration: 5.403 ms Changes: ---------- ID: /srv/user_salt/locale Function: file.directory Result: True Comment: The directory /srv/user_salt/locale is in the correct state Started: 23:35:28.725191 Duration: 4.67 ms Changes: ---------- ID: /srv/user_salt/locale/init.sls Function: file.managed Result: True Comment: File /srv/user_salt/locale/init.sls exists with proper permissions. No changes made. Started: 23:35:28.730780 Duration: 4.454 ms Changes: Summary for local ------------ Succeeded: 7 (changed=1) Failed: 0 ------------ Total states run: 7 Total run time: 64.273 ms [user@dom0 ~]$ sudo qubesctl top.enable fed-min-qubes-dev pillarenv=user saltenv=user local: ---------- fed-min-qubes-dev.top: ---------- status: unchanged [user@dom0 ~]$ sudo qubesctl state.highstate [WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order. local: ---------- ID: directory_srv_user_salt Function: file.directory Name: /srv/user_salt Result: True Comment: The directory /srv/user_salt is in the correct state Started: 23:37:12.507977 Duration: 22.533 ms Changes: ---------- ID: directory_srv_user_pillar Function: file.directory Name: /srv/user_pillar Result: True Comment: The directory /srv/user_pillar is in the correct state Started: 23:37:12.530926 Duration: 5.021 ms Changes: ---------- ID: directory_srv_user_formulas Function: file.directory Name: /srv/user_formulas Result: True Comment: The directory /srv/user_formulas is in the correct state Started: 23:37:12.536434 Duration: 4.279 ms Changes: ---------- ID: /srv/user_salt/top.sls Function: file.managed Result: True Comment: File /srv/user_salt/top.sls exists with proper permissions. No changes made. Started: 23:37:12.542105 Duration: 5.844 ms Changes: ---------- ID: /srv/user_pillar/top.sls Function: file.managed Result: True Comment: File /srv/user_pillar/top.sls exists with proper permissions. No changes made. Started: 23:37:12.548920 Duration: 5.505 ms Changes: ---------- ID: /srv/user_salt/locale Function: file.directory Result: True Comment: The directory /srv/user_salt/locale is in the correct state Started: 23:37:12.554852 Duration: 5.135 ms Changes: ---------- ID: /srv/user_salt/locale/init.sls Function: file.managed Result: True Comment: File /srv/user_salt/locale/init.sls exists with proper permissions. No changes made. Started: 23:37:12.560989 Duration: 5.065 ms Changes: ---------- ID: ensure-fedora-38-minimal-installed Function: qvm.template_installed Name: fedora-38-minimal Result: True Comment: Template fedora-38-minimal version 4.2.0 already installed Started: 23:37:12.658814 Duration: 803.721 ms Changes: ---------- ID: create-fedora-38-builder Function: qvm.clone Name: fedora-38-builder Result: True Comment: [SKIP] A VM with the name 'fedora-38-builder' already exists. None Started: 23:37:13.463809 Duration: 1467.564 ms Changes: ---------- ID: create-qubes-builder-dvm Function: qvm.vm Name: qubes-builder-dvm Result: True Comment: ====== ['present'] ====== [SKIP] A VM with the name 'qubes-builder-dvm' already exists. ====== ['prefs'] ====== [SKIP] template_for_dispvms: True [SKIP] virt_mode : pvh [SKIP] label : red [SKIP] maxmem : 8000 [SKIP] memory : 800 [SKIP] vcpus : 6 Started: 23:37:14.937008 Duration: 1409.907 ms Changes: ---------- ID: create-qubes-work Function: qvm.vm Name: qubes-work Result: True Comment: ====== ['present'] ====== [SKIP] A VM with the name 'qubes-work' already exists. ====== ['prefs'] ====== [SKIP] default_dispvm : qubes-builder-dvm [SKIP] virt_mode : pvh [SKIP] label : red [SKIP] maxmem : 1000 [SKIP] memory : 400 Started: 23:37:16.351096 Duration: 1324.138 ms Changes: ---------- ID: resize-builder-volume Function: cmd.run Name: qvm-volume resize qubes-builder-dvm:private 32212254720 ;qvm-volume resize qubes-work:private 32212254720 Result: True Comment: Command "qvm-volume resize qubes-builder-dvm:private 32212254720 ;qvm-volume resize qubes-work:private 32212254720" run Started: 23:37:17.690604 Duration: 4129.296 ms Changes: ---------- pid: 13881 retcode: 0 stderr: stdout: ---------- ID: builder-policy Function: file.managed Name: /etc/qubes/policy.d/30-qubesbuilder.policy Result: True Comment: File /etc/qubes/policy.d/30-qubesbuilder.policy is in the correct state Started: 23:37:21.820492 Duration: 347.165 ms Changes: Summary for local ------------- Succeeded: 13 (changed=1) Failed: 0 ------------- Total states run: 13 Total run time: 9.535 s ```
tlaurion commented 7 months ago

Simulated a rerun since there is nothing that was installed in the template (packages needed) not git clone or other dependencies deployed and qubes-work is basically a fedora-38-minimal's dvm without any customization applied.

@Augsch123 Basically the following part is never applied:

{% elif grains['id'] == [fedoraver, '-builder']|join %}

update:
  cmd.run:
    - name: dnf install -y qubes-core-agent-networking qubes-core-agent-passwordless-root qubes-core-agent-thunar xfce4-terminal unzip wqy-microhei-fonts python3-packaging createrepo_c devscripts gpg python3-pyyaml rpm reprepro python3-pathspec mktorrent rpm-sign tree mock openssl python3-jinja2-cli m4 asciidoc rsync qubes-gpg-split dnf-plugins-core debootstrap dpkg-dev git mock pbuilder which perl-Digest-MD5 perl-Digest-SHA  python3-sh rpm-build rpmdevtools wget python3-debian python3-lxml systemd-udev python3-click qubes-gpg-split

{% elif grains['id'] == 'qubes-builder-dvm' %}

builderdirectory:
  cmd.run:
    - name: mkdir -p /rw/bind-dirs/builder /rw/config/qubes-bind-dirs.d ; echo "binds+=('/builder')" > /rw/config/qubes-bind-dirs.d/builder.conf ;

appendrc:
  file.append:
    - name: /rw/config/rc.local
    - text: mount /builder -o dev,suid,remount

{% elif grains['id'] == 'qubes-work' %}

git-fetch:
  cmd.run:
    - name: cd /home/user ; git clone https://github.com/QubesOS/qubes-infrastructure-mirrors ; git clone /https://github.com/QubesOS/qubes-builderv2
    - runas: user
install-infrastructure:
  cmd.run:
    - name: cd /home/user/qubes-infrastructure-mirrors ; python3 setup.py build ; python3 setup.py install

{% endif %}

As can be seen by the following replication trace.

Replication trace ``` [user@dom0 ~]$ qvm-remove qubes-work qubes-builder-dvm fedora-38-builder This will completely remove the selected VM(s)... qubes-work qubes-builder-dvm fedora-38-builder Are you sure? [y/N] y [user@dom0 ~]$ sudo qubesctl state.highstate [WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order. local: ---------- ID: directory_srv_user_salt Function: file.directory Name: /srv/user_salt Result: True Comment: The directory /srv/user_salt is in the correct state Started: 23:48:09.718120 Duration: 27.391 ms Changes: ---------- ID: directory_srv_user_pillar Function: file.directory Name: /srv/user_pillar Result: True Comment: The directory /srv/user_pillar is in the correct state Started: 23:48:09.746520 Duration: 8.4 ms Changes: ---------- ID: directory_srv_user_formulas Function: file.directory Name: /srv/user_formulas Result: True Comment: The directory /srv/user_formulas is in the correct state Started: 23:48:09.755448 Duration: 7.028 ms Changes: ---------- ID: /srv/user_salt/top.sls Function: file.managed Result: True Comment: File /srv/user_salt/top.sls exists with proper permissions. No changes made. Started: 23:48:09.763764 Duration: 4.848 ms Changes: ---------- ID: /srv/user_pillar/top.sls Function: file.managed Result: True Comment: File /srv/user_pillar/top.sls exists with proper permissions. No changes made. Started: 23:48:09.769609 Duration: 6.421 ms Changes: ---------- ID: /srv/user_salt/locale Function: file.directory Result: True Comment: The directory /srv/user_salt/locale is in the correct state Started: 23:48:09.776534 Duration: 6.966 ms Changes: ---------- ID: /srv/user_salt/locale/init.sls Function: file.managed Result: True Comment: File /srv/user_salt/locale/init.sls exists with proper permissions. No changes made. Started: 23:48:09.784758 Duration: 7.806 ms Changes: ---------- ID: ensure-fedora-38-minimal-installed Function: qvm.template_installed Name: fedora-38-minimal Result: True Comment: Template fedora-38-minimal version 4.2.0 already installed Started: 23:48:09.906612 Duration: 901.826 ms Changes: ---------- ID: create-fedora-38-builder Function: qvm.clone Name: fedora-38-builder Result: True Comment: /usr/bin/qvm-check fedora-38-builder None Started: 23:48:10.809752 Duration: 75730.856 ms Changes: ---------- ID: create-qubes-builder-dvm Function: qvm.vm Name: qubes-builder-dvm Result: True Comment: ====== ['present'] ====== /usr/bin/qvm-create qubes-builder-dvm --class=AppVM --template=fedora-38-builder --label=red ====== ['prefs'] ====== [SKIP] label : red Started: 23:49:26.546246 Duration: 15199.906 ms Changes: ---------- qvm.prefs: ---------- qvm.create: ---------- maxmem: ---------- new: 8000 old: *default* memory: ---------- new: 800 old: *default* template_for_dispvms: ---------- new: True old: *default* vcpus: ---------- new: 6 old: *default* virt_mode: ---------- new: pvh old: *default* ---------- ID: create-qubes-work Function: qvm.vm Name: qubes-work Result: True Comment: ====== ['present'] ====== /usr/bin/qvm-create qubes-work --class=AppVM --template=fedora-38-builder --label=red ====== ['prefs'] ====== [SKIP] label : red Started: 23:49:41.751467 Duration: 13517.488 ms Changes: ---------- qvm.prefs: ---------- qvm.create: ---------- default_dispvm: ---------- new: qubes-builder-dvm old: *default* maxmem: ---------- new: 1000 old: *default* memory: ---------- new: 400 old: *default* virt_mode: ---------- new: pvh old: *default* ---------- ID: resize-builder-volume Function: cmd.run Name: qvm-volume resize qubes-builder-dvm:private 32212254720 ;qvm-volume resize qubes-work:private 32212254720 Result: True Comment: Command "qvm-volume resize qubes-builder-dvm:private 32212254720 ;qvm-volume resize qubes-work:private 32212254720" run Started: 23:49:55.285884 Duration: 2933.93 ms Changes: ---------- pid: 15454 retcode: 0 stderr: stdout: ---------- ID: builder-policy Function: file.managed Name: /etc/qubes/policy.d/30-qubesbuilder.policy Result: True Comment: File /etc/qubes/policy.d/30-qubesbuilder.policy is in the correct state Started: 23:49:58.220385 Duration: 358.991 ms Changes: Summary for local ------------- Succeeded: 13 (changed=3) Failed: 0 ------------- Total states run: 13 Total run time: 108.712 s ```
ben-grande commented 7 months ago

I also have a Salt formula for Qubes Builder V2.

Please read the warning first, it is true, I will do breaking changes while it is in alpha and I won't be able to debug if this occurs or any other issue, I am simply sharing my alternative. Provided as is, without any warranty of any kind to the extent permitted by law.

You may try the formula. Notice it is a work in progress and the policy allows calls to sys-pgp, so you have to set isolate_home_dirs in the split-gpg2 configuration for security reasons (waiting for https://github.com/QubesOS/qubes-issues/issues/8792 to package this part).

Notice you have to copy the whole repository to the salt path because it references other states that are outside the qubes-builder directory.

Follow the installation steps.

Augsch123 commented 7 months ago

Basically the following part is never applied:

Have you tried rerun sudo qubesctl state.highstate while not deleting fedora-38-builder? It was supposed to work in one go but I'm unsure what broke it.

tlaurion commented 7 months ago

Basically the following part is never applied:

Have you tried rerun sudo qubesctl state.highstate while not deleting fedora-38-builder? It was supposed to work in one go but I'm unsure what broke it.

Replication trace ``` [user@dom0 ~]$ sudo qubes-dom0-update --action=search salt-base-config Using sys-whonix as UpdateVM to download updates for Dom0; this may take some time... Unable to detect release version (use '--releasever' to specify release version) Last metadata expiration check: 11:22:40 ago on Sun Jan 14 05:17:08 2024. ======================== Name Matched: salt-base-config ======================== qubes-mgmt-salt-base-config.noarch : Qubes+Salt Management base configuration for SaltStack's Salt Infrastructure automation and management system qubes-mgmt-salt-base-config.src : Qubes+Salt Management base configuration for SaltStack's Salt Infrastructure automation and management system No packages downloaded [user@dom0 ~]$ sudo qubes-dom0-update --action=reinstall qubes-mgmt-salt-base-config Using sys-whonix as UpdateVM to download updates for Dom0; this may take some time... Unable to detect release version (use '--releasever' to specify release version) Fedora 37 - x86_64 16 kB/s | 22 kB 00:01 Fedora 37 - x86_64 - Updates 17 kB/s | 21 kB 00:01 Qubes Host Repository (updates) 2.8 kB/s | 2.7 kB 00:00 Dependencies resolved. ================================================================================ Package Arch Version Repository Size ================================================================================ Reinstalling: qubes-mgmt-salt-base-config noarch 4.1.2-1.fc37 qubes-dom0-current 11 k Transaction Summary ================================================================================ Total download size: 11 k Installed size: 8.5 k DNF will only download packages for the transaction. Downloading Packages: qubes-mgmt-salt-base-config-4.1.2-1.fc37.noarch 3.7 kB/s | 11 kB 00:03 -------------------------------------------------------------------------------- Total 2.8 kB/s | 11 kB 00:04 Complete! The downloaded packages were saved in cache until the next successful transaction. You can remove cached packages by executing 'dnf clean packages'. Qubes OS Repository for Dom0 2.9 MB/s | 3.0 kB 00:00 Qubes OS Repository for Dom0 130 kB/s | 1.1 kB 00:00 Dependencies resolved. ============================================================================================================================================================================================== Package Architecture Version Repository Size ============================================================================================================================================================================================== Reinstalling: qubes-mgmt-salt-base-config noarch 4.1.2-1.fc37 qubes-dom0-cached 8.0 k Transaction Summary ============================================================================================================================================================================================== Total size: 8.0 k Installed size: 8.5 k Is this ok [y/N]: y Downloading Packages: Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Reinstalling : qubes-mgmt-salt-base-config-4.1.2-1.fc37.noarch 1/2 Running scriptlet: qubes-mgmt-salt-base-config-4.1.2-1.fc37.noarch 1/2 Cleanup : qubes-mgmt-salt-base-config-4.1.2-1.fc37.noarch 2/2 Verifying : qubes-mgmt-salt-base-config-4.1.2-1.fc37.noarch 1/2 Verifying : qubes-mgmt-salt-base-config-4.1.2-1.fc37.noarch 2/2 Reinstalled: qubes-mgmt-salt-base-config-4.1.2-1.fc37.noarch Complete! [user@dom0 ~]$ sudo rm /srv/salt/_tops/base/user-dirs.top [user@dom0 ~]$ sudo ln -s /srv/salt/qubes/user-dirs.top /srv/salt/_tops/base/user-dirs.top [user@dom0 ~]$ qubesctl top.enabled [WARNING ] Failed to open log file, do you have permission to write to /var/log/salt/minion? [CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable ^C Exiting gracefully on Ctrl-c ^[[A [user@dom0 ~]$ sudo qubesctl top.enabled [CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable 'top.enabled' is not available. DOM0 configuration failed, not continuing (failed reverse-i-search)`': ^Cear [user@dom0 ~]$ ^C [user@dom0 ~]$ ^C [user@dom0 ~]$ sudo rm -rf /srv/user_salt/fed-min-qubes-dev.* [user@dom0 ~]$ sudo qubesctl state.sls qubes.user-dirs [CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable [CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable local: ---------- ID: directory_srv_user_salt Function: file.directory Name: /srv/user_salt Result: True Comment: The directory /srv/user_salt is in the correct state Started: 11:42:09.475355 Duration: 4.91 ms Changes: ---------- ID: directory_srv_user_pillar Function: file.directory Name: /srv/user_pillar Result: True Comment: The directory /srv/user_pillar is in the correct state Started: 11:42:09.480438 Duration: 1.531 ms Changes: ---------- ID: directory_srv_user_formulas Function: file.directory Name: /srv/user_formulas Result: True Comment: The directory /srv/user_formulas is in the correct state Started: 11:42:09.482094 Duration: 1.238 ms Changes: ---------- ID: /srv/user_salt/top.sls Function: file.managed Result: True Comment: File /srv/user_salt/top.sls exists with proper permissions. No changes made. Started: 11:42:09.483653 Duration: 1.548 ms Changes: ---------- ID: /srv/user_pillar/top.sls Function: file.managed Result: True Comment: File /srv/user_pillar/top.sls exists with proper permissions. No changes made. Started: 11:42:09.485547 Duration: 1.687 ms Changes: ---------- ID: /srv/user_salt/locale Function: file.directory Result: True Comment: The directory /srv/user_salt/locale is in the correct state Started: 11:42:09.487373 Duration: 1.49 ms Changes: ---------- ID: /srv/user_salt/locale/init.sls Function: file.managed Result: True Comment: File /srv/user_salt/locale/init.sls exists with proper permissions. No changes made. Started: 11:42:09.489152 Duration: 1.461 ms Changes: Summary for local ------------ Succeeded: 7 Failed: 0 ------------ Total states run: 7 Total run time: 13.865 ms [user@dom0 ~]$ sudo qubesctl saltutil.sync_all [CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable [WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order. local: ---------- beacons: clouds: engines: executors: grains: - grains.boot_mode - grains.pci_devs - grains.redefined_dom0_grains - grains.whonix log_handlers: matchers: modules: - modules.debug - modules.ext_module_qvm - modules.module_utils - modules.qubes - modules.qubes_dom0_update - modules.topd output: pillar: - pillar.qvm_prefs proxymodules: renderers: returners: sdb: serializers: states: - states.debug - states.ext_state_qvm - states.status thorium: utils: - utils.__init__ - utils.fileinfo - utils.matcher - utils.nulltype - utils.pathinfo - utils.pathutils - utils.qubes_utils - utils.toputils [user@dom0 ~]$ sudo qubesctl state.sls qubes.user-dirs local: ---------- ID: directory_srv_user_salt Function: file.directory Name: /srv/user_salt Result: True Comment: The directory /srv/user_salt is in the correct state Started: 11:42:32.073260 Duration: 8.32 ms Changes: ---------- ID: directory_srv_user_pillar Function: file.directory Name: /srv/user_pillar Result: True Comment: The directory /srv/user_pillar is in the correct state Started: 11:42:32.081741 Duration: 1.579 ms Changes: ---------- ID: directory_srv_user_formulas Function: file.directory Name: /srv/user_formulas Result: True Comment: The directory /srv/user_formulas is in the correct state Started: 11:42:32.083536 Duration: 1.536 ms Changes: ---------- ID: /srv/user_salt/top.sls Function: file.managed Result: True Comment: File /srv/user_salt/top.sls exists with proper permissions. No changes made. Started: 11:42:32.085379 Duration: 1.44 ms Changes: ---------- ID: /srv/user_pillar/top.sls Function: file.managed Result: True Comment: File /srv/user_pillar/top.sls exists with proper permissions. No changes made. Started: 11:42:32.087093 Duration: 1.531 ms Changes: ---------- ID: /srv/user_salt/locale Function: file.directory Result: True Comment: The directory /srv/user_salt/locale is in the correct state Started: 11:42:32.088765 Duration: 1.573 ms Changes: ---------- ID: /srv/user_salt/locale/init.sls Function: file.managed Result: True Comment: File /srv/user_salt/locale/init.sls exists with proper permissions. No changes made. Started: 11:42:32.090667 Duration: 1.605 ms Changes: Summary for local ------------ Succeeded: 7 Failed: 0 ------------ Total states run: 7 Total run time: 17.584 ms [user@dom0 ~]$ qubesctl top.enabled^C [user@dom0 ~]$ sudo cp ~/Downloads/qubes-builder-v2-salt/srv/user_salt/fed-min-qubes-dev.* /srv/user_salt/ [user@dom0 ~]$ sudo qubesctl saltutil.sync_all [WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order. local: ---------- beacons: clouds: engines: executors: grains: log_handlers: matchers: modules: output: pillar: proxymodules: renderers: returners: sdb: serializers: states: thorium: utils: [user@dom0 ~]$ sudo qubesctl top.enable fed-min-qubes-dev pillarenv=user saltenv=user local: ---------- fed-min-qubes-dev.top: ---------- status: unchanged [user@dom0 ~]$ sudo qubesctl state.highstate [WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order. local: ---------- ID: directory_srv_user_salt Function: file.directory Name: /srv/user_salt Result: True Comment: The directory /srv/user_salt is in the correct state Started: 11:43:19.729867 Duration: 8.163 ms Changes: ---------- ID: directory_srv_user_pillar Function: file.directory Name: /srv/user_pillar Result: True Comment: The directory /srv/user_pillar is in the correct state Started: 11:43:19.738194 Duration: 1.692 ms Changes: ---------- ID: directory_srv_user_formulas Function: file.directory Name: /srv/user_formulas Result: True Comment: The directory /srv/user_formulas is in the correct state Started: 11:43:19.740024 Duration: 1.304 ms Changes: ---------- ID: /srv/user_salt/top.sls Function: file.managed Result: True Comment: File /srv/user_salt/top.sls exists with proper permissions. No changes made. Started: 11:43:19.741667 Duration: 1.442 ms Changes: ---------- ID: /srv/user_pillar/top.sls Function: file.managed Result: True Comment: File /srv/user_pillar/top.sls exists with proper permissions. No changes made. Started: 11:43:19.743389 Duration: 1.586 ms Changes: ---------- ID: /srv/user_salt/locale Function: file.directory Result: True Comment: The directory /srv/user_salt/locale is in the correct state Started: 11:43:19.745103 Duration: 1.467 ms Changes: ---------- ID: /srv/user_salt/locale/init.sls Function: file.managed Result: True Comment: File /srv/user_salt/locale/init.sls exists with proper permissions. No changes made. Started: 11:43:19.746850 Duration: 1.456 ms Changes: ---------- ID: ensure-fedora-38-minimal-installed Function: qvm.template_installed Name: fedora-38-minimal Result: True Comment: Template fedora-38-minimal version 4.2.0 already installed Started: 11:43:19.780676 Duration: 245.082 ms Changes: ---------- ID: create-fedora-38-builder Function: qvm.clone Name: fedora-38-builder Result: True Comment: /usr/bin/qvm-check fedora-38-builder None Started: 11:43:20.026235 Duration: 34375.304 ms Changes: ---------- ID: create-qubes-builder-dvm Function: qvm.vm Name: qubes-builder-dvm Result: True Comment: ====== ['present'] ====== /usr/bin/qvm-create qubes-builder-dvm --class=AppVM --template=fedora-38-builder --label=red ====== ['prefs'] ====== [SKIP] label : red Started: 11:43:54.404600 Duration: 6131.348 ms Changes: ---------- qvm.prefs: ---------- qvm.create: ---------- maxmem: ---------- new: 8000 old: *default* memory: ---------- new: 800 old: *default* template_for_dispvms: ---------- new: True old: *default* vcpus: ---------- new: 6 old: *default* virt_mode: ---------- new: pvh old: *default* ---------- ID: create-qubes-work Function: qvm.vm Name: qubes-work Result: True Comment: ====== ['present'] ====== /usr/bin/qvm-create qubes-work --class=AppVM --template=fedora-38-builder --label=red ====== ['prefs'] ====== [SKIP] label : red Started: 11:44:00.537573 Duration: 4348.516 ms Changes: ---------- qvm.prefs: ---------- qvm.create: ---------- default_dispvm: ---------- new: qubes-builder-dvm old: *default* maxmem: ---------- new: 1000 old: *default* memory: ---------- new: 400 old: *default* virt_mode: ---------- new: pvh old: *default* ---------- ID: resize-builder-volume Function: cmd.run Name: qvm-volume resize qubes-builder-dvm:private 32212254720 ;qvm-volume resize qubes-work:private 32212254720 Result: True Comment: Command "qvm-volume resize qubes-builder-dvm:private 32212254720 ;qvm-volume resize qubes-work:private 32212254720" run Started: 11:44:04.890373 Duration: 1072.971 ms Changes: ---------- pid: 20701 retcode: 0 stderr: stdout: ---------- ID: builder-policy Function: file.managed Name: /etc/qubes/policy.d/30-qubesbuilder.policy Result: True Comment: File /etc/qubes/policy.d/30-qubesbuilder.policy is in the correct state Started: 11:44:05.963623 Duration: 129.732 ms Changes: Summary for local ------------- Succeeded: 13 (changed=3) Failed: 0 ------------- Total states run: 13 Total run time: 46.320 s [user@dom0 ~]$ sudo qubesctl state.highstate [WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order. local: ---------- ID: directory_srv_user_salt Function: file.directory Name: /srv/user_salt Result: True Comment: The directory /srv/user_salt is in the correct state Started: 11:46:07.352948 Duration: 8.645 ms Changes: ---------- ID: directory_srv_user_pillar Function: file.directory Name: /srv/user_pillar Result: True Comment: The directory /srv/user_pillar is in the correct state Started: 11:46:07.361761 Duration: 1.598 ms Changes: ---------- ID: directory_srv_user_formulas Function: file.directory Name: /srv/user_formulas Result: True Comment: The directory /srv/user_formulas is in the correct state Started: 11:46:07.363548 Duration: 1.373 ms Changes: ---------- ID: /srv/user_salt/top.sls Function: file.managed Result: True Comment: File /srv/user_salt/top.sls exists with proper permissions. No changes made. Started: 11:46:07.365252 Duration: 1.534 ms Changes: ---------- ID: /srv/user_pillar/top.sls Function: file.managed Result: True Comment: File /srv/user_pillar/top.sls exists with proper permissions. No changes made. Started: 11:46:07.367092 Duration: 1.496 ms Changes: ---------- ID: /srv/user_salt/locale Function: file.directory Result: True Comment: The directory /srv/user_salt/locale is in the correct state Started: 11:46:07.368726 Duration: 1.546 ms Changes: ---------- ID: /srv/user_salt/locale/init.sls Function: file.managed Result: True Comment: File /srv/user_salt/locale/init.sls exists with proper permissions. No changes made. Started: 11:46:07.370613 Duration: 1.722 ms Changes: ---------- ID: ensure-fedora-38-minimal-installed Function: qvm.template_installed Name: fedora-38-minimal Result: True Comment: Template fedora-38-minimal version 4.2.0 already installed Started: 11:46:07.404521 Duration: 244.081 ms Changes: ---------- ID: create-fedora-38-builder Function: qvm.clone Name: fedora-38-builder Result: True Comment: [SKIP] A VM with the name 'fedora-38-builder' already exists. None Started: 11:46:07.649021 Duration: 449.352 ms Changes: ---------- ID: create-qubes-builder-dvm Function: qvm.vm Name: qubes-builder-dvm Result: True Comment: ====== ['present'] ====== [SKIP] A VM with the name 'qubes-builder-dvm' already exists. ====== ['prefs'] ====== [SKIP] template_for_dispvms: True [SKIP] virt_mode : pvh [SKIP] label : red [SKIP] maxmem : 8000 [SKIP] memory : 800 [SKIP] vcpus : 6 Started: 11:46:08.100132 Duration: 396.898 ms Changes: ---------- ID: create-qubes-work Function: qvm.vm Name: qubes-work Result: True Comment: ====== ['present'] ====== [SKIP] A VM with the name 'qubes-work' already exists. ====== ['prefs'] ====== [SKIP] default_dispvm : qubes-builder-dvm [SKIP] virt_mode : pvh [SKIP] label : red [SKIP] maxmem : 1000 [SKIP] memory : 400 Started: 11:46:08.498304 Duration: 393.888 ms Changes: ---------- ID: resize-builder-volume Function: cmd.run Name: qvm-volume resize qubes-builder-dvm:private 32212254720 ;qvm-volume resize qubes-work:private 32212254720 Result: True Comment: Command "qvm-volume resize qubes-builder-dvm:private 32212254720 ;qvm-volume resize qubes-work:private 32212254720" run Started: 11:46:08.897372 Duration: 919.378 ms Changes: ---------- pid: 20913 retcode: 0 stderr: stdout: ---------- ID: builder-policy Function: file.managed Name: /etc/qubes/policy.d/30-qubesbuilder.policy Result: True Comment: File /etc/qubes/policy.d/30-qubesbuilder.policy is in the correct state Started: 11:46:09.816974 Duration: 112.398 ms Changes: Summary for local ------------- Succeeded: 13 (changed=1) Failed: 0 ------------- Total states run: 13 Total run time: 2.534 s [user@dom0 ~]$ qvm-remove qubes-work qubes-builder-dvm This will completely remove the selected VM(s)... qubes-work qubes-builder-dvm Are you sure? [y/N] y [user@dom0 ~]$ sudo qubesctl state.highstate [WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order. local: ---------- ID: directory_srv_user_salt Function: file.directory Name: /srv/user_salt Result: True Comment: The directory /srv/user_salt is in the correct state Started: 11:46:43.148069 Duration: 9.289 ms Changes: ---------- ID: directory_srv_user_pillar Function: file.directory Name: /srv/user_pillar Result: True Comment: The directory /srv/user_pillar is in the correct state Started: 11:46:43.157559 Duration: 1.542 ms Changes: ---------- ID: directory_srv_user_formulas Function: file.directory Name: /srv/user_formulas Result: True Comment: The directory /srv/user_formulas is in the correct state Started: 11:46:43.159275 Duration: 1.372 ms Changes: ---------- ID: /srv/user_salt/top.sls Function: file.managed Result: True Comment: File /srv/user_salt/top.sls exists with proper permissions. No changes made. Started: 11:46:43.160975 Duration: 1.492 ms Changes: ---------- ID: /srv/user_pillar/top.sls Function: file.managed Result: True Comment: File /srv/user_pillar/top.sls exists with proper permissions. No changes made. Started: 11:46:43.162767 Duration: 1.498 ms Changes: ---------- ID: /srv/user_salt/locale Function: file.directory Result: True Comment: The directory /srv/user_salt/locale is in the correct state Started: 11:46:43.164460 Duration: 1.541 ms Changes: ---------- ID: /srv/user_salt/locale/init.sls Function: file.managed Result: True Comment: File /srv/user_salt/locale/init.sls exists with proper permissions. No changes made. Started: 11:46:43.166309 Duration: 1.485 ms Changes: ---------- ID: ensure-fedora-38-minimal-installed Function: qvm.template_installed Name: fedora-38-minimal Result: True Comment: Template fedora-38-minimal version 4.2.0 already installed Started: 11:46:43.200305 Duration: 353.189 ms Changes: ---------- ID: create-fedora-38-builder Function: qvm.clone Name: fedora-38-builder Result: True Comment: [SKIP] A VM with the name 'fedora-38-builder' already exists. None Started: 11:46:43.554045 Duration: 436.957 ms Changes: ---------- ID: create-qubes-builder-dvm Function: qvm.vm Name: qubes-builder-dvm Result: True Comment: ====== ['present'] ====== /usr/bin/qvm-create qubes-builder-dvm --class=AppVM --template=fedora-38-builder --label=red ====== ['prefs'] ====== [SKIP] label : red Started: 11:46:43.992799 Duration: 4804.425 ms Changes: ---------- qvm.prefs: ---------- qvm.create: ---------- maxmem: ---------- new: 8000 old: *default* memory: ---------- new: 800 old: *default* template_for_dispvms: ---------- new: True old: *default* vcpus: ---------- new: 6 old: *default* virt_mode: ---------- new: pvh old: *default* ---------- ID: create-qubes-work Function: qvm.vm Name: qubes-work Result: True Comment: ====== ['present'] ====== /usr/bin/qvm-create qubes-work --class=AppVM --template=fedora-38-builder --label=red ====== ['prefs'] ====== [SKIP] label : red Started: 11:46:48.798601 Duration: 4163.979 ms Changes: ---------- qvm.prefs: ---------- qvm.create: ---------- default_dispvm: ---------- new: qubes-builder-dvm old: *default* maxmem: ---------- new: 1000 old: *default* memory: ---------- new: 400 old: *default* virt_mode: ---------- new: pvh old: *default* ---------- ID: resize-builder-volume Function: cmd.run Name: qvm-volume resize qubes-builder-dvm:private 32212254720 ;qvm-volume resize qubes-work:private 32212254720 Result: True Comment: Command "qvm-volume resize qubes-builder-dvm:private 32212254720 ;qvm-volume resize qubes-work:private 32212254720" run Started: 11:46:52.966964 Duration: 1002.94 ms Changes: ---------- pid: 21777 retcode: 0 stderr: stdout: ---------- ID: builder-policy Function: file.managed Name: /etc/qubes/policy.d/30-qubesbuilder.policy Result: True Comment: File /etc/qubes/policy.d/30-qubesbuilder.policy is in the correct state Started: 11:46:53.970140 Duration: 114.066 ms Changes: Summary for local ------------- Succeeded: 13 (changed=3) Failed: 0 ------------- Total states run: 13 Total run time: 10.894 s [user@dom0 ~]$ ```

Note that

tlaurion commented 7 months ago

This aims to test https://github.com/QubesOS/qubes-mgmt-salt-base-config/pull/11.

https://github.com/QubesOS/qubes-mgmt-salt-base-config/pull/11 works and replaces manual ln -s needed otherwise as explained under https://github.com/QubesOS/qubes-issues/issues/8491#issuecomment-1890843137.

The rest is the same:

Replication trace ``` [user@dom0 ~]$ qvm-remove qubes-work qubes-builder-dvm fedora-38-builder This will completely remove the selected VM(s)... qubes-work qubes-builder-dvm fedora-38-builder Are you sure? [y/N] y [user@dom0 ~]$ sudo rm /srv/salt/_tops/base/user-dirs.top [user@dom0 ~]$ qvm-run --pass-io heads-tests-deb12 "cat ~/Downloads/user-dirs_47cdad6a06e9ce43dfe8cc0fc657c689d61698d2.sls" | sudo tee /srv/salt/qubes/user-dirs.sls # -*- coding: utf-8 -*- # vim: set syntax=yaml ts=2 sw=2 sts=2 et : ## # qubes.user-dirs # =============== # # Install and maintain user salt and pillar directories for personal state # configurations: # # Includes a simple locale state file # # User defined scripts will not be removed on removal of qubes-mgmt-salt # by design nor will they be modified on any updates, other than permissions # being enforced. # # Execute: # -------- # qubesctl state.sls qubes.user-dirs # # Note: # Using using custom ID's to prevent possible conflicts ## directory_srv_user_salt: file.directory: - name: /srv/user_salt - user: root - group: root - dir_mode: 750 - file_mode: 640 - recurse: - user - group - mode # User 'pillar' directory and file permissions # Note: using custom ID due to possible conflicts directory_srv_user_pillar: file.directory: - name: /srv/user_pillar - user: root - group: root - dir_mode: 750 - file_mode: 640 - recurse: - user - group - mode enable_user_salt: file.symlink: - name: /srv/salt/_tops/base/user-dirs.top - target: ../../qubes/user-dirs.top # User 'formulas' directory and file permissions # Note: using custom ID due to possible conflicts directory_srv_user_formulas: file.directory: - name: /srv/user_formulas - user: root - group: root - dir_mode: 750 - file_mode: 640 - recurse: - user - group - mode # User 'state' top.sls file /srv/user_salt/top.sls: file.managed: - source: salt://qubes/files/top.sls - replace: False - makedirs: True - user: root - group: root - mode: 640 - require: - file: /srv/user_salt # User 'pillar' top.sls file /srv/user_pillar/top.sls: file.managed: - name: - source: salt://qubes/files/pillar.sls - replace: False - makedirs: True - user: root - group: root - mode: 640 - require: - file: /srv/user_pillar # Sample locale state directory /srv/user_salt/locale: file.directory: - user: root - group: root - dir_mode: 750 - file_mode: 640 - recurse: - user - group - mode # Sample locale state file /srv/user_salt/locale/init.sls: file.managed: - source: salt://qubes/files/locale.sls - replace: False - makedirs: True - user: root - group: root - mode: 640 - require: - file: /srv/user_salt/locale [user@dom0 ~]$ sudo rm -rf /srv/user_salt/ [user@dom0 ~]$ sudo qubesctl saltutil.sync_all local: ---------- beacons: clouds: engines: executors: grains: log_handlers: matchers: modules: output: pillar: proxymodules: renderers: returners: sdb: serializers: states: thorium: utils: [user@dom0 ~]$ sudo qubesctl state.sls qubes.user-dirs local: ---------- ID: directory_srv_user_salt Function: file.directory Name: /srv/user_salt Result: True Comment: Started: 11:58:05.933208 Duration: 8.446 ms Changes: ---------- /srv/user_salt: ---------- directory: new ---------- ID: directory_srv_user_pillar Function: file.directory Name: /srv/user_pillar Result: True Comment: The directory /srv/user_pillar is in the correct state Started: 11:58:05.941814 Duration: 1.897 ms Changes: ---------- ID: enable_user_salt Function: file.symlink Name: /srv/salt/_tops/base/user-dirs.top Result: True Comment: Created new symlink /srv/salt/_tops/base/user-dirs.top -> ../../qubes/user-dirs.top Started: 11:58:05.943854 Duration: 10.165 ms Changes: ---------- new: /srv/salt/_tops/base/user-dirs.top ---------- ID: directory_srv_user_formulas Function: file.directory Name: /srv/user_formulas Result: True Comment: The directory /srv/user_formulas is in the correct state Started: 11:58:05.954191 Duration: 1.884 ms Changes: ---------- ID: /srv/user_salt/top.sls Function: file.managed Result: True Comment: File /srv/user_salt/top.sls updated Started: 11:58:05.956573 Duration: 92.267 ms Changes: ---------- diff: New file mode: 0640 ---------- ID: /srv/user_pillar/top.sls Function: file.managed Result: True Comment: File /srv/user_pillar/top.sls exists with proper permissions. No changes made. Started: 11:58:06.049205 Duration: 1.631 ms Changes: ---------- ID: /srv/user_salt/locale Function: file.directory Result: True Comment: Started: 11:58:06.050982 Duration: 1.866 ms Changes: ---------- /srv/user_salt/locale: ---------- directory: new ---------- ID: /srv/user_salt/locale/init.sls Function: file.managed Result: True Comment: File /srv/user_salt/locale/init.sls updated Started: 11:58:06.053196 Duration: 86.66 ms Changes: ---------- diff: New file mode: 0640 Summary for local ------------ Succeeded: 8 (changed=5) Failed: 0 ------------ Total states run: 8 Total run time: 204.816 ms [user@dom0 ~]$ sudo cp ~/Downloads/qubes-builder-v2-salt/srv/user_salt/fed-min-qubes-dev.* /srv/user_salt/ [user@dom0 ~]$ sudo qubesctl state.sls qubes.user-dirs local: ---------- ID: directory_srv_user_salt Function: file.directory Name: /srv/user_salt Result: True Comment: Directory /srv/user_salt updated Started: 11:58:33.450524 Duration: 12.998 ms Changes: ---------- /srv/user_salt/fed-min-qubes-dev.sls: ---------- mode: 0640 /srv/user_salt/fed-min-qubes-dev.top: ---------- mode: 0640 mode: 0640 ---------- ID: directory_srv_user_pillar Function: file.directory Name: /srv/user_pillar Result: True Comment: The directory /srv/user_pillar is in the correct state Started: 11:58:33.463696 Duration: 1.639 ms Changes: ---------- ID: enable_user_salt Function: file.symlink Name: /srv/salt/_tops/base/user-dirs.top Result: True Comment: Symlink /srv/salt/_tops/base/user-dirs.top is present and owned by root:root Started: 11:58:33.465499 Duration: 10.273 ms Changes: ---------- ID: directory_srv_user_formulas Function: file.directory Name: /srv/user_formulas Result: True Comment: The directory /srv/user_formulas is in the correct state Started: 11:58:33.475934 Duration: 1.633 ms Changes: ---------- ID: /srv/user_salt/top.sls Function: file.managed Result: True Comment: File /srv/user_salt/top.sls exists with proper permissions. No changes made. Started: 11:58:33.477902 Duration: 7.343 ms Changes: ---------- ID: /srv/user_pillar/top.sls Function: file.managed Result: True Comment: File /srv/user_pillar/top.sls exists with proper permissions. No changes made. Started: 11:58:33.485661 Duration: 2.562 ms Changes: ---------- ID: /srv/user_salt/locale Function: file.directory Result: True Comment: The directory /srv/user_salt/locale is in the correct state Started: 11:58:33.488359 Duration: 1.535 ms Changes: ---------- ID: /srv/user_salt/locale/init.sls Function: file.managed Result: True Comment: File /srv/user_salt/locale/init.sls exists with proper permissions. No changes made. Started: 11:58:33.490707 Duration: 1.562 ms Changes: Summary for local ------------ Succeeded: 8 (changed=1) Failed: 0 ------------ Total states run: 8 Total run time: 39.545 ms [user@dom0 ~]$ sudo qubesctl saltutil.sync_all [WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order. local: ---------- beacons: clouds: engines: executors: grains: log_handlers: matchers: modules: output: pillar: proxymodules: renderers: returners: sdb: serializers: states: thorium: utils: [user@dom0 ~]$ sudo qubesctl top.enable fed-min-qubes-dev pillarenv=user saltenv=user local: ---------- fed-min-qubes-dev.top: ---------- status: unchanged [user@dom0 ~]$ sudo qubesctl state.highstate [WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order. local: ---------- ID: directory_srv_user_salt Function: file.directory Name: /srv/user_salt Result: True Comment: The directory /srv/user_salt is in the correct state Started: 11:59:23.261381 Duration: 10.333 ms Changes: ---------- ID: directory_srv_user_pillar Function: file.directory Name: /srv/user_pillar Result: True Comment: The directory /srv/user_pillar is in the correct state Started: 11:59:23.271882 Duration: 1.563 ms Changes: ---------- ID: enable_user_salt Function: file.symlink Name: /srv/salt/_tops/base/user-dirs.top Result: True Comment: Symlink /srv/salt/_tops/base/user-dirs.top is present and owned by root:root Started: 11:59:23.273586 Duration: 8.912 ms Changes: ---------- ID: directory_srv_user_formulas Function: file.directory Name: /srv/user_formulas Result: True Comment: The directory /srv/user_formulas is in the correct state Started: 11:59:23.282672 Duration: 1.471 ms Changes: ---------- ID: /srv/user_salt/top.sls Function: file.managed Result: True Comment: File /srv/user_salt/top.sls exists with proper permissions. No changes made. Started: 11:59:23.284523 Duration: 1.531 ms Changes: ---------- ID: /srv/user_pillar/top.sls Function: file.managed Result: True Comment: File /srv/user_pillar/top.sls exists with proper permissions. No changes made. Started: 11:59:23.286372 Duration: 1.884 ms Changes: ---------- ID: /srv/user_salt/locale Function: file.directory Result: True Comment: The directory /srv/user_salt/locale is in the correct state Started: 11:59:23.288434 Duration: 1.579 ms Changes: ---------- ID: /srv/user_salt/locale/init.sls Function: file.managed Result: True Comment: File /srv/user_salt/locale/init.sls exists with proper permissions. No changes made. Started: 11:59:23.290364 Duration: 1.525 ms Changes: ---------- ID: ensure-fedora-38-minimal-installed Function: qvm.template_installed Name: fedora-38-minimal Result: True Comment: Template fedora-38-minimal version 4.2.0 already installed Started: 11:59:23.325225 Duration: 245.271 ms Changes: ---------- ID: create-fedora-38-builder Function: qvm.clone Name: fedora-38-builder Result: True Comment: /usr/bin/qvm-check fedora-38-builder None Started: 11:59:23.570914 Duration: 24587.834 ms Changes: ---------- ID: create-qubes-builder-dvm Function: qvm.vm Name: qubes-builder-dvm Result: True Comment: ====== ['present'] ====== /usr/bin/qvm-create qubes-builder-dvm --class=AppVM --template=fedora-38-builder --label=red ====== ['prefs'] ====== [SKIP] label : red Started: 11:59:48.160670 Duration: 4645.926 ms Changes: ---------- qvm.prefs: ---------- qvm.create: ---------- maxmem: ---------- new: 8000 old: *default* memory: ---------- new: 800 old: *default* template_for_dispvms: ---------- new: True old: *default* vcpus: ---------- new: 6 old: *default* virt_mode: ---------- new: pvh old: *default* ---------- ID: create-qubes-work Function: qvm.vm Name: qubes-work Result: True Comment: ====== ['present'] ====== /usr/bin/qvm-create qubes-work --class=AppVM --template=fedora-38-builder --label=red ====== ['prefs'] ====== [SKIP] label : red Started: 11:59:52.808124 Duration: 4652.354 ms Changes: ---------- qvm.prefs: ---------- qvm.create: ---------- default_dispvm: ---------- new: qubes-builder-dvm old: *default* maxmem: ---------- new: 1000 old: *default* memory: ---------- new: 400 old: *default* virt_mode: ---------- new: pvh old: *default* ---------- ID: resize-builder-volume Function: cmd.run Name: qvm-volume resize qubes-builder-dvm:private 32212254720 ;qvm-volume resize qubes-work:private 32212254720 Result: True Comment: Command "qvm-volume resize qubes-builder-dvm:private 32212254720 ;qvm-volume resize qubes-work:private 32212254720" run Started: 11:59:57.464803 Duration: 1531.841 ms Changes: ---------- pid: 25531 retcode: 0 stderr: stdout: ---------- ID: builder-policy Function: file.managed Name: /etc/qubes/policy.d/30-qubesbuilder.policy Result: True Comment: File /etc/qubes/policy.d/30-qubesbuilder.policy is in the correct state Started: 11:59:58.996895 Duration: 114.441 ms Changes: Summary for local ------------- Succeeded: 14 (changed=3) Failed: 0 ------------- Total states run: 14 Total run time: 35.806 s ```
andrewdavidwong commented 7 months ago

@tlaurion, please use collapsible sections for long, nonessential content.

tlaurion commented 7 months ago

@tlaurion, please use collapsible sections for long, nonessential content.

@andrewdavidwong done. Thanks, learned something today.

antler5 commented 4 months ago

https://github.com/QubesOS/qubes-issues/issues/8774#issuecomment-1890846073 https://github.com/QubesOS/qubes-issues/issues/8491#issuecomment-1733355611

I just wanted to say thanks for figuring this out. I'm trying Salt for the first time tonight, started in base, and in the process of migrating to user I get myself into a state reminiscent of the symptoms in https://github.com/QubesOS/qubes-issues/issues/8491: every other run of saltutil.sync_all alternated two sections in qubesctl --target dom0 --show-output piller.items from qubes: to salt-{smth} and back, several commands would only work in one state, and some didn't seem to be working at all. I was pondering the easiest way to reset dom0 when I found the tip to create that symlink, which fixed it for me. I'm not sure it's still known to be something that can be stumbled upon. (Un-)fortunately I can't tell you if the link was missing to begin with short of retracing my steps on a fresh install, which I don't plan on following up on atm, but hopefully it's useful to know that I still needed it as a new and disoriented user on 4.2.1.

ben-grande commented 1 month ago

I also have a Salt formula for Qubes Builder V2.

But more tested now.

Summary:

You may try the formula. Notice it is a work in progress and the policy allows calls to sys-pgp, so you have to set isolate_home_dirs in the split-gpg2 configuration for security reasons (waiting for #8792 to package this part).

Not true anymore, Qusal deploys a qubes-split-gpg2.conf to isolate_home_dirs in sys-pgp.