Open ddevz opened 8 months ago
I think distributing a salt file to configure a Qubes OS developing environment is better than maintaining and distributing a full template.
In fact I have made something for myself:
fed-min-qubes-dev.sls:
fed-min-qubes-dev.top:
So this basically creates a template fedora-38-builder
and configures it to be able to work as both the actual qubes executor and where qubes-builderv2 locates. And it creates qubes-builder-dvm
which is the executor, and qubes-work
which is where you work with the source code and issue commands to build the packages.
Update: Fixed the top file.
Update2: Fixed the sls file.
Update3: Keep up with builderv2 code changes.
Neat. Any chance we could get something like this added to the distribution?
Any chance we could get something like this added to the distribution?
That would be nice.
Having many people using the same script would make the process more tested
In fact almost all things that this script does come from the official builderv2 repo's README. I just put them all together to ease my future re-install or something.
If fedora-38 template is in stable, and fedora-40 template is being developed, then things like ('fedoraver', 'fedora-40') could be in the script in the development branch version and ('fedoraver', 'fedora-38) could be in the script in the stable branch )
In my opinion the template chosen is not something critical to make things work, as long as it's a supported template ( i.e. not have reached EOL ). That's because even with qubes executor, the builder still uses mock to be able to build against different target distributions.
People would be more likely to use a salt script in dom0 that has been reviewed and added to the qubes distro, rather then one they download.
I agree. This script was originally just made for myself, but since you brought up this topic, I decided to post it here, thinking it might inspire someone to look into salt and eventually benefit the whole community.
@marmarek @fepitre I would strongly advice distributing a qubes-builder salt recipe. To be honest, its the third time I try to follow instructions of README at https://github.com/QubesOS/qubes-builderv2 and i'm still failing. Will give https://github.com/QubesOS/qubes-issues/issues/8774#issuecomment-1867854489 a shot myself.
@ddevz Can you rename this issue to "distribute salt recipe under qubes 4.2" or something similar? @andrewdavidwong ?
Note that @unman did a builder recipe at https://github.com/unman/shaker/tree/main/builder for v1 builder, which is packaged anfd worked for Q4.1. Maybe propose a PR there?
So this basically creates a template
fedora-38-builder
and configures it to be able to work as both the actual qubes executor and where qubes-builderv2 locates. And it createsqubes-builder-dvm
which is the executor, andqubes-work
which is where you work with the source code and issue commands to build the packages.
@Augsch123 : mind to explain how to use this properly?
Can you rename this issue to "distribute salt recipe under qubes 4.2" or something similar? @andrewdavidwong ?
How about the title now? I don't want to put "4.2" in the title, because that's redundant with labels, prone to going out-of-date, and isn't necessarily accurate, as Qubes Builder v2 isn't necessarily specific to 4.2. (For example, it could be used to build 4.3 or 5.0, and presumably people would still want this at that time.)
I also don't want to narrow the title to just "Distribute Salt recipe" because that assumes that Salt is the optimal implementation, which we don't know for certain. The issue title should state the goal you want to achieve, then let the developers decide on the best means to achieve it.
On a more general note @marmarek @fepitre @andrewdavidwong @DemiMarie @marmarta : @unman salt recipes under shaker project were spot on for Q4.1, but as for anything not upstreamed, it's lagging behind since the effort of one man army and is lacking eyes of qubesos devs/community to thrive properly and reach their qubesos users.
May I suggest forking shaker under contrib repo and collaborating with @unman, so that qubesos minimally packs shaker salt recipes for users to decide if they want to deploy such from contrib packages manually, but ideally, have rpm packages deployed per rpm spec files already existing and even have the salt installer also forked and contrib package of https://github.com/unman/qubes-task deployed as well?
This has been discussed numerous times in the past (persona related templates, moved to salt recipes to ease qubes customization deployment) going in the idealistic dream of having qubesos installer connected to the internet and permitting users/oem to deploy customized installation without needing to craft a custom qubesos iso for their needs?
Some references:
This is the best example I have for documentation as code. This explains directly how salt works to the curious, is easy to audit and facilitates learning. And permits people to understand easily what is happening (audit), simply looking at the spec files, which does the magic in an explicit fashion. More people would contribute salt recipes. More salt recipes would be installable by qubes-task GUI effortlessly within OS. More eyes. More easyness to contribute. Less documentation. Win win everywhere.
And then at term, qubes-task integration in qubesos installer with network access making the system behave exactly as needed on first boot? Wow. That would be a dream came true.
@Augsch123 : mind to explain how to use this properly?
I followed the steps below:
/srv/user_salt/
by following https://forum.qubes-os.org/t/qubes-salt-beginners-guide/20126/srv/user_salt/fed-min-qubes-dev.sls
with content pasted above./srv/user_salt/fed-min-qubes-dev.top
with content pasted above.sudo qubesctl top.enable fed-min-qubes-dev pillarenv=user saltenv=user
in dom0.sudo qubesctl state.highstate
, and qubes development environment should be ready.@Augsch123 : mind to explain how to use this properly?
I followed the steps below:
1. Set up `/srv/user_salt/` by following https://forum.qubes-os.org/t/qubes-salt-beginners-guide/20126 2. Create `/srv/user_salt/fed-min-qubes-dev.sls` with content pasted above. 3. Create `/srv/user_salt/fed-min-qubes-dev.top` with content pasted above. 4. Run `sudo qubesctl top.enable fed-min-qubes-dev pillarenv=user saltenv=user` in dom0. 5. Run `sudo qubesctl state.highstate`, and qubes development environment should be ready.
Hello @Augsch123 ! Thanks you for sharing and updating your salts with V3.
I tried to replicate.
V3 fed-min-qubes-dev.sls :
[user@dom0 qubes-builder-v2-salt]$ sudo cat /srv/user_salt/fed-min-qubes-dev.sls
{% set fedoraver = salt['pillar.get']('fedoraver', 'fedora-38') %}
{% if grains['id'] == 'dom0' %}
ensure-{{ fedoraver }}-minimal-installed:
qvm.template_installed:
- name: {{ fedoraver }}-minimal
- fromrepo: qubes-templates-itl
create-{{ fedoraver }}-builder:
qvm.clone:
- name: {{ fedoraver }}-builder
- source: {{ fedoraver }}-minimal
- require:
- qvm: ensure-{{ fedoraver }}-minimal-installed
create-qubes-builder-dvm:
qvm.vm:
- name: qubes-builder-dvm
- present:
- template: {{ fedoraver }}-builder
- label: red
- prefs:
- virt_mode: pvh
- label: red
- template-for-dispvms: true
- memory: 800
- maxmem: 8000
- vcpus: 6
- require:
- qvm: create-{{ fedoraver }}-builder
create-qubes-work:
qvm.vm:
- name: qubes-work
- present:
- template: {{ fedoraver }}-builder
- label: red
- prefs:
- virt_mode: pvh
- label: red
- memory: 400
- maxmem: 1000
- default-dispvm: qubes-builder-dvm
- require:
- qvm: create-qubes-builder-dvm
resize-builder-volume:
cmd.run:
- name: qvm-volume resize qubes-builder-dvm:private 32212254720 ;qvm-volume resize qubes-work:private 32212254720
builder-policy:
file.managed:
- name: /etc/qubes/policy.d/30-qubesbuilder.policy
- contents: |
admin.vm.CreateDisposable * qubes-work qubes-builder-dvm allow target=dom0
admin.vm.Start * qubes-work @tag:disp-created-by-qubes-work allow target=dom0
admin.vm.Kill * qubes-work @tag:disp-created-by-qubes-work allow target=dom0
qubesbuilder.FileCopyIn * qubes-work @tag:disp-created-by-qubes-work allow
qubesbuilder.FileCopyOut * qubes-work @tag:disp-created-by-qubes-work allow
qubes.Filecopy * qubes-work @tag:disp-created-by-qubes-work allow
qubes.WaitForSession * qubes-work @tag:disp-created-by-qubes-work allow
qubes.VMShell * qubes-work @tag:disp-created-by-qubes-work allow
{% elif grains['id'] == [fedoraver, '-builder']|join %}
update:
cmd.run:
- name: dnf install -y qubes-core-agent-networking qubes-core-agent-passwordless-root qubes-core-agent-thunar xfce4-terminal unzip wqy-microhei-fonts python3-packaging createrepo_c devscripts gpg python3-pyyaml rpm reprepro python3-pathspec mktorrent rpm-sign tree mock openssl python3-jinja2-cli m4 asciidoc rsync qubes-gpg-split dnf-plugins-core debootstrap dpkg-dev git mock pbuilder which perl-Digest-MD5 perl-Digest-SHA python3-sh rpm-build rpmdevtools wget python3-debian python3-lxml systemd-udev python3-click qubes-gpg-split
{% elif grains['id'] == 'qubes-builder-dvm' %}
builderdirectory:
cmd.run:
- name: mkdir -p /rw/bind-dirs/builder /rw/config/qubes-bind-dirs.d ; echo "binds+=('/builder')" > /rw/config/qubes-bind-dirs.d/builder.conf ;
appendrc:
file.append:
- name: /rw/config/rc.local
- text: mount /builder -o dev,suid,remount
{% elif grains['id'] == 'qubes-work' %}
git-fetch:
cmd.run:
- name: cd /home/user ; git clone https://github.com/QubesOS/qubes-infrastructure-mirrors ; git clone /https://github.com/QubesOS/qubes-builderv2
- runas: user
install-infrastructure:
cmd.run:
- name: cd /home/user/qubes-infrastructure-mirrors ; python3 setup.py build ; python3 setup.py install
{% endif %}
[user@dom0 qubes-builder-v2-salt]$ sudo cat /srv/user_salt/fed-min-qubes-dev.top
{% set fedoraver = salt['pillar.get']('fedoraver', 'fedora-38') %}
user:
dom0 or {{ fedoraver }}-builder or qubes-builder-dvm or qubes-work:
- fed-min-qubes-dev
V3 fed-min-qubes-dev.top:
[user@dom0 qubes-builder-v2-salt]$ sudo cat /srv/user_salt/fed-min-qubes-dev.top
{% set fedoraver = salt['pillar.get']('fedoraver', 'fedora-38') %}
user:
dom0 or {{ fedoraver }}-builder or qubes-builder-dvm or qubes-work:
- fed-min-qubes-dev
What am I doing wrong?
@tlaurion Did you manually create /srv/user_salt
? Unfortunately that's not what I meant. I think you'll need to run sudo qubesctl state.sls qubes.user-dirs
to set up it properly, as mentioned in that forum thread.
We run the command sudo qubesctl state.sls qubes.user-dirs. Salt applies the corresponding state, and tells us that some files and directories were created. Among these directories we can find /srv/user_salt/: this is the main directory where we’ll place our own state configuration files.
Reading the logs, [ERROR ] State 'qvm.template_installed' was not found in SLS 'fed-min-qubes-dev'
is strange and the only reason that I can come up with for it not being found, is that you might manually created/srv/user_salt
.
@tlaurion Did you manually create
/srv/user_salt
? Unfortunately that's not what I meant. I think you'll need to runsudo qubesctl state.sls qubes.user-dirs
to set up it properly, as mentioned in that forum thread.We run the command sudo qubesctl state.sls qubes.user-dirs. Salt applies the corresponding state, and tells us that some files and directories were created. Among these directories we can find /srv/user_salt/: this is the main directory where we’ll place our own state configuration files.
Reading the logs,
[ERROR ] State 'qvm.template_installed' was not found in SLS 'fed-min-qubes-dev'
is strange and the only reason that I can come up with for it not being found, is that you might manually created/srv/user_salt
.
Right. I didn't understand that part and re-did. On Q4.2, fedora-38-minimal installed from command line with qvm-template.
@Augsch123 Thank for your time.
Multiple errors. fedora-38-minimal not found, actions are applied in wrong order... Too new to salt to understand what i'm doing here but unfortunately this is not working out of the box on Q4.2, just like upstream instructions. :/
Any idea for an update 4 at https://github.com/QubesOS/qubes-issues/issues/8774#issuecomment-1867854489 ? @unman (shaker: builderv2?) @marmarek?
@Augsch123 Seems like there is a bug on Q4.2 that was unfixed. https://github.com/QubesOS/qubes-issues/issues/8491#issuecomment-1890843137
Basically this worked:
sudo ln -s /srv/salt/qubes/user-dirs.top /srv/salt/_tops/base/user-dirs.top
#needs bugfix @marmarek
Set up
/srv/user_salt/
by following https://forum.qubes-os.org/t/qubes-salt-beginners-guide/20126Create
/srv/user_salt/fed-min-qubes-dev.sls
with content pasted above.Create
/srv/user_salt/fed-min-qubes-dev.top
with content pasted above.Run
sudo qubesctl top.enable fed-min-qubes-dev pillarenv=user saltenv=user
in dom0.Run
sudo qubesctl state.highstate
, and qubes development environment should be ready.
Simulated a rerun since there is nothing that was installed in the template (packages needed) not git clone or other dependencies deployed and qubes-work is basically a fedora-38-minimal's dvm without any customization applied.
@Augsch123 Basically the following part is never applied:
{% elif grains['id'] == [fedoraver, '-builder']|join %}
update:
cmd.run:
- name: dnf install -y qubes-core-agent-networking qubes-core-agent-passwordless-root qubes-core-agent-thunar xfce4-terminal unzip wqy-microhei-fonts python3-packaging createrepo_c devscripts gpg python3-pyyaml rpm reprepro python3-pathspec mktorrent rpm-sign tree mock openssl python3-jinja2-cli m4 asciidoc rsync qubes-gpg-split dnf-plugins-core debootstrap dpkg-dev git mock pbuilder which perl-Digest-MD5 perl-Digest-SHA python3-sh rpm-build rpmdevtools wget python3-debian python3-lxml systemd-udev python3-click qubes-gpg-split
{% elif grains['id'] == 'qubes-builder-dvm' %}
builderdirectory:
cmd.run:
- name: mkdir -p /rw/bind-dirs/builder /rw/config/qubes-bind-dirs.d ; echo "binds+=('/builder')" > /rw/config/qubes-bind-dirs.d/builder.conf ;
appendrc:
file.append:
- name: /rw/config/rc.local
- text: mount /builder -o dev,suid,remount
{% elif grains['id'] == 'qubes-work' %}
git-fetch:
cmd.run:
- name: cd /home/user ; git clone https://github.com/QubesOS/qubes-infrastructure-mirrors ; git clone /https://github.com/QubesOS/qubes-builderv2
- runas: user
install-infrastructure:
cmd.run:
- name: cd /home/user/qubes-infrastructure-mirrors ; python3 setup.py build ; python3 setup.py install
{% endif %}
As can be seen by the following replication trace.
I also have a Salt formula for Qubes Builder V2.
Please read the warning first, it is true, I will do breaking changes while it is in alpha and I won't be able to debug if this occurs or any other issue, I am simply sharing my alternative. Provided as is, without any warranty of any kind to the extent permitted by law.
You may try the formula. Notice it is a work in progress and the policy allows calls to sys-pgp
, so you have to set isolate_home_dirs
in the split-gpg2 configuration for security reasons (waiting for https://github.com/QubesOS/qubes-issues/issues/8792 to package this part).
Notice you have to copy the whole repository to the salt path because it references other states that are outside the qubes-builder directory.
Follow the installation steps.
Basically the following part is never applied:
Have you tried rerun sudo qubesctl state.highstate
while not deleting fedora-38-builder? It was supposed to work in one go but I'm unsure what broke it.
Basically the following part is never applied:
Have you tried rerun
sudo qubesctl state.highstate
while not deleting fedora-38-builder? It was supposed to work in one go but I'm unsure what broke it.
Note that
This aims to test https://github.com/QubesOS/qubes-mgmt-salt-base-config/pull/11.
https://github.com/QubesOS/qubes-mgmt-salt-base-config/pull/11 works and replaces manual ln -s needed otherwise as explained under https://github.com/QubesOS/qubes-issues/issues/8491#issuecomment-1890843137.
The rest is the same:
@tlaurion, please use collapsible sections for long, nonessential content.
@tlaurion, please use collapsible sections for long, nonessential content.
@andrewdavidwong done. Thanks, learned something today.
https://github.com/QubesOS/qubes-issues/issues/8774#issuecomment-1890846073 https://github.com/QubesOS/qubes-issues/issues/8491#issuecomment-1733355611
I just wanted to say thanks for figuring this out.
I'm trying Salt for the first time tonight, started in base
, and in the process of migrating to user
I get myself into a state reminiscent of the symptoms in https://github.com/QubesOS/qubes-issues/issues/8491: every other run of saltutil.sync_all
alternated two sections in qubesctl --target dom0 --show-output piller.items
from qubes:
to salt-{smth}
and back, several commands would only work in one state, and some didn't seem to be working at all. I was pondering the easiest way to reset dom0
when I found the tip to create that symlink, which fixed it for me. I'm not sure it's still known to be something that can be stumbled upon. (Un-)fortunately I can't tell you if the link was missing to begin with short of retracing my steps on a fresh install, which I don't plan on following up on atm, but hopefully it's useful to know that I still needed it as a new and disoriented user on 4.2.1.
I also have a Salt formula for Qubes Builder V2.
But more tested now.
Summary:
v2
v2
gpg-client: gpg
until Qubes Builder example configurations stops using gpg-client: qubes-gpg-client-wrapper
.fedora-40-minimal
local
, docker
, podman
, qubes
qubes
executor, it may be possible to not have netvm on the builder (untested).http(s)
, ssh
qubes
executor can have no netvm.You may try the formula. Notice it is a work in progress and the policy allows calls to
sys-pgp
, so you have to setisolate_home_dirs
in the split-gpg2 configuration for security reasons (waiting for #8792 to package this part).
Not true anymore, Qusal deploys a qubes-split-gpg2.conf
to isolate_home_dirs
in sys-pgp
.
How to file a helpful issue
The problem you're addressing (if any)
When following the "qubes builder" instructions at https://www.qubes-os.org/doc/qubes-builder/ (and therefore https://www.qubes-os.org/doc/qubes-iso-building/) and it does not work (one example would be aborting the build with
make[1]: *** [Makefile:74: linux-6.1.62.tar.UNTRUSTED] Error 1
), it is not obvious if it is because the version of fedora you chose is not in sync with the version of qubes-builder you are using. (For example, you could be using a new version of fedora that hasn't been tested with the build process yet). It is also not obvious if the part of the instructions which mentions fedora versions is out of date.The solution you'd like
Distribute the build environment needed to run qubes builder without error. One option would be to make a template for "qubes builder" which allows people to install with something like:
sudo qubes-dom0-update qubes-template-qubes-builder
The value to a user, and who that user might be
Easier way to figure out the build process for the first time. Easier way for people to build the non-distributed templates (example: ubuntu)