`invalid user` when using chown with custom user

Hi!

First of all, this is an amazing tool! Thanks for the work you've done on it!

I've been setting up my Pi's for a while manually, and as you know it becomes a pain. Yesterday doing some research I discovered sdm, and here I am learning about it, to automate all the systems I use.

In this example I'm trying to set up a basic system with a hotspot, and that part I have it working already.

The issue comes when I try to do it under a different user than the default pi.

Reading the logs, I finally noticed the error:

chown: invalid user: ‘shadow:shadow'

Looking for related issue I found #126 , so I tried changing shadow:shadow to shadow.shadow, and later on also to only shadow, without specifying the group.

In all cases is the same outcome, Invalid user.

I don't know if I might be doing something wrong, or if it's actually an issue, so I decided to open this one.

The customize command I'm running is:

sudo sdm --customize --plugin L10n:host --plugin user:"adduser=shadow" --plugin mkdir:"dir=/home/shadow/.ssh|chown=shadow:shadow|chmod=700" --plugin copyfile:"from=authorized_keys|to=/home/shadow/.ssh|chown=shadow:shadow|chmod=600|mkdirif" --plugin hotspot:"config=hotspot.cfg" --host pearl 2024-07-04-raspios-bookworm-arm64-lite.img

And the trace of the command is:

Mount IMG '2024-07-04-raspios-bookworm-arm64-lite.img'
mount: /dev/loop24 mounted on /mnt/sdm.
mount: /dev/loop25 mounted on /mnt/sdm/boot/firmware.
* Start Configuration
> Command Line: /usr/local/bin/sdm --customize --plugin L10n:host --plugin user:adduser=shadow 
  --plugin mkdir:dir=/home/shadow/.ssh|chown=shadow:shadow|chmod=700 --plugin 
  copyfile:from=authorized_keys|to=/home/shadow/.ssh|chown=shadow:shadow|chmod=600|mkdirif 
  --plugin hotspot:config=hotspot.cfg --host pearl 2024-07-04-raspios-bookworm-arm64-lite.img
* Host Information
   Hostname:  star
   Memory:    32592024 kB
   uname:     Linux star 6.8.11-amd64
   os-release Name: Kali GNU/Linux Rolling
   Version: 2024.2 (kali-rolling)
   Like OS: debian
   sdm Version: V12.4
* IMG Information
   Name: 2024-07-04-raspios-bookworm-arm64-lite.img
   Date: 2024-07-04
   RasPiOS Version: 12
   RasPiOS Architecture: 64-bit aarch64
   os-release Version: 12 (bookworm)
% sdm will use qemu 'aarch64'
% sdm will use systemd-nspawn on this 64-bit x86-64 host
  Retry the command with --chroot if this fails
* Plugins selected:
   * L10n
       Args: host
   * user
       Args: adduser=shadow
   * mkdir
       Args: dir=/home/shadow/.ssh|chown=shadow:shadow|chmod=700
   * copyfile
       Args: from=authorized_keys|to=/home/shadow/.ssh|chown=shadow:shadow|chmod=600|mkdirif
   * hotspot
       Args: config=hotspot.cfg
> IMG '2024-07-04-raspios-bookworm-arm64-lite.img' has 343072 1K-blocks (0.4GB, 0.3GiB) free 
> Copy sdm to /usr/local/sdm in the IMG
* Start Phase 0 image customization
> Run Plugins Phase '0'
> Run Plugin 'L10n' (/mnt/sdm/usr/local/sdm/plugins/L10n) Phase 0 with arguments: 'host'
* Plugin L10n: Start Phase 0
> Plugin L10n: Keys/values found:
   host: 
> Plugin L10n: Read L10n configuration from host system
> Plugin L10n: Load Localization (L10N) settings from running system
> Plugin L10n:   Keymap:       es
> Plugin L10n:   Locale:       en_US.UTF-8
> Plugin L10n:   Timezone:     Asia/Bangkok
> Plugin L10n:   WiFi Country: 
> Plugin L10n: Save L10n configuration
* Plugin L10n: Complete Phase 0
> Run Plugin 'user' (/mnt/sdm/usr/local/sdm/plugins/user) Phase 0 with arguments: 
  'adduser=shadow'
* Plugin user: Start Phase 0
> Plugin user: Keys/values found:
   adduser: shadow
> Plugin user: Create user 'shadow' home directory '/home/shadow' so available for other plugins
* Plugin user: Complete Phase 0
> Run Plugin 'mkdir' (/mnt/sdm/usr/local/sdm/plugins/mkdir) Phase 0 with arguments: 
  'dir=/home/shadow/.ssh|chown=shadow:shadow|chmod=700'
* Plugin mkdir: Start Phase 0
> Plugin mkdir: Keys/values found:
   dir: /home/shadow/.ssh
   chown: shadow:shadow
   chmod: 700
> Plugin mkdir: Create directory '/home/shadow/.ssh'
* Plugin mkdir: Complete Phase 0
> Run Plugin 'copyfile' (/mnt/sdm/usr/local/sdm/plugins/copyfile) Phase 0 with arguments: 
  'from=authorized_keys|to=/home/shadow/.ssh|chown=shadow:shadow|chmod=600|mkdirif'
* Plugin copyfile: Start Phase 0
> Plugin copyfile: Keys/values found:
   from: authorized_keys
   to: /home/shadow/.ssh
   chown: shadow:shadow
   chmod: 600
   mkdirif: 
> Plugin copyfile: Add 'authorized_keys' to files to copy
> Plugin copyfile: Copy 'authorized_keys' to '/mnt/sdm/etc/sdm/assets/copyfile/.'
* Plugin copyfile: Complete Phase 0
> Run Plugin 'hotspot' (/mnt/sdm/usr/local/sdm/plugins/hotspot) Phase 0 with arguments: 
  'config=hotspot.cfg'
* Plugin hotspot: Start Phase 0
> Plugin hotspot: Keys/values found:
   config: hotspot.cfg
> Plugin hotspot: Read hotspot config 'hotspot.cfg' and update missing with defaults:
   channel: auto
   country: us
   device: wlan0
   dhcprange: 10.0.0.2,10.0.0.32,255.255.255.0
   domain: wlan.net
   enable: true
   hwmode: g
   leasetime: 24h
   passphrase: password
   ssid: MyNet
   wlanip: 10.0.0.1
   type: routed
   hsmode: nm                                                   [Default]
> Plugin hotspot: Write hotspot config to /mnt/sdm/etc/sdm/assets/hotspot/config
* Plugin hotspot: Complete Phase 0
* Phase 0 Completed
* Enter image '2024-07-04-raspios-bookworm-arm64-lite.img' for Phase 1
* Start Phase 1 image customization
> IMG '2024-07-04-raspios-bookworm-arm64-lite.img' has 343000 1K-blocks (0.4GB, 0.3GiB) free at start of Phase 1 image customization
> Configure and enable sdm FirstBoot service (sdm-firstboot)
> Set hostname 'pearl'
> Start 'apt update'
> IMG '2024-07-04-raspios-bookworm-arm64-lite.img' has 342996 1K-blocks (0.4GB, 0.3GiB) free at start of 'apt update'
> IMG '2024-07-04-raspios-bookworm-arm64-lite.img' has 228016 1K-blocks (0.2GB, 0.2GiB) free at end of 'apt update'
> Run Plugins Phase '1'
> Run Plugin 'L10n' (/usr/local/sdm/plugins/L10n) Phase 1 with arguments: 'host'
* Plugin L10n: Start Phase 1
% Plugin L10n: Keymap 'es' will be set during sdm FirstBoot
> Plugin L10n: Disable keyboard-setup service; will be re-enabled during sdm FirstBoot
Removed "/etc/systemd/system/sysinit.target.wants/keyboard-setup.service".
> Plugin L10n: Set locale to 'en_US.UTF-8'
Generating locales (this might take a while)...
  en_US.UTF-8... done
Generation complete.
> Plugin L10n: Set timezone to 'Asia/Bangkok'
* Plugin L10n: Complete Phase 1
> Run Plugin 'user' (/usr/local/sdm/plugins/user) Phase 1 with arguments: 'adduser=shadow'
* Plugin user: Start Phase 1
> Plugin user: Add user 'shadow'
useradd: group shadow exists - if you want to add this user to that group, use -g.
? Plugin user: useradd returned an error
* Plugin user: Complete Phase 1
> Run Plugin 'mkdir' (/usr/local/sdm/plugins/mkdir) Phase 1 with arguments: 
  'dir=/home/shadow/.ssh|chown=shadow:shadow|chmod=700'
* Plugin mkdir: Start Phase 1
* Plugin mkdir: Complete Phase 1
> Run Plugin 'copyfile' (/usr/local/sdm/plugins/copyfile) Phase 1 with arguments: 
  'from=authorized_keys|to=/home/shadow/.ssh|chown=shadow:shadow|chmod=600|mkdirif'
* Plugin copyfile: Start Phase 1
> Plugin copyfile: Run /etc/sdm/assets/copyfile/run-phase1-authorized_keys-homeshadow.ssh
> Plugin copyfile: Process file 'authorized_keys'
> Plugin copyfile: Copy 'authorized_keys' from /etc/sdm/assets/copyfile/. to '/home/shadow/.ssh'
> Plugin copyfile: Set file '/home/shadow/.ssh/authorized_keys' owner 'shadow:shadow'
chown: invalid user: ‘shadow:shadow’
> Plugin copyfile: Set file '/home/shadow/.ssh/authorized_keys' protection '600'
* Plugin copyfile: Complete Phase 1
> Run Plugin 'hotspot' (/usr/local/sdm/plugins/hotspot) Phase 1 with arguments: 
  'config=hotspot.cfg'
* Plugin hotspot: Start Phase 1
> Plugin hotspot: Hotspot will be enabled by sdm FirstBoot
* Plugin hotspot: Complete Phase 1
* Phase 1 post-app installation/configuration
> Start 'apt upgrade'
> IMG '2024-07-04-raspios-bookworm-arm64-lite.img' has 223164 1K-blocks (0.2GB, 0.2GiB) free at start of 'apt upgrade'
> IMG '2024-07-04-raspios-bookworm-arm64-lite.img' has 221748 1K-blocks (0.2GB, 0.2GiB) free at end of 'apt upgrade'
> Start 'apt autoremove'
> IMG '2024-07-04-raspios-bookworm-arm64-lite.img' has 221748 1K-blocks (0.2GB, 0.2GiB) free at start of 'apt autoremove'
> IMG '2024-07-04-raspios-bookworm-arm64-lite.img' has 221748 1K-blocks (0.2GB, 0.2GiB) free at end of 'apt autoremove'
> IMG '2024-07-04-raspios-bookworm-arm64-lite.img' has 221748 1K-blocks (0.2GB, 0.2GiB) free at end of Phase 1 image customization
* Phase 1 Completed
> Run Plugins Phase 'post-install'
> Run Plugin 'L10n' (/usr/local/sdm/plugins/L10n) Phase post-install with arguments: 'host'
> Run Plugin 'user' (/usr/local/sdm/plugins/user) Phase post-install with arguments: 
  'adduser=shadow'
* Plugin user: Start Phase post-install
* Plugin user: Complete Phase post-install
> Run Plugin 'mkdir' (/usr/local/sdm/plugins/mkdir) Phase post-install with arguments: 
  'dir=/home/shadow/.ssh|chown=shadow:shadow|chmod=700'
* Plugin mkdir: Start Phase post-install
> Plugin mkdir: Change owner on '/home/shadow/.ssh' to 'shadow:shadow'
chown: invalid user: ‘shadow:shadow’
> Plugin mkdir: Change directory protection for '/home/shadow/.ssh' to '700'
* Plugin mkdir: Complete Phase post-install
> Run Plugin 'copyfile' (/usr/local/sdm/plugins/copyfile) Phase post-install with arguments: 
  'from=authorized_keys|to=/home/shadow/.ssh|chown=shadow:shadow|chmod=600|mkdirif'
* Plugin copyfile: Start Phase post-install
* Plugin copyfile: Complete Phase post-install
> Run Plugin 'hotspot' (/usr/local/sdm/plugins/hotspot) Phase post-install with arguments: 
  'config=hotspot.cfg'
* Plugin hotspot: Start Phase post-install
> Prefix hotspot: Configure hotspot 'Hotspot' with Network Manager
* Plugin hotspot: Complete Phase post-install
> Run graphics post-install configuration
> Enable SSH
> Plugin phase1: Enable SSH service
> IMG '2024-07-04-raspios-bookworm-arm64-lite.img' has 221744 1K-blocks (0.2GB, 0.2GiB) free at end of image customization
> Customize elapsed time: 00:01:07
* Enter Shell Command Prompt
  'exit' to exit back to host system
root@sdm:/# exit
exit
* Customization complete
umount: /mnt/sdm/boot/firmware unmounted
umount: /mnt/sdm unmounted

Do you have any idea what could be?

Again, with pi user I get it to work, because the user obv is already there, but it would be cool to be able to set others :)

Thanks!!

Glad that you're finding sdm useful! Here's what is happening, I'll leave it to you to decide your path forward.

In a brand new, untouched IMG there is already a group named shadow, and it's a system group (In Linux, group IDs less than 1000 are typically reserved for system groups).
When sdm creates a user, it uses the useradd command to create the user, which also tries to create a group with the same name. Unfortunately, there's already a group named shadow, so the useradd command fails:

Plugin user: Add user 'shadow' useradd: group shadow exists - if you want to add this user to that group, use -g. ? Plugin user: useradd returned an error
Arguably, this should be a fatal error and stop the customization process, but at the moment it doesn't. Will look into that for a future release.
Since the user shadow was not created, all subsequent attempts that try to use the user shadow also fail: specifically, your mkdir and copyfile plugin invocations

The easiest fix for you is to rename the user shadow to something else. At the moment there's no workaround in the user plugin for this.

Here are some other notes and comments that I hope are helpful.

I'd encourage you to consider moving all your plugins into a pluglist file documented here. It eliminates clutter on the command line, makes it easier to see what plugins you're using, and you don't need all the double quotes around plugin arguments. So, your customize command line would be

sudo sdm --customize  --plugin @/path/to/pluglist --host pearl 2024-07-04-raspios-bookworm-arm64-lite.img`

And /path/to/pluglist file would contain (I used xshadow instead of shadow):

L10n:host
user:adduser=xshadow
mkdir:dir=/home/xshadow/.ssh|chown=xshadow:xshadow|chmod=700
copyfile:from=authorized_keys|to=/home/xshadow/.ssh|chown=xshadow:xshadow|chmod=600|mkdirif
hotspot:config=hotspot.cfg

I don't know about you, but I think this is a lot easier to work with! [NOTE: I did not test the above]

I typically don't set the hostname during a customize, saving that until I burn the disk. That way, a customized IMG can be used for multiple target systems. Of course, you can still set the hostname on a burn if you've set it during customization, so not a problem, but I find that being rigorously prescriptive about the details keeps me from confusing myself 😉

I haven't looked at the hotspot plugin in quite a while. LMK if you run into any problems with it.

Oh man! Good catch! When I was writing the name I thought.. shadow is too common of a name, thinking of /etc/shadow, but I just went on with it ^_^ I wasn't aware of the existing group.

It works now! Thanks!

For organizing the plugins/commands I'm moving to the script ezsdm, so I can have different folders for each card, with the script ready to be run, with some assets that might be necessary for each set up. I also find that having it on a list is way more managable than on cluttered commands, and easier to version :)

Good tip also on the hostname :+1: , I'm still figuring the commands and how they interact on the different phases. Jumping soon on the code to understand some internals I'm still missing :p

I love it!! It's prob becoming my default setup from now :)

Thanks!!

Closing the issue, since it was never such a thing :pray:

Oh man! Good catch! When I was writing the name I thought.. shadow is too common of a name, thinking of /etc/shadow, but I just went on with it ^_^ I wasn't aware of the existing group.

It works now! Thanks!

Excellent ;) Thanks for including /etc/sdm/history, that makes it SO easy to see what's going on. I've been reading them for a few years so very good at scanning for problems.

For organizing the plugins/commands I'm moving to the script ezsdm, so I can have different folders for each card, with the script ready to be run, with some assets that might be necessary for each set up. I also find that having it on a list is way more managable than on cluttered commands, and easier to version :)

Great! You're headed down the right path!

Good tip also on the hostname 👍 , I'm still figuring the commands and how they interact on the different phases. Jumping soon on the code to understand some internals I'm still missing :p

I love it!! It's prob becoming my default setup from now :)

Thanks!! Thanks for the positive feedback! Happy to help with any further questions, problems, suggestions, etc.

Thanks! :pray:

I think I'm starting to figure out the best way to use the tool. In fact most of my cards share some base set up, like hotspot, aliases, etc.

So I think having an ezsdm for the base image makes sense. Then, specific projects can start from there. But this is where I'm not so sure how to proceed.

Can I run another --customize on top of an already customized image? I assume the last customize will override something that was set on the base if I specify so, but keeping the rest intact. Is that a valid approach?

The thing is, for more complex cards, I think I'll need to set them up manually with --explore. In this case the project folder will actually not hold its own ezsdm, but just an already customized base image.

I would like to be able to back up everything with only the script, without having to push the *.img's, but in the approach I explained above the "manual steps" would be lost.

It is still light-years better than what I have currently tho, which is some notes and bad memory :joy:

PS: I'm thinking it would be crazy dope to be able to connect Ansible playbooks with sdm, ultimate combo :rocket: Probably doesn't make sense to make any improvements on sdm for that tho, once the pi is up I could run the playbooks through the hotspot... hm... looking at that :)

@sraver some answers and a bit about what I do.

Yes, you can run another --customize on an IMG, but I think there are better ways. That said, when you try to customize an already-customized IMG, sdm warns you. You can omit the warning by using --redo-customize.

I try to script everything, so none of my systems are done manually. At the moment I have 3 Pi "servers":

One with DHCP/DNS (not pihole!). It has a few other services, such as apt-cacher-ng. BTW, if you have multiple Pi systems and aren't using apt-cacher-ng, you should make getting that working a top priority. You're welcome in advance 😉
One runs a VPN server, serves a 1TB work-disk, and is a "cold" backup DHCP/DNS server. It copies the DHCP/DNS json database at regular intervals (currently daily) and rebuilds it's database if needed. Failover is manual because in 25 years of running my own DHCP/DNS server I've not had a system fail. If that first time ever happens, I can have the network back up and running quicky enough for my needs
One piwatch/flightradar24 system

My other Pi systems are all configured as workstations.

(There's also a X64 NUC10-based file server and VirtualBox host that I hate having to upgrade it (every new Debian release) because it's such a pain without sdm 🤣. And I always do a fresh install. Only the best fresh new bugs for MY systems 🤣, followed by 2 years of basically not futzing with it)

I have a "standard" configuration for my Pis. It's the "workstation" configuration. When I want to build a new server, regardless of what it is, I burn the disk for that server and run whatever plugins are needed to turn it into the server. For instance, to rebuild my DHCP/DNS server I run plugins to:

Install the DNS/DHCP server (using sdm's ndm plugin)
Install apt-cacher-ng using the like-named plugin
Install postfix using the like-named plugin
Install logwatch using an as-yet unreleased like-named plugin
Add a .nmconnection file using the network plugin that configures it's static IP address (kinda needed for the DHCP/DNS server)
Use the system plugin to configure /etc/exports
Run another plugin that does host-specific stuff not yet done via other plugins.

The burn command. $dev, $host, and $img are appropriately set.

sdm --burn $dev --hostname $host --expand-root --plugin @/l/work/mystuff/$host/$host.plugins  $img

I have similar pluglist files ($host.plugins) for each server.

I'd argue that there's very little, if anything that you can only do manually by exploring into an IMG. It's taken me a while to get to this point, of course. For Bullseye, I was partially there. I spent some time completely redoing the implementation for my servers to take advantage of pluglists, new plugins, and new system capabilities, such as NetworkManager. I think it's much easier to achieve this nirvana with the current sdm and Bookworm.

The beauty of this, which I'm sure you recognize, is that bad memory is not a failure mode. Bad code might be 🤣 but I try to avoid that.

As far as backups, I don't do full backups. I figure out what config files I need to backup, and only back them up if they've changed. I have a script that runs nightly to do that.

re Ansible: I used it for a while, but it seemed fairly heavyweight and sluggish for my impatience to tolerate. That said, I see the value in it. If I were to use Ansible now, I would use sdm to install whatever bits needed to be installed on every system in my workstation IMG (ssh keys, etc). I'd install Ansible itself on whatever systems I want to be able to run ansible commands on, and I'd install the configuration data on an NFS share so that I can run ansible on any server as needed.

That's it in a nutshell. You'll undoubtedly do things differently than me.

TLDR: Figure out how to configure each and every system completely using sdm, and use as few customized IMGs as possible. Defer system-specific customization to when you burn the disk for that system.

Thanks for such detailed explanation!

Very helpful to get a glimpse of how you are using it, it helps understanding the thought process on the tool itself.

You opened a new rabbit hole for me now, ndm xD Previosuly I had been using pi-hole, but I'm in the search of something more lightweight and easily manageble, from the terminal if possible, and ndm seems to tick all the boxes!

I'm a feeling a bit stuck right now tho, not because any issue I'm finding in the tool, but more because of my lack my understanding of some pieces of the whole system I'm trying to implement.

Would you mind me writing an email to you to explain what I'm trying to achieve, and maybe I could get your thoughts on it? I don't mind sharing the final approach here, even though it would be completely off-topic on this very issue, but I feel I'm a bit lost, and I'd rather share a clean approach once I have it.

Regardless, thanks for your time, and all the tips and tricks! :heart:

PS: One related question! All the ndm set up process, I'm guessing this is something that you do once the system is running, which makes it "volatile", in the sense that it's not backed up, right? But I guess that's not really an issue, since configuration for ndm seems quite lightweight to set up, and also you are porbably not setting up this box very often, more likely is kind of a set-and-forget one, right?

ndm: yep, it does dhcp and dns really well, and it's lightweight. pihole has some nice features, but I've seen enough people having problems with it...

re ndm and backups: Correct, you still need to back up the configuration, but it's exactly one (json) file. With that you can easily rebuild and reinstall the dhcp and dns servers configs with 2 ndm commands. The json file is one of the files that I back up nightly, BTW, and my network doesn't change very often so once a day is fine. If you're changing the network a lot, you can back it up more often.

re email: 👍

gitbls / sdm

`invalid user` when using chown with custom user #237