Closed vinceskahan closed 1 year ago
I think you are wrong... I tried it and it worked just fine...
Open imager, write a new card and add your settings. Then open the card in your file manager and look at the first-run.sh file and the changes to cmdline.txt
Please reopen. NOT resolved in the least.
Again, I do NOT use imager so that's not a good answer. You did not specify 'what' settings to add nor now/where so that is not a good answer either. There is no first-run.sh file in the .img.xz files that I can find so that again is not an accurate answer.
What do we add WITHOUT using rpi-imager to make hands-off wifi setup happen ? If such functionality exists, we need accurate and complete information on what we need to do.
I find this removal inconvenient for workflows that heavily or entirely rely on cli. But it's not really annoying as there still exists ways to minimize the use of raspi-imager, say, for deployment purposes. If you really don't want to use the imager, then perhaps try out other distros that still allow this (e.g. alpine, etc.).
Ok, I did it for you... RPI-Imager creates a file, firstrun.sh containing:
#!/bin/bash
set +e
cat >/etc/wpa_supplicant/wpa_supplicant.conf <<'WPAEOF'
country=GB
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
ap_scan=1
update_config=1
network={
ssid="MYSSID"
psk=51f2c81286aaaa24f39e1f6258244c8afb5f3a9a54802cd0a845805a9eb87798
}
WPAEOF
chmod 600 /etc/wpa_supplicant/wpa_supplicant.conf
rfkill unblock wifi
for filename in /var/lib/systemd/rfkill/*:wlan ; do
echo 0 > $filename
done
rm -f /boot/firstrun.sh
sed -i 's| systemd.run.*||g' /boot/cmdline.txt
exit 0
It then edits the cmdline.txt to add the following to the end:
systemd.run=/boot/firstrun.sh systemd.run_success_action=reboot systemd.unit=kernel-command-line.target
If such functionality exists, we need accurate and complete information on what we need to do.
rpi-imager is using functionality of systemd to get a working firstrun.sh. You can find documentation here.
I find this removal inconvenient for workflows that heavily or entirely rely on cli. But it's not really annoying as there still exists ways to minimize the use of raspi-imager, say, for deployment purposes. If you really don't want to use the imager, then perhaps try out other distros that still allow this (e.g. alpine, etc.).
Yeah, that's perfectly understandable and we didn't just remove the functionality for fun. With the switch to NM, just copying wpa_supplicant.conf isn't going to do anything, so another mechanism will need to be added.
The plan is to add support for cloud-init to give users a more standard way to configure everything. In the past, cloud-init didn't have the required functionality, but AFAIK that has been resolved. I'll be re-visiting it in the near future.
In the meantime, the legacy release still exists for users who require the old behaviour and are not in a rush to move to bookworm.
Understood, thanks for the explanation. Tbh, was hoping that the official docs mentioned (or linked a reference to) an explanation like yours at the least as the removal was important.
@ghollingworth @XECDesign - thank you. That is all I was asking for.
When you guys remove a longstanding feature many rely on you need to (a) document it's gone and (b) provide a complete+accurately documented replacement for that functionality.
Don't just close the issues people open and say use imager (for example) when it was clear that folks were saying they do not use that tool and can't for their workflow.
FWIW - this firstrun method works fine and (to me) seems to not need any changes if you have other things to do. It just needs to actually be documented so users know what 'they' need to do to use this method. I'll add a pointer to this issue in the other one(s) related to the documentation of how to do headless wifi configuration.
Aware this issue is closed but after struggling through the same process, I have some sympathy with @vinceskahan's surprise that the issue was closed so quickly, and I eventually solved it. I hope my solution below is useful for anyone that might stumble on this ticket like I did.
I've been working on upgrading some tooling I have to image my raspberry pi SD cards, from pre-bookworm to post-bookworm. The key features of this tooling were:
I can see a lot of discussion about cloud-init and perhaps that's the correct tool for this, however, I have no experience with it and I am using packer to add the file to the raspios image, and I succeeded in replicating the pre-bookworm behaviour by simply adding a file and setting the correct permissions as per the script below (both file contents AND the 600 permissions are important):
cat >/etc/NetworkManager/system-connections/wifi.nmconnection <<EOF
[connection]
id=wifi
uuid=95b17351-297c-4d9c-90f9-669d3ab81464
type=wifi
[wifi]
mode=infrastructure
ssid=${SSID}
[wifi-security]
key-mgmt=wpa-psk
psk=${PASSWORD}
[ipv4]
method=auto
[ipv6]
addr-gen-mode=default
method=auto
[proxy]
EOF
chmod 600 /etc/NetworkManager/system-connections/wifi.nmconnection
Note: this file is the same as the one created using the raspi-config
program to set up wifi connections manually, so in case your settings such as key-mgmt are different, you can copy them from your own file.
The Pi will recognise this on first boot and the wifi connection will start automatically, just like it used to with the wpa_supplicant.conf
method.
I tried the method suggested above by @ghollingworth (to "reverse engineer" the first-run.sh created by the pi imager program) however, I could not get it to work. Furthermore, I got confused about the difference between NetworkManager and wpa_supplicant and it seems that Pi is moving towards NetworkManager therefore to have a solution that enables wpa_supplicant again seemed to move against the tide somehow, from my position of complete ignorance about both technologies and raspios.
I think it's worth noting that there are some linked tickets to this one, and many many forum posts, blog posts, etc over the years on this topic. It's a really useful feature and could perhaps be better documented like the wpa_supplicant method was. That could be something simple along the lines of my solution here (which is quite similar to what is suggested in https://github.com/raspberrypi/rpi-imager/issues/706 ) or a full blown hello-world headless configuration script using cloud-init or some other automation technology, I don't know. In the meantime, I'll keep fumbling my own way towards such a thing. Perhaps I'll check out cloud-init next.
The code is currently in a private repo but if anyone wants it then let me know and I'll consider breaking it out into a public one.
Just to again chime in a little, the method to run rpi-imager 'once' and save custom settings to a firstrun.sh script does work fine if you do the minor additions to config.txt in the boot partition in your modified image. I've tested it in various images of the current raspios and it works fine. The 'good' part of doing it the hard way like that is you can do other things in firstrun.sh as well such as setting keyboard/keymap/timezone as well as an initial user and encrypted password. The 'bad' part is the need to use a certain version of rpi-imager....and the need to use it at all. The rpi docs are too opinionated by far re: what tools they seem to expect folks to use, and the seeming expectation we run graphical desktops.
A public repo with how to add stuff to a downloaded upstream image via Packer sounds like a great thing to have.
One question though - where do you determine the uuid for wifi ? If that is documented somewhere, I've never been able to find it.
Regardless, my flow is manual currently. Download image, click on it on the Mac and it opens the boot partition. Drop in firstrun.sh, userconf.txt (probably not needed), and touch ssh (possibly not needed. Edit config.txt to add the firstrun options. Unmount the boot partition. The resulting modified image then makes hands-off boot+configure work fine.
But packer for an end-to-end automated procedure would be great to have a 'fully working well documented' example to try !
Enjoy: https://github.com/tonicsoft/raspios-headless-imager
Regarding the UUID, I used a newly generated random one and it worked fine.
NIce idea to start with the image produced by the imager application. The convenient thing about my packer script in the repo above is that it pulls the image direct from the raspberry pi website, but I'm sure there would be a way to pull it from a local file or you could host it on the web yourself. You could probably add those "other things" that you wanted to do in firstrun.sh using packer too.
I think your giving the docs a hard time, they're aimed at making it easier for as many people as possible, not people like us trying to nerd out on automated workflows :)
If you end up testing my packer thing, let me know if it works out for you. I'm curious how robust it is and whether it will work out of the box for you.
And for anyone else coming across this thread, I'd love to see something similar using cloud-init since I have no idea what that is.
The rpi docs are too opinionated by far re: what tools they seem to expect folks to use,
I would entirely disagree with that statement.
Two of the main ways the documentation are split up is based on whether they need to be task lead or featured lead.
Feature lead tends to involve the lower- level functions – giving information on how, e.g., SPI or i2c work and how you can access them. This is the documentation that most expert-level users are looking for, it documents the thing as it exists and doesn't necessariy document how to accomplish any particular task. It merely documents the existence of something, expert users are supposed to know what to do with that documentation. A classic example of this sort of documentation is datasheets. Datasheets tell you that things are a certain way.
Task lead tends to be a bit meatier, giving you all the information you need to perform an action. Documentation that offers many alternatives to accomplish the same task is poor documentation. Good documentation should choose a methodology and document it properly, it should not give the end user choices, "you could do it this way, or this way, or this other way" instead it should say "if you want to do x you should do y."
Most of the online documentation for Raspberry Pi is task lead except for documentation directly around the hardware.
and the seeming expectation we run graphical desktops.
Again, I would disagree.
There is plenty of documentation that takes a command line first approach, for instance our network documentation.
However here we're discussing topics around installation and getting started, so here yes, we're graphical first. Most of our users use Raspberry Pi Imager and install the graphical desktop version of the OS. Our documentation is tailored for that situation.
I would be happy to take a look at any submission you have to improve the documentation. We take Pull Requests. However, additions need not just to be technically correct, they have to fit within scope, and within the narrative of the documentation.
I think your giving the docs a hard time, they're aimed at making it easier for as many people as possible, not people like us trying to nerd out on automated workflows :)
This is by definition not-a-documentation problem. We have removed wpa_supplicant for setting up WiFi before first boot. There is no officially supported alternative. Therefore there is no documentation. I'm not going to have documentation on the site which details a temporary workaround for a problem unless that problem is a problem for the majority (or at least a large number of) users. This is not that sort of problem. The workaround isn't simple, and would be hard to document so that most users would understand how to implement it. It's therefore going to add confusion and increase support burden rather than reduce it. So it will not be documented.
Let me add 3 cents from me 😉
Why not use that same approach like with wpa_supplicant for NetworkManager ? If boot-script find in /boot
(/boot/firmware
) any files *.nmconnection
it'll move them and overwrite (if exists) and change it's rights. As for file "proper content" it'd user problem (the file content is not that complicated anyway).
As for set-up WIFI_COUNTRY
it could be done the same way like with userconf
or ssh
- just look for file e.g. wifi_country
/wifi-country.txt
, read it content (first line/2 first letters). In file should be only 2 letters with country code. If file is found, it's validated like in script /usr/lib/raspberrypi-sys-mods/imager_custom
and if country code is OK then script run raspi-config nonint do_wifi_country "<country-code>"
(like mentioned script).
Unfortunately NetworkManager still didn't implemented ability to set wifi country from it's configuration file :/ , there is even issue for this feature: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/960
Still use a file (placed in /boot
partition), but one file would contain all(multiple) settings for wifi, file could have json format, name.... let it have for e.g. wifi-setup.json
/ wifi-setup.txt
In file user would write everything that is needed for wifi to set-up and script would just convert that to nmconnection
files:
hidden
= 1
{
"country":"GB",
"ssids":
[
{
"ssid":"TheHomeWIFI",
"name":"Home",
"pass":"SuperSecretWifiPassword"
},
{
"ssid":"TheSecretWIFI",
"name":"SafehouseXD",
"hidden":1,
"psk":"abcdef....00"
}
]
}
using this approach is more simpler for users because json file format don't care for line-ends, while using copy-paste method which is simpler to implement might cause problems with windows-line-ends (\r\n
0x0d 0x0a
)
🤔 I don't know how NetworkManager will behave if it'll encounter \r
in nmconnection
file .
Ah, sorry but I hate using any "dedicated tools", so far I could just copy-paste few files to mounted sd card /boot partition, like: # cp ./pi-headless/* /mnt ; sync ; umount /dev/sdb2
, now just to setup stupid wifi I need to use some tool ☹️ 😒 and remember wifi password etc, /or/ ssh via LAN and then copy over nmconnection file and set it's rights (chmod 0600
) 😑 (at least I made a one-liner so I just paste a 'command' 😀 )...
(it's PITA to use anything expect built-in standard commands, like: cp , mount , vi , mv, rm...)
It's ok to have dedicated tools and use them in examples(docs?) , but there is a lot of people that you'd call a power user like myself who don't care for those things, we have a lot of "prepared" files/configs/data/scripts that just works by copy-paste so we don't need to waste time for e.g. to figuring out how to use some tools or wait until it "download" (again) something (and waste time & disk space) . (I didn't even know (nor I needed) there is a whole page with docs for rpi until November previous year 😛 )
For those that would like to know my one-liner here it is, just change: SSID, fill psk key and correct country code for your:
root@rpi:~# echo -e "[connection]\nid=wifi1\nuuid=ac4a8bf2-afcd-4bd4-bdb4-456d5a92c520\ntype=wifi\n[wifi]\nmode=infrastructure\nssid=SSID\nhidden=false\n[ipv4]\nmethod=auto\n[ipv6]\naddr-gen-mode=default\nmethod=auto\n[proxy]\n[wifi-security]\nkey-mgmt=wpa-psk\npsk=psk_key_here">wifi1.nmconnection ; mv wifi1.nmconnection /etc/NetworkManager/system-connections/ ; chmod 0600 /etc/NetworkManager/system-connections/*.nmconnection ; raspi-config nonint do_wifi_country "PL"
Just a minor addition: looking into what raspi-config
does when used to configure WiFi country on bookworm
, I see that it adds the following to the end of cmdline.txt
in the boot firmware partitiion:
cfg80211.ieee80211_regdom=XX
where XX
is the 2 digit country code for the regulatory domain under which the WiFi to which the Pi connecting falls.
Can a stop gap arrangement be put in place that takes wpa_supplicant
dropped by the user on sdcard and creates a nm wifi connection? A pi without proper headless capability is just a poor man's nuc. Also to mention, plugging a cat6 cable between laptop and rpi5 doesn't cut any mustard either. For some strange reason link-local is restricted to ipv6, this setup is flaky as it can get. Local link barely runs for 1.5min before it keels over. So better carry a monitor, a keyboard, and a mouse when taking pi5 out in back pocket.
Had same issue upgrading an old tablet accessible only via wifi :-(
Please note that NetworkManager default is to retry connecting only 4 times, which may be not enough in some circumstances, if you want to have it retry forever you want to make sure that
connection.autoconnect-retries has the value of 0 (forever), instead of the default of -1 (see man nm-settings)
which you can modify with
nmcli connection edit <ssid>
It appears Raspberry Pi Imager does not create an accompanying link-local
profile when creating Wired connection 1
profile for eth0
. Which means in event of wifi failing to connect, there is no option to making a direct cable link to another computer and try get wifi working from there. My work around was to add another profile for eth0
and setting its ip4.method link-local
and ipv6.method link-local
. Now I can get access to headless even without wifi working, just have to remember to carry an Ethernet lead with the laptop. Pre-Bookworm option to drop wpa_supplicant
into sdcard root IMO was an extremely convenient way of ensuring headless access.
I agree, this is pretty frustrating. I've tried for hours getting wifi to work with a wpa_supplicant file, only to see this thread now. Please bring back dropping the wpa_supplicant on the ssd as an option for configuring wifi
@vinniejames it wouldn't be complicated to do exactly the same thing with .nmconnection
file(s) like with wpa_supplicant
... but the only issue is to set the wifi country...
It has been long enough to conclude they don't care enough to work this longstanding issue. It clearly is not going to happen.
----- @.*** ----
On Wed, Jun 19, 2024 at 3:46 PM saper-2 @.***> wrote:
@vinniejames https://github.com/vinniejames it wouldn't be complicated to do exactly the same thing with .nmconnection file(s) like with wpa_supplicant ...
— Reply to this email directly, view it on GitHub https://github.com/raspberrypi/bookworm-feedback/issues/72#issuecomment-2179539337, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAETMKSZLAPSWRWDJEHFSFLZIIC47AVCNFSM6AAAAAA6AQPA5WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZZGUZTSMZTG4 . You are receiving this because you were mentioned.Message ID: @.***>
Headless operation is really a strong point for raspberry pi among developers. It'll be good to have multiple ways to gain access without the use of monitor and keyboard in Bookworm. VNC and XRDP should be bug free. Another issue I am finding with Bookworm on a PI3B [Linux rpi3bw 6.6.31+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.31-1+rpt1 (2024-05-29) aarch64 GNU/Linux] is that avahi-daemon
regularly sets addresses such as raspberrypi-1
and occasionally loses IP address binding on wlan0
altogether. Another related issue is with the driver for Realtek usb wifi dongles, which support Hotspot and are very common. Bookworm's Realtek drivers are broken, but thankfully github.com/morrownr/8821cu-20210916.git
driver works and that assures access to a headless PI over Hotspot. Perhaps morrownr repo can be made part of updates.
It has been long enough to conclude they don't care enough to work this longstanding issue. It clearly is not going to happen. …
Thats unfortunate, especially given the number of man-hours wasted on figuring out wpa_supplicant doesnt work anymore. Seems like a pretty easy thing to continue support for, almost feels like they just want to force people into using the Raspberry Imager over Etcher
It has been long enough to conclude they don't care enough to work this longstanding issue. It clearly is not going to happen. …
Thats unfortunate, especially given the number of man-hours wasted on figuring out wpa_supplicant doesnt work anymore. Seems like a pretty easy thing to continue support for, almost feels like they just want to force people into using the Raspberry Imager over Etcher
I think RPI imager means waiting for entire OS getting copied over sdcard for a trivial change to networking setting. I use a Realtek usb wifi dongle as Hotspot giving a back door to regain control of my headless PIs plus some other uses. However agreed, it doesn't matches the flexibility of dropping a wpa_supplicant into sdcard.
It has been long enough to conclude they don't care enough to work this longstanding issue. It clearly is not going to happen. …
Thats unfortunate, especially given the number of man-hours wasted on figuring out wpa_supplicant doesnt work anymore. Seems like a pretty easy thing to continue support for, almost feels like they just want to force people into using the Raspberry Imager over Etcher
I think RPI imager means waiting for entire OS getting copied over sdcard for a trivial change to networking setting. I use a Realtek usb wifi dongle as Hotspot giving a back door to regain control of my headless PIs plus some other uses. However agreed, it doesn't matches the flexibility of dropping a wpa_supplicant into sdcard.
Yep, it's highly cumbersome to have to use the RPI imager. Yet another application to download to achieve a simple task that was working for ages before this change. What use to work and was easily scriptable with basic file system commands now requires manual user intervention and a full-blown re-image which can take 10-15 mins per board / per edit. Advanced users DO NOT want to have to go through GUI to configure the basics of 1) login 2) wifi access and 3) SSH access. It's unfortunate RPi decided to remove this workflow without having a sufficient alternative available.
It has been long enough to conclude they don't care enough to work this longstanding issue. It clearly is not going to happen. …
Thats unfortunate, especially given the number of man-hours wasted on figuring out wpa_supplicant doesnt work anymore. Seems like a pretty easy thing to continue support for, almost feels like they just want to force people into using the Raspberry Imager over Etcher
I think RPI imager means waiting for entire OS getting copied over sdcard for a trivial change to networking setting. I use a Realtek usb wifi dongle as Hotspot giving a back door to regain control of my headless PIs plus some other uses. However agreed, it doesn't matches the flexibility of dropping a wpa_supplicant into sdcard.
Wow, I hadn’t even considered changing the WiFi later, that’s crazy.
Maybe time to switch to Arduino for most things
Is a proper fix taking longer than I'd like? Yeah, absolutely. If people want to claim it's because we don't care, okay, but that's not what github issues are for. I wish I could address every single issue within a reasonable timeframe, but that's just not possible, so some people are always going to be upset.
Seems like a pretty easy thing to continue support for, almost feels like they just want to force people into using the Raspberry Imager over Etcher.
Couldn't care less which people use. Whatever works for your use case is the best choice.rpi-imager doesn't do anything special that a user can't trivially do themselves.
Advanced users DO NOT want to have to go through GUI to configure the basics of 1) login 2) wifi access and 3) SSH access.
Aside from wifi, you can do all of this without rpi-imager like you could before. Advanced users will have 0 trouble working around the limitation.
A temporary workaround for the issue reported here has been provided already and the plan for a proper fix has been provided.
There have been some good points raised here, and I'd like to be able to refer to them later when implementing the necessary changes. However, they're lost in some messages which are less than helpful. I'd recommend opening individual issues for individual problems and suggestions and using the forum for general discussion.
the same issue has plagued my team, who relies on raspberryPis to log data from meteorological sensors for scientific research. The removal of this feature - setting WPA config via rasp-config - has led to my instrumentation colleagues spending several days trying to figure why the Pi was not connecting to the wifi. On closer inspection I noted that the wpa_supplicant.conf was empty!!
It is hard to understand why this feature was removed in the first place. But also why the raspi-config option S1 Wireless LAN
is still there, if it produces not change.
the same issue has plagued my team, who relies on raspberryPis to log data from meteorological sensors for scientific research. The removal of this feature - setting WPA config via rasp-config - has led to my instrumentation colleagues spending several days trying to figure why the Pi was not connecting to the wifi. On closer inspection I noted that the wpa_supplicant.conf was empty!!
raspi-config can still be used to configure wifi, that's not what this issue was about. Raspberry Pi OS has switched to Network Manager a while ago, which doesn't use wpa_supplicant.conf. It speaks to wpa_supplicant directly.
Again, please open individual issues for individual problems and I'll do my best to help. This one doesn't have anything to do with it.
@XECDesign One more thing, I consider the language used in your reply not productive or respectful. Specially coming from a developer of @raspberrypi, a project focused on education when many young people get its first introduction to Linux and open hardware.
If people want to claim it's because we don't care, okay, but that's not what github issues are for.
Couldn't care less which people use
Advanced users will have 0 trouble working around the limitation.
Probably it would be a better strategy to think firs of the non-advance users, and try to design, implement and address things in a way that is supportive and welcoming to new comes "
Is a proper fix taking longer than I'd like? Yeah, absolutely. If people want to claim it's because we don't care, okay, but that's not what github issues are for. I wish I could address every single issue within a reasonable timeframe, but that's just not possible, so some people are always going to be upset.
Seems like a pretty easy thing to continue support for, almost feels like they just want to force people into using the Raspberry Imager over Etcher.
Couldn't care less which people use. Whatever works for your use case is the best choice.rpi-imager doesn't do anything special that a user can't trivially do themselves.
Advanced users DO NOT want to have to go through GUI to configure the basics of 1) login 2) wifi access and 3) SSH access.
Aside from wifi, you can do all of this without rpi-imager like you could before. Advanced users will have 0 trouble working around the limitation.
A temporary workaround for the issue reported here has been provided already and the plan for a proper fix has been provided.
There have been some good points raised here, and I'd like to be able to refer to them later when implementing the necessary changes. However, they're lost in some messages which are less than helpful. I'd recommend opening individual issues for individual problems and suggestions and using the forum for general discussion.
Probably it would be a better strategy to think firs of the non-advance users, and try to design, implement and address things in a way that is supportive and welcoming to new comes
Previous responses form other people specifically referred to advanced users and that's what I was addressing.
One more thing, I consider the language used in your reply not productive or respectful. Specially coming from a developer of @raspberrypi, a project focused on education when many young people get its first introduction to Linux and open hardware.
That's fine, I have no problem with people having and expressing opinions, but the main point is that this isn't the place for it.
Just opening an issue so this one doesn't get forgotten. Since at least 2016 the os has supported hands-off configuring of wifi via dropping a wpa_supplicant.conf file into the boot partition in the image. This was removed in the bookworm-based os but this longstanding (and very very helpful) functionality has not been replaced with some other equivalent feature.
For those of us who (a) do not use rpi-imager at all and (b) do a lot of burning of SD cards for headless raspi-lite systems, it is very important to give us an alternate way to seed our images with the wifi configuration. Please make it so in the very near future.
suggestion - if we could have a firstboot file where we simply wrote the nmcli options needed to set up wifi, you could do something like "nmcli `cat /boot/firmware/nmcli.options`" or the like to exec nmcli for us based on data that was there, if that file was present. Or just do a cp of a NetworkManager/ tree from the boot partition into place under /etc. Or any other mechanism you want. It's not necessary to go full cloud-init but I see rpi-imager uses that. 'Any' mechanism that works is fine and much better than the current 'no' mechanism in the 10-10-2023 os. Thanks.