[FEATURE]: use URL to download custom OS

[Vincent-Stragier]

Is your feature request related to a problem? Please describe.

It's not easy for all to install custom OS from downloaded images. For example, MyNatureWatch, is not available in the Other specific-purpose OS (if there is a way to add it, ping me) and the users are not necessary educated to the digital technologies (it's aimed at the general public).

Describe the solution you would like to see implemented

rpi-imager should provide an option to Use custom OS from URL.

Describe alternatives you've considered

Add a section for experimental / unsafe OS, where developers can easily publish images for all.

Additional context

Maybe even activate a Paste from clipboard button if a URL is detected in the clipboard.

Version

1.8.5 (Default)

[Vincent-Stragier]

I just saw there is a form to request an OS, I will see what can be done with the maintainers.

[tdewey-rpi]

I'm deeply reticent to provide such a function.

The manifest used by rpi-imager contains more than just the URL. Among other things, it also includes a hash to allow rpi-imager to confirm what was downloaded was expected - a useful function that I would not like to see bypassed.

For the truly experimental operating systems, the 'Custom Image' option is provided - but this has an assumed technical competence that I believe is a fair request, given the lower standards that can be applied to a completely custom image.

As such, I'm not convinced that reducing the barrier to using rpi-imager with completely unverified targets represents an improvement for our first-time-users. Instead, I would recommend that operating systems request to be added to the Raspberry Pi Imager OS List, where they can provide better context around their OS.

That said, I'm happy to accept when I'm wrong, and I'm prepared to receive well reasoned arguments.

[Vincent-Stragier]

The manifest used by rpi-imager contains more than just the URL. Among other things, it also includes a hash to allow rpi-imager to confirm what was downloaded was expected - a useful function that I would not like to see bypassed.

I didn't know that. And I didn't know either about the hashes on the rpi-imager download page (it's not something that is advertised on the main download page). But since we can already install custom images from the computer, the hash security feature is already bypassed (that and the fact that most won't bother to check their rpi-imager package hash anyway).

However, you could encourage or force the user who would download the image with rpi-imager to use a hash (which can be easily provided by the developers), thus providing a more secure way to obtain custom images.

It could also be a plus to have some sort of more involved confirmation mechanism before writing a custom image, like writing, "yes" or "yes, I understand the risks" and giving a proper disclaimer ("You are going to install an unverified image, which may contain malicious code that could infect the devices connected to your network; only proceed to the installation if you know what you are doing and are confident enough about the source of your image, etc.").

Also, balenaEtcher already provides this feature, but does not allow customizing the image settings, which is a security risk by itself (most of the time, the user pi with the default password will be used).

Instead, I would recommend that operating systems request to be added to the Raspberry Pi Imager OS List, where they can provide better context around their OS.

That makes sense, but you don't want to publicly register your custom images every time. I can imagine building a custom image for a company or a school project without having the intention to share it in a broad way (it could be stored on an intranet).

I don't know if the above arguments are well reasoned, but I think there are some valid points, that can help to improve the user experience and the security of the app.

[lurch]

That makes sense, but you don't want to publicly register your custom images every time. I can imagine building a custom image for a company or a school project without having the intention to share it in a broad way (it could be stored on an intranet).

In that scenario, https://github.com/raspberrypi/rpi-imager/blob/qml/README.md#custom-repository might be a better option? https://downloads.raspberrypi.com/os_list_imagingutility_v4.json shows the kind of info that you'll want to include in your custom JSON file.

[Vincent-Stragier]

Hi @lurch,

Yes, it could do the job. I think it's somehow a bit more cumbersome to setup than simply generating a hash and hosting the image. It could be interesting to have a MWE in the documentation.

I see that the rpi-imager can be used in CLI mode. Why not allowing <image file to write> to be either a path or an URL? I don't know if it works on Windows though.

rpi-imager --cli [--disable-verify] [--sha256 <expected hash>] [--debug] [--quiet] <image file to write> <destination drive device>

[tdewey-rpi]

The manifest used by rpi-imager contains more than just the URL. Among other things, it also includes a hash to allow rpi-imager to confirm what was downloaded was expected - a useful function that I would not like to see bypassed.

I didn't know that. And I didn't know either about the hashes on the rpi-imager download page (it's not something that is advertised on the main download page). But since we can already install custom images from the computer, the hash security feature is already bypassed (that and the fact that most won't bother to check their rpi-imager package hash anyway).

Indeed - but the level of effort and implied support for such a flow is very different compared to the items in the OS list - that flow is sufficiently different it can indicate "here be dragons" (as the old maps would say).

However, you could encourage or force the user who would download the image with rpi-imager to use a hash (which can be easily provided by the developers), thus providing a more secure way to obtain custom images.

Entirely possible - but I fear it's going to be obviated by the custom repo URL support, which would present a richer experience for a custom OS distributor.

It could also be a plus to have some sort of more involved confirmation mechanism before writing a custom image, like writing, "yes" or "yes, I understand the risks" and giving a proper disclaimer ("You are going to install an unverified image, which may contain malicious code that could infect the devices connected to your network; only proceed to the installation if you know what you are doing and are confident enough about the source of your image, etc.").

I've been considering this as part of the Imager-2.0 release. Unfortunately, users are largely conditioned to ignore such warnings by other software and web properties.

Also, balenaEtcher already provides this feature, but does not allow customizing the image settings, which is a security risk by itself (most of the time, the user pi with the default password will be used).

Unfortunately we wouldn't be able to offer OS Customisation on an image without the explicit metadata offered by a manifest file - because we wouldn't know which customisation scheme to use to implement the user's choices. That's the case today if you select Use custom.

To provide customisation, the OS would need to implement one of the customisation schemes we offer (either the firstrun.sh currently used by Raspberry Pi OS, or the cloud-init mechanism used by Ubuntu Server)

Instead, I would recommend that operating systems request to be added to the Raspberry Pi Imager OS List, where they can provide better context around their OS.

That makes sense, but you don't want to publicly register your custom images every time. I can imagine building a custom image for a company or a school project without having the intention to share it in a broad way (it could be stored on an intranet).

In such a case, I'd strongly recommend using the suggestion from @lurch, where you specify a custom repository URL:

rpi-imager --repo ${YOUR_OS_LIST_URL} Handled in the source code at: https://github.com/raspberrypi/rpi-imager/blob/686ad14308639b0589a2971903066f56271033e7/src/main.cpp#L211

I don't know if the above arguments are well reasoned, but I think there are some valid points, that can help to improve the user experience and the security of the app.

Thank you for providing a clear rationale behind your request. This is exactly the sort of discourse I hope to see on the feature requests in rpi-imager, and will provide good context around any decision taken.

[tdewey-rpi]

Hi @lurch,

Yes, it could do the job. I think it's somehow a bit more cumbersome to setup than simply generating a hash and hosting the image. It could be interesting to have a MWE in the documentation.

I'll certainly consider expanding the documentation around hosting your own OS list - it's not a particularly complicated concept, as that is in effect what each OS vendor is doing under today's model.

I see that the rpi-imager can be used in CLI mode. Why not allowing <image file to write> to be either a path or an URL? I don't know if it works on Windows though.

rpi-imager --cli [--disable-verify] [--sha256 <expected hash>] [--debug] [--quiet] <image file to write> <destination drive device>

A completely fair suggestion. I initially thought we were allowing anything that could be encoded with a QUrl - but it appears we're constricting that to actual files, which may be unnecessary.

As such, that change is approved - it's hard for a new user to discover or be driven down that path by a malicious actor, and provides a mechanism to allow a more advanced user that has sufficient trust in the remote to bypass the a few actions.

[tdewey-rpi]

Took another look at this today - I think it might already be implemented.

@Vincent-Stragier Have you tried supplying a HTTP or HTTPs URL to the CLI Imager?

https://github.com/raspberrypi/rpi-imager/blob/06abbadbb501869c29076df0f0409b977869e992/src/cli.cpp#L93

[tdewey-rpi]

No reply in 2 weeks, closing as stale - I believe this function is already available on the CLI imager.

[Vincent-Stragier]

No reply in 2 weeks, closing as stale - I believe this function is already available on the CLI imager.

Yes, sorry, I didn't take the time to test it up to now. It works, at least on Debian. As you said, we can use the CLI Imager.

For those who would like to use it, you can get the list of mandatory and optional argument via a terminal:

$ rpi-imager --help
QSocketNotifier: Can only be used with threads started with QThread
rpi-imager [--debug] [--version] [--repo <repository URL>] [--qm <custom qm translation file>] [--disable-telemetry] [<image file to write>]
-OR- rpi-imager --cli [--disable-verify] [--sha256 <expected hash>] [--debug] [--quiet] <image file to write> <destination drive device>

So you would at least need to input the following command and arguments:

rpi-imager --cli <image file to write> <destination drive device>

Where <image file to write> is to be replaced with the URL of the image, and <destination drive device> is the path to the SD card you want to write on.

With Debian based systems, you can use lsblk to find your SD card's name:

$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0         7:0    0     4K  1 loop /snap/bare/5
loop1         7:1    0  74.2M  1 loop /snap/core22/1380
loop2         7:2    0 505.1M  1 loop /snap/gnome-42-2204/176
loop3         7:3    0  91.7M  1 loop /snap/gtk-common-themes/1535
loop4         7:4    0  38.8M  1 loop /snap/snapd/21759
loop5         7:5    0 100.6M  1 loop /snap/prusa-slicer/96
sda           8:0    0   1.8T  0 disk 
├─sda1        8:1    0   1.8T  0 part /
└─sda2        8:2    0   8.8G  0 part [SWAP]
mmcblk0     179:0    0  59.5G  0 disk 
├─mmcblk0p1 179:1    0   256M  0 part 
└─mmcblk0p2 179:2    0   3.2G  0 part 

In my case the device name was mmcblk0 which meant I had to write the following command:

rpi-imager --cli https://github.com/interactionresearchstudio/NaturewatchCameraServer/releases/download/v1.7/mynaturewatch-camera.img.gz /dev/mmcblk0

You will be prompted with a pop-up window asking for your password, you can also run the command as root or with sudo, but as you are going to erase the drive or the SD card, it's way better to do it as an unprivileged user and to take the time to check the path to your device (e.g. /dev/mmcblk0).