1.8.1 - Many issues with filtering

[theofficialgman]

There are multiple issues with 1.8.1 and I am combining them all here since I assume they are related:

What is happening: Select Pi5:

• only 64-bit PiOS bookworm images show
• No ubuntu images show

Select Pi3:

• No Bookworm images

Select Pi1:

• No Bookworm images
• 64-bit and 32-bit Ubuntu images show

Expected result: Select Pi5:

• PiOS bookworm 64-bit and 32-bit images show
• Ubuntu Mantic 23.10 Desktop and Server 64-bit image show

Select Pi3:

• PiOS bookworm 64-bit and 32-bit images show

Select Pi1:

• Bookworm images show
• ONLY 32-bit Ubuntu images show

[theofficialgman]

I believe all the problems lie not in the software but the poor/strange choices made in the json lists https://downloads.raspberrypi.org/os_list_imagingutility_v3.json

Pi3 and older have indeed been left out on all bookworm images. Maybe this was an internal decision to not expose it until wayland is available for it.

Pi5 not having any 32-bit images available even in the (other) category makes no sense to me though. I think there are many genuine reasons that advanced users might want a 32-bit image. If they are looking in the (other) category then they are likely looking for it.

As for Ubuntu images not being filtered properly, I am guessing nobody notified them in advance (or they did and ubuntu devs have been slow to implement) that they need to update their automation for creating https://changelogs.ubuntu.com/raspi/os_list_imagingutility_ubuntu.json to add devices

To me devices seems like a poor choice of filtering methods. If I were to implement this I would have broken OS architecture and device model out separately. all options:

"archs": ["armhf", "arm64"]
"models": ["pi1", "pi2", "pi3", "pi4", "pi5"]

and so your device list becomes

        "devices": [
            {
                "name": "No filtering",
                "archs": [],
                "models": [],
                "description": "Show every possible image",
                "matching_type": "inclusive"
            },
            {
                "name": "Raspberry Pi 5",
                "archs": ["armhf", "arm64"],
                "models": ["pi5"],
                "default": true,
                "icon": "https://downloads.raspberrypi.com/imager/icons/RPi_5.png",
                "description": "The latest Raspberry Pi, Raspberry Pi 5",
                "matching_type": "exclusive"
            },
            {
                "name": "Raspberry Pi 4",
                "archs": ["armhf", "arm64"],
                "models": ["pi4"],
                "icon": "https://downloads.raspberrypi.com/imager/icons/RPi_4.png",
                "description": "Models B, 400, and Compute Modules 4, 4S",
                "matching_type": "inclusive"
            },
            {
                "name": "Raspberry Pi Zero 2 W",
                "archs": ["armhf", "arm64"],
                "models": ["pi3"],
                "icon": "https://downloads.raspberrypi.com/imager/icons/RPi_Zero_2_W.png",
                "description": "The Raspberry Pi Zero 2 W",
                "matching_type": "inclusive"
            },
            {
                "name": "Raspberry Pi 3",
                "archs": ["armhf", "arm64"],
                "models": ["pi3"],
                "icon": "https://downloads.raspberrypi.com/imager/icons/RPi_3.png",
                "description": "Models B, A+, B+ and Compute Module 3, 3+",
                "matching_type": "inclusive"
            },
            {
                "name": "Raspberry Pi 2",
                "archs": ["armhf"],
                "models": ["pi2"],
                "icon": "https://downloads.raspberrypi.com/imager/icons/RPi_2.png",
                "description": "Model B",
                "matching_type": "inclusive"
            },
            {
                "name": "Raspberry Pi 1",
                "archs": ["armhf"],
                "models": ["pi1"],
                "icon": "https://downloads.raspberrypi.com/imager/icons/RPi_1B+.png",
                "description": "Models A, B, A+, B+, Zero, Zero W, Zero WH and Compute Module 1",
                "matching_type": "inclusive"
            }
        ] 

and then your OS lists uses the same format

        {
            "name": "Raspberry Pi OS (64-bit)",
            "description": "A port of Debian Bookworm with the Raspberry Pi Desktop (Recommended)",
            "icon": "https://downloads.raspberrypi.com/raspios_armhf/Raspberry_Pi_OS_(32-bit).png",
            "url": "https://downloads.raspberrypi.com/raspios_arm64/images/raspios_arm64-2023-10-10/2023-10-10-raspios-bookworm-arm64.img.xz",
            "extract_size": 5800722432,
            "extract_sha256": "f5236272f5fe2a0e3999c8c0a574772684d3bfd18f395304292ddf59508d5ae1",
            "image_download_size": 1146559440,
            "release_date": "2023-10-10",
            "init_format": "systemd",
            "archs": ["arm64"],
            "models": ["pi4", "pi5"]
        },
        {
            "name": "Raspberry Pi OS (32-bit)",
            "description": "A port of Debian Bookworm with the Raspberry Pi Desktop",
            "icon": "https://downloads.raspberrypi.com/raspios_armhf/Raspberry_Pi_OS_(32-bit).png",
            "url": "https://downloads.raspberrypi.com/raspios_armhf/images/raspios_armhf-2023-10-10/2023-10-10-raspios-bookworm-armhf.img.xz",
            "extract_size": 5301600256,
            "extract_sha256": "3b030fabc6f9ed14d2b51315c2001a1a0df0a5128257f45fee78c555d11ebbba",
            "image_download_size": 1299076172,
            "release_date": "2023-10-10",
            "init_format": "systemd",
            "archs": ["armhf"],
            "models": ["pi4", "pi5"]
        },

               {
                    "name": "Raspberry Pi OS (Legacy)",
                    "description": "A port of Debian Bullseye with security updates and desktop environment",
                    "icon": "https://downloads.raspberrypi.com/raspios_armhf/Raspberry_Pi_OS_(32-bit).png",
                    "url": "https://downloads.raspberrypi.com/raspios_oldstable_armhf/images/raspios_oldstable_armhf-2023-10-10/2023-05-03-raspios-bullseye-armhf.img.xz",
                    "extract_size": 4143972352,
                    "extract_sha256": "b68288da5133e8acf87a656f8a03526ef5cb8b7d7521833ef9bfe5df832e7346",
                    "image_download_size": 914230732,
                    "release_date": "2023-05-03",
                    "init_format": "systemd",
                    "archs": ["armhf"],
                    "models": ["pi1", "pi2", "pi3", "pi4"]
                },

[tdewey-rpi]

Thanks for the report, @theofficialgman

This is a pretty big report, so I'll break this comment into chunks:

RE: Pi3 and older, Bookworm

We are indeed not recommending Pi3 and older migrate to Bookworm at this time. We may revise that recommendation if we gain confidence these platforms will work well with Bookworm.

RE: Ubuntu

You're correct in that we're waiting for the Ubuntu team to update their sublist JSON file. They've been provided directions for how to do this, and we're looking at how we can better work with iamge providers when it comes to Imager and JSON updates.

RE: Pi5, 32bit Bookworm

This is a genuine mistake on my part. I'll be publishing a revision to the JSON file later today with additional feedback incorporated.

RE: Filtering mechanism

I'm happy to review a PR, however the design imperative for this system was to introduce the simplest possible mechanism to acheive our UX goal (specifically, to try and ensure Pi5 users get a simple path to selecting images that work for them, and pre-Pi4 devices get a simple path for selecting images that work for them, too).

I hope this comment offers some insight. Let me know if there's anything particularly unclear, or if you think I've made a bad call.

[TerraTech]

zero 2w: doesn't offer the "Bookworm" images, only the legacy Raspberry Pi Bullseye images. This Raspberry Pi OS download page does show that "Zero 2W" is compatible with Bookworm, plus I've been running it for a few weeks now without any problems.

Today I upgraded the imager from v1.7.5 ==> v1.8.1, and I could select "Bookworm" on the older v1.7.5 but not now. I would consider this to be a regression with the rpi-imager.

Until there is a solid reason for Bookworm's exclusion, I'll revert back to v1.7.5.

[lurch]

Until there is a solid reason for Bookworm's exclusion, I'll revert back to v1.7.5.

Zero2W falls under the same "Pi3 and older" category, as mentioned above.

RE: Pi3 and older, Bookworm

We are indeed not recommending Pi3 and older migrate to Bookworm at this time. We may revise that recommendation if we gain confidence these platforms will work well with Bookworm.

[maxnet]

Not "recommending" and not showing it at all do are two different things. Can't you make a subfolder for "not recommend" ( but technically able to run) hidden at the bottom?

[tdewey-rpi]

Thanks for the report, @TerraTech

zero 2w: doesn't offer the "Bookworm" images, only the legacy Raspberry Pi Bullseye images. This Raspberry Pi OS download page does show that "Zero 2W" is compatible with Bookworm, plus I've been running it for a few weeks now without any problems.

Could you please confirm which flavour of Bookworm you're running on Zero 2 W - lite, "regular", or full?

Today I upgraded the imager from v1.7.5 ==> v1.8.1, and I could select "Bookworm" on the older v1.7.5 but not now. I would consider this to be a regression with the rpi-imager.

You could still select Bookworm by opting out of filtering - which should be the first item of the Raspberry Pi Device selection dialog. If you tried this mechanism, did it not work as you expected?

Until there is a solid reason for Bookworm's exclusion, I'll revert back to v1.7.5.

As above, I don't believe you'd need to go back to v1.7.5 (there are numerous improvements beyond device filtering in v1.8.1), as we provide a mechanism to opt out of the Device Filtering entirely. If that's not working, I'd request you file an additional issue to track that.

[tdewey-rpi]

Not "recommending" and not showing it at all do are two different things. Can't you make a subfolder for "not recommend" ( but technically able to run) hidden at the bottom?

If you want to opt out of the filtering entirely, you can do so by selecting "No filtering, Show every possible image" at the top of the device selection screen. My initial thinking was that this would be more durable and simpler to use than a catch-all mixed category in the OS list - which would raise all sorts of additional problems (do you preserve heirarchy, do you offer grades of compatibility, etc).

[maxnet]

If you want to opt out of the filtering entirely, you can do so by selecting "No filtering, Show every possible image" at the top of the device selection screen.

Problem is that it is not clear to the end-user at all that you are hiding things that do can run on the device, and that need to be unlocked by using that option. If I select a device I have the expectation that it displays all images that can technically run on my device.

[tdewey-rpi]

If you want to opt out of the filtering entirely, you can do so by selecting "No filtering, Show every possible image" at the top of the device selection screen.

Problem is that it is not clear to the end-user at all that you are hiding things that do can run on the device, and that need to be unlocked by using that option. If I select a device I have the expectation that it displays all images that can technically run on my device.

Unfortunately you've hit the exact problem - what people expect when we assert it "runs". Running a lite image is a very different proposition to running a full fledged desktop, with numerous factors that would impact a user's perception of how it "runs". In the current filtering configuration, we've tried to make choices that result in users getting a simple path to a good experience - a higher bar than "technically running" - but we've also retained the option for users who want to ignore the recommendations and exercise a wider choice to do so.

In any system where you're providing a subset of options to a user, there's a discoverability cliff. In our current UI paradigm, I hoped this solution would present the smallest obstacle - in particular, because in the OS list you still retain the "Custom Image" flashing mechanism regardless of your choice of device filter.

Of course, the obvious criticism there is there are multiple quite separate paths to getting to the same solution - and I'd accept that, and say that the choices I've made so far unashamedly aim to give priority to the experience for newer and less technical users, on the assumption that more technical users will be more capable of finding those alternative paths.

[theofficialgman]

RE: Pi5, 32bit Bookworm

This is a genuine mistake on my part. I'll be publishing a revision to the JSON file later today with additional feedback incorporated.

Still don't see the change yet. Hopefully you can do that at the start of the workweek.

RE: Filtering mechanism

I'm happy to review a PR, however the design imperative for this system was to introduce the simplest possible mechanism to acheive our UX goal (specifically, to try and ensure Pi5 users get a simple path to selecting images that work for them, and pre-Pi4 devices get a simple path for selecting images that work for them, too).

Your current system works it is just not what I would have done. Thats the only reason I detailed what I would have done.

[holta]

This challenging design debate is super-interesting, enlightening and important!

So many people have earnestly contributed in good faith over countless years.

Thanks everyone (including @maxnet, @tdewey-rpi and many others) for very sincerely persisting and respecting everyone's heartfelt beliefs along the way,

🕊️

[tdewey-rpi]

RE: Pi5, 32bit Bookworm This is a genuine mistake on my part. I'll be publishing a revision to the JSON file later today with additional feedback incorporated.

Still don't see the change yet. Hopefully you can do that at the start of the workweek.

Apologies, yes, this has fallen behind schedule. I'm expecting to drop an update circa 8am GMT on Monday, which should resolve this. For now, as a workaround, please select "No filtering" if you want the 32bit Bookworm on Raspberry Pi 5.

RE: Filtering mechanism I'm happy to review a PR, however the design imperative for this system was to introduce the simplest possible mechanism to acheive our UX goal (specifically, to try and ensure Pi5 users get a simple path to selecting images that work for them, and pre-Pi4 devices get a simple path for selecting images that work for them, too).

Your current system works it is just not what I would have done. Thats the only reason I detailed what I would have done.

Completely fair. Your point on architectural filtering is well noted - there are additional dimensions that I'd also like to consider (most notably, minimum fitted RAM size) - but I'd rather we had a basic mechanism in place for the Raspberry Pi 5 launch, where at least initially the choice of image will be important for getting a great experience.

Thankfully, there'll be further opportunities to revise this mechanism - but I'll be considering that work against other UX work and projects.

@holta Thank you for the comment. I'm confident that anyone who comes to this repository is only looking to create the best version of this software, and that between our respective viewpoints is the chance to build a great experience for the users of the Raspberry Pi Imager.

[TerraTech]

Thanks for the report, @TerraTech

Thank you for your insightful response.

zero 2w: doesn't offer the "Bookworm" images, only the legacy Raspberry Pi Bullseye images. This Raspberry Pi OS download page does show that "Zero 2W" is compatible with Bookworm, plus I've been running it for a few weeks now without any problems.

Could you please confirm which flavour of Bookworm you're running on Zero 2 W - lite, "regular", or full?

On one board, I run the lite version, and the other I use regular. I have never used the full version.

Today I upgraded the imager from v1.7.5 ==> v1.8.1, and I could select "Bookworm" on the older v1.7.5 but not now. I would consider this to be a regression with the rpi-imager.

You could still select Bookworm by opting out of filtering - which should be the first item of the Raspberry Pi Device selection dialog. If you tried this mechanism, did it not work as you expected?

If I turn off filtering, I can find the Raspberry Pi OS Lite (64-bit) image.

Until there is a solid reason for Bookworm's exclusion, I'll revert back to v1.7.5.

As above, I don't believe you'd need to go back to v1.7.5 (there are numerous improvements beyond device filtering in v1.8.1), as we provide a mechanism to opt out of the Device Filtering entirely. If that's not working, I'd request you file an additional issue to track that.

Agreed, knowing that filtering can be disabled, I can still get to the Bookworm image I want to use. I believe that there was a subtle misunderstanding on my part, that when selecting the board, there were (perhaps) some minor tweaks made during the process for the targeted board. I now understand it is for 'filtering' only and nothing beyond that.

[denverpilot]

Noticed that with the current filter breakout there's no specific option for the Pi Zero W, only the 2 W.

[tdewey-rpi]

Noticed that with the current filter breakout there's no specific option for the Pi Zero W, only the 2 W.

Thanks for the report, @denverpilot. That's correct today - for filtering purposes, the Zero W can be considered in the same bracket as the original Raspberry Pi - because they use the chip (albeit at different performance points). If you don't find the device you're looking to create an image for as a headline, that's likely because I've grouped it with others for filtering purposes.

Let me know if you think it should be broken out. It's entirely possible that I've missed something.

[tdewey-rpi]

Could you please confirm which flavour of Bookworm you're running on Zero 2 W - lite, "regular", or full?

On one board, I run the lite version, and the other I use regular. I have never used the full version.

Thanks for the extra detail, @TerraTech. I'll follow up internally to see if we can vary this filtering.

[tdewey-rpi]

@theofficialgman

Rapsberry Pi 5 will now expose Bookworm 32bit, and Raspberry Pi 4 and earlier will also show Bullseye at the top level, to help people flashing older devices. Please let me know if this goes some way to addressing your concerns - I'm going to follow up on Zero 2 W's Bookworm recommendation.

On the Ubuntu listing, the resolution is essentially blocked on their server update. I expect I'll know it's fixed when you do!

[denverpilot]

@tdewey-rpi hi Tom. Welll… I’m an old hat at Linux so I was able to ready various places and kinda figure out the status of the perennially orphaned regular Zero W. Ha.

As far as user experience goes, if a non Linux user attempts to load a regular zero W they have lying around for whatever reason, the main website currently says two things:

• All versions of the pi are supported by the latest release, no caveats no links to any release notes or hints that might not be the best idea ever.
• Load up our flashing tool and it’ll guide you.

Then they don’t see a regular Zero W in the filter options. If they try a 2W thinking it looks close enough, their pi won’t boot…

As they say, not my circus and not my monkeys and I know you all have enough work with just getting docs somehow caught up to the big change … so I’m just sharing the above scenario as what I think the average end user would hit.

I almost fell into the trap myself but knew enough about Linux that I knew it was just related to the big core switch and went a-hunting. And I’d also seen various other projects drop zero W support because the upstream build tree is so goofy for it compared to the other pis. (Grin)

I had to tell a few folks this weekend certain downstream userlsnd project docs are wrong, because they literally say “get rpi imager and follow it, then come back here and do X”. Asked them if they had regular zero W and recommended they just go grab a 2W and save themselves far more than $15 US messing with their old one. (Grin… or call me and I’d tell em which image to load and then they can go from there. Most of those userspace projects still assume bullseye right now anyway. Best they avoid all of this while the dust settles on both OS and project side.)

Sorry. Wordy. tl;dr version: Current main website instructions don’t work with a real Zero W. :-)

[tdewey-rpi]

Thanks for the report, @denverpilot.

That's a pretty interesting report - we do list Zero W (see attachment), but for filtering purposes it's treated like the original Raspberry Pi. My hope was that people would scroll the list and discover the additional options.

Your report, however, suggests I've not done enough to make the groupings clear enough. Are there specific changes that you think would improve discoverability of the model groupings?

Alternatively, we could consider breaking our the Zero devices into their own families.

Let me know which you think would be better.

[tdewey-rpi]

I understand that as of writing, Canonical have updated their OS list to include device tags. Raspberry Pi do not directly control the tags they've chosen - so their lack of support for the original Raspberry Pi may suprise you, @theofficialgman.

Unfortunately, this project isn't the place to raise issues with the tags we will be supplied by third parties - we don't have control over the JSON they supply. That said, I think they've made sensible choices that mean Raspberry Pi users will get a good experience with their OS when using the device filters.

[maxnet]

That's a pretty interesting report - we do list Zero W (see attachment)

Besides the scrolling issue (which may be solvable by multiple columns), having an icon of a normal full-size Pi also doesn't help.

[denverpilot]

Wow @tdewey-rpi I have some serious egg on my face. I’ll be honest, I’m usually pretty careful to read things — but my brain made no connection between “Raspberry Pi 1” and the Zero W, after scrolling past the prominent Zero 2 W item in the list.

I doubt my eyes (maybe) even made it past the Pi 2 honestly, although I probably did scroll far enough.

Not sure I’m qualified to suggest a correct solution, considering even the Compute Module is in that list too.

I certainly should have paid better attention however, and I appreciate you letting me bend your ear a little and hope I am not wasting your time on more important things.

I can change by advice to newbies to scroll all the way down and really look for the Zero W. It’s there… and hint that it’s listed next to the Pi 1 label.

Shaking my head at myself… completely missed it. My brain was just expecting an item per Pi I guess. But I see Pi 4 and 3 entries also include the Compute Modules… and I understand why under the hood. Hmmm.

Each line item is a processor family, but the typical user doesn’t truly understand that. I definitely should have noticed it at the 4 and 3 entries.

Being extremely pedantic I also understand how the “Zero 2 W” ends up above “3” in the list — but could insert the Microsoft numbering scheme meme as a lighthearted joke … here! Hahaha.

5, 4, Zero 2W, 3, 2, 1… giggle. Poor Zero he’s so weird. And unique. Hahaha.

I’m going to blame this one on the bifocals. Yeah, that’s it! Cheers!

[tdewey-rpi]

@denverpilot No egg that I can see - just another sharp edge in the UI that we can try and round off.

I'll make a change today to break out the 2835-generation Raspberry Pi Zero devices into their own category, with an icon appropriate. These will go linto the list above the original Raspberry Pi (so, penultimate).

Hope that'll help.

[tdewey-rpi]

@denverpilot Based on your feedback, I've broken Raspberry Pi Zero into it's own list item, with the matching icon.

Hope this helps spot it in the future.

[tdewey-rpi]

Checking in, do we have any additional feedback on Device Filtering that could do with an explanation, or am I clear to close this issue, and rely on people to raise specific point issues going forward?

[TerraTech]

The only open item for me was the release variance [lite,regular,full], however it doesn't pertain to the original topic. I'd say best advice to close belongs to @theofficialgman. Trying to close an issue that branches out as it progresses is never easy. Any branches left unresolved, it is best to have them made into a new issue (w/linkback) as long as original topic is satisfied. (just my 2cents)

[theofficialgman]

Checking in, do we have any additional feedback on Device Filtering that could do with an explanation, or am I clear to close this issue, and rely on people to raise specific point issues going forward?

I'll close it. The issues that I brought up have been solved one way or another. It will be easier to track suggestions/issues with filtering with individual issues in the future.