NOOBS compatibility problem (failure to adequately handle filesystem_type=raw?)

[whitpa]

Our (thinlinx.com's) O/S, TLXOS, fails to boot under PINN, resulting in an infinite reboot loop. This worked just fine under NOOBS, but not under PINN.

I have found that the problem seems to be that (a) PINN will not recognize as bootable any partition that doesn't have a FAT16/32 partition type, and (b) PINN identifies the bootable TLXOS partitions as type 83 (Linux). This is almost certainly because our filesystems are installed raw (filesystem_type == raw in partitions.json) rather than using tarballs. FYI our bootable partitions (there are two of them, a primary boot and a recovery partition) are both FAT16 LBA, i.e. type == 0xe. We are setting "default_partition_to_boot" correctly in [/dev/mmcblk0p5/]noobs.conf, but this makes no difference.

Could you please add code to run some kind of filesystem type detection in the event that filesystem_type == raw (e.g "blkid -p" will tell you that this is TYPE="vfat", VERSION="FAT16") and set the partition type code accordingly?

[procount]

Hi, This is very surprising as I'm sure it always used to work, although I haven't tested it for ages. It should work the same as NOOBS but maybe one of my enhancements is incompatible with your image. Have you updated it recently? I will take a look and see what the issue is.

[procount]

I think I found the issue. Your partitions.json file is missing the "partition_type" attribute for your raw partitions, hence PINN is using the default partition type of 83. Looking at the code for NOOBS I suspect the same would happen there if you tried it with your latest installation as I can't see any major differences. I see you updated Thinlinx recently on 01-02-23. Maybe this was omitted on your last update? 🤷‍♂️

You'll need to add something like: "partition_type" = "0e", to each of your raw partitions. The value should be the hex number as used by the fdisk type value, but enclosed in quotes as it is treated as a string in partitions.json.

By the way, if you do update your image, please drop me a line so that I can update the catalog of OSes to reflect your current version. Ideally if you could send me an os_list.json file I can update it very easily. This contains minimal information from os.json and partitions.json so PINN can display the correct information and knows how to download the appropriate files.

Below is the current version of this file for Thinlinx from NOOBS, which you can see is a bit out of date:

        {
            "description": "30-day trial of ThinLinX's Debian-based thin client for Raspberry Pi 2, 3 and 4",
            "group": "General",
            "icon": "http://rpi.thinlinx.com/NOOBS/TLX_RPi/TLXOS.png",
            "marketing_info": "http://rpi.thinlinx.com/NOOBS/TLX_RPi/marketing.tar",
            "nominal_size": 2064,
            "os_info": "http://rpi.thinlinx.com/NOOBS/TLX_RPi/os.json",
            "os_name": "TLXOS ",
            "partition_setup": "http://rpi.thinlinx.com/NOOBS/TLX_RPi/partition_setup.sh",
            "partitions_info": "http://rpi.thinlinx.com/NOOBS/TLX_RPi/partitions.json",
            "release_date": "2021-12-11",
            "supported_models": [
                "Pi 2",
                "Pi 3",
                "Pi Compute Module 3",
                "Pi Zero 2",
                "Pi 4",
                "Pi Compute Module 4"
            ],
            "supports_backup": "update",
            "tarballs": [
                "http://rpi.thinlinx.com/NOOBS/TLX_RPi/boot.xz",
                "http://rpi.thinlinx.com/NOOBS/TLX_RPi/tfm.xz",
                "http://rpi.thinlinx.com/NOOBS/TLX_RPi/root.xz"
            ],
            "url": "https://thinlinx.com/",
            "version": "4.10.1"
        },

Typically, only the nominal_size, release_date and version will need updating on each new release, but maybe others depending on your upgrade.

[whitpa]

Thanks, I didn't even know "partition_type" was even a thing (there being a complete lack of documentation on the JSON file syntax, such that unless we have lots of spare time to read unfamiliar source code, we can only copy what everybody else is using in their JSON files (and no-one else uses raw filesystem images)). NOOBS used to work without this, but we haven't really done anything with NOOBS for a long time, our current efforts being focussed on Raspberry Pi Imager.

I have now added the "partition_type" tag to our live partitions.json files.

All of the rpi.thinlinx.com URLs in the above os_list.json fragment (presumably scraped from http://downloads.raspberrypi.org/os_list_v3.json, which we didn't bother to get updated for the latest releases, NOOBS being pretty much dead by then) actually point to our 4.11.1 release at present (see contents of http://rpi.thinlinx.com/NOOBS/TLX_RPi/os.json and http://rpi.thinlinx.com/NOOBS/TLX_RPi_IoT/os.json for current release details). We continually overwrite these files with the current version.

Since we maintain multiple releases (progressive and conservative (Long Term Stable)) on multiple architectures (Pi 2+ and Pi 1/Zero), our preference would be to use a versionless top-level entry, and then provide a submenu under our own control, the way we do with Raspberry Pi Imager. However, original NOOBS was never capable of that. It was an annoyance to us that NOOBS had no means whatsoever to display the "version" and "release_date" information provided in os.json in leaf-level entries. It looks like PINN still has that problem(?).

[procount]

Although PINN is compatible with NOOBS, I have added a lot of improvements to try and resolve some of NOOBS shortcomings.

The JSON fields are now documented here -> https://github.com/procount/pinn/wiki/JSON-fields There is also some other useful information on my Wiki, but the main documentation can be found at -> https://github.com/procount/pinn/blob/master/README_PINN.md

You effectively have a rolling release as far as PINN is concerned, which will work, but it will just show outdated information on the install screen if the top level information is not maintained. LIke Rpi Imager, PINN can effectively now delegate "sub-menus" to maintainers, This will allow you to update the top level entry and maintain a number of different versions/releases for different RPi models as you wish.

PINN displays the version/release-dates when you hover the cursor over an OS entry. With control over the top level entries you can also put this information in the OS name, or in the description.

If you'd like to maintain your own sub-menus, just let me know. You would need to provide your own os_list_v3.json file and its URL. I can guide you through your initial version if you need it.

If you have any further questions, please don't hesitate to ask me.

[whitpa]

Many thanks for the JSON field doco, this is most helpful. I have now updated our os.json files to be more explicit about supports_backup and supports_usb_boot (I'm still a bit uncertain what the backup feature actually does, but it's presumably a moot point on account of our use of Reiser4). I noticed that you have "emptyfs" rather than "empty_fs" - we've always used the latter. Is this a documentation mistake, or do you support both?

[procount]

Thanks for spotting the typo - it's fixed now. The Backup feature will backup the installed OS in PINN format, i.e. as tar.gz or img.gz files per partition with all associated meta files. To do this, the user must have a USB drive, formatted as FAT32 or ext4, with a /os folder on it where the backup will be stored. Useful for backing up the OS, or cloning to another drive etc.

It works on most OSes but is not guaranteed. Not sure about Thinlinx, especially as it uses ReiserFS so probably wise to set it to false. Might be worthwhile trying it though.

[whitpa]

FYI, I just discovered why our install used to work without partition_type="0e" - I had forgotten that we hacked our way around this by adding an "sfdisk --change-id" command in partition_setup.sh. It seems that NOOBS' initramfs must have included an sfdisk binary/applet and that PINN's doesn't. Now that I've set partition_type="0e", I'll get rid of that code and have partition_setup.sh do nothing (exit 0) instead.

[procount]

Interesting, but PINN does still use sfdisk. I haven't looked at your script but I would have thought it should have still worked.

[procount]

@whitpa - I have updated the os_listing for your ThinLinx OS.

One suggestion would be to choose different names for the TLX_RPi_IoT and TLX_RPi versions, because at the moment only one of them will ever be displayed in the menu, even if PINN's show_all option is selected. This means for example it is not possible to install both versions to the same SD card, use a RPi4 to install the IoT version for a Pi zero or vice versa. It just seems a bit confusing not to differentiate them.

Have you had any more thoughts about hosting your own os_list_v3.json file?

[whitpa]

We recently released new versions of TLXOS, and we're now ready to take you up on the offer of switching to hosting our own os_list_v3.json, so that we don't have to bother you with any future updates.

If I've got the JSON right, you should now be able to remove the TLXOS entries from the master os_list_v3.json, and add the following to your repo_list.json: { "name": "TLXOS", "url": "http://rpi.thinlinx.com/NOOBS/os_list_v3.json" },

Note the use of "http://" rather than "https://". rpi.thinlinx.com is a reverse proxy load balancer, so there will be certificate validation problems if HTTPS is attempted.

As per your suggestion below, I have changed the values of name/os_name so that they're all unique.

P.S. It seems to me that when using os_list_v3.json, os.json is completely redundant. But I'm guessing that we have to retain them because something will break if we remove the os_info tags(?).

[procount]

I shall check these out for you and add them to my repo_list.

os_list.json and os.json are inherited from NOOBS and served different purposes. A local installation uses os.json and partitions.json for its information. os_list.json is a fragment of the combined os_list_v3.json which contained just enough information to display and choose the remote OSes. On installation, the os.json and parittions.json files were still downloaded and used for the installation/configuration of the OS.

If an OS is downloaded to local storage, or backed up, it is the os.json and parittions.json files that are used for the meta-data They are very similar and their subtle differences sometimes drive me mad, but it is what it is. For compatibility reasons I have left them as they are and just extended them with new features.

[whitpa]

Does this mean that it is not necessary to provide the "release_date" and "version" tags in our os_list_v3.json file? I would prefer to update these in the os.json files only, if possible.

[procount]

I will review your files and let you know.

[procount]

I did a quick test by installing the version for the Pi5 and it seems to work ok. Did you ever test the backup feature of PINN? I tried backing up TLXOS to USB card with PINN, and then reinstalling the backup. It also seemed to work ok, so unless you found any corner cases that didn't work, you could enable that feature.

[whitpa]

OK, I have changed "supports_backup" to "true". I don't remember why I disabled this - it might have had to do with the old version of partition_setup.sh that I have since replaced.

[whitpa]

You haven't yet removed the old TLXOS entries from https://raw.githubusercontent.com/procount/pinn-os/master/os/os_3rdparty.json nor added the entry to https://raw.githubusercontent.com/procount/pinn-os/master/os/repo_list.json as I requested above, so I still can't install via PINN. Is there anything more you need from me before you doing this?

[procount]

Sorry, life got in the way 😉 It should be installable now.