Open meltwater2000 opened 10 years ago
Any suggestions as to how you'd like to indicate that silentinstall should install both the OS and the data_partition? :) Or should silentinstall be modified to always automatically install all non-bootable "distros" alongside the single bootable distro?
Currently what happens if more than one OS is present? (If I recall correctly nothing is installed). Would there be an issue if it installed any which are present? Thanks TimC
Andrew Scheller notifications@github.com wrote:
Any suggestions as to how you'd like to indicate that silentinstall should install both the OS and the data_partition? :) Or should silentinstall be modified to always automatically install all non-bootable "distros" alongside the single bootable distro?
Reply to this email directly or view it on GitHub: https://github.com/raspberrypi/noobs/issues/196#issuecomment-47242328
IIRC if more than one OS is present, it simply 'aborts' the silentinstall and simply loads the NOOBS GUI as normal.
I'd be very hesitant for silentinstall to simply install all OSes present on the card, because it would likely lead to users not understanding what it's intended for, and trying to trigger silentinstall without actually deleting any OSes from the SD card, which is likely to them then running out of diskspace?
The primary goal is for NOOBS to be noob-friendly, not necessarily be an auto-installer for lazy advanced users ;-)
Yes that is fair enough, want to keep the tool easy to use and as clear as possible.
Certainly the ability to install any non-OS partitions would be very helpful. Would an additional command line option be a way to do it, --forceinstallall or something like that?
Or a command line option to specify which OS is installed during a silent install (if option not present on the commandline it installs as before if only one OS is present)? In the case of non-silent installs this command line could pre-select (tick) the specified items (allowing a preselected recommended flavour or setup without forcing the user). eg. --install Raspbians XBMC Raspbian_Scratch
Yeah, I guess the latter option may work, and would potentially even allow silentinstall to work with NOOBS lite :-)
This is using kernel-command-line syntax (see https://github.com/raspberrypi/noobs/blob/master/recovery/cmdline.txt ), rather than application-command-line syntax, so it could be something like install_os_list=raspbian,data_partition,openelec
. Or perhaps install_flavour_list=Raspbian_-_Boot_to_Scratch,Data_Partition,OpenELEC
given that each os subdirectory can contain multiple flavours.
Although maybe there also needs to be another cmdline.txt option which specifies whether to install the OS contained on the card, or the latest OS available from the internet...? (silentinstall_use_internet=1
?)
Always a bit of a balancing act between simplicity & flexibility ;-)
Certainly the ability to install any non-OS partitions would be very helpful.
Just open the partitions.json in the folder of the OS being silent installed and add to the list:
,{
"label": "mypartition",
"filesystem_type": "ext4",
"partition_size_nominal": 512,
"empty_fs": true
}
empty_fs
is an undocumented option that was used during testing to indicate a partition does not have a corresponding tarball with the contents of the partition.
Aaah, handy :) I guess if you wanted to you could also modify partition_setup.sh to automatically add this new partition to /etc/fstab too - e.g. you could add something like this for Raspbian (between the sed and umount lines) - but beware that this is untested
mkdir /tmp/2/mnt/mypartition
echo "$part3 /mnt/mypartition ext4 defaults 0 0" >> /tmp/2/etc/fstab
The flexibility selecting the distros has is excellent, and I think avoiding the need to take drastic action of deleting stuff off the card is a good plan (those who have brought pre-loaded cards won't be keen to remove things off it - I know it is easily replaced).
"Silent install latest from the internet"...now that IS fancy! In fact, I think that might become a very popular way to do it once you've decided what kind of setup you want, avoiding the need to continuously check if the SD card is the latest, while being able to pop the card in and let it run (good for schools/workshops). Just need the satisfying "Wall-E charged" sound when it is cooked!
maxnet - empty_fs, that is very useful and also solves the original question for me. I'd not looked too closely at that file, and that is perfect for what I wanted.
Hmmm, the other advantage of silentinstall is that it auto-boots that OS when the installation is finished - so I guess if we allowed silentinstall to install multiple OSes, we'd also need another option to tell it which of those OSes to boot once it'd finished installing everything? :-S
Maybe it'd be better to take a different tack and ask why you think editing recovery.cmdline to add half a dozen new options (before booting NOOBS for the first time), would be any easier than just letting NOOBS load as normal and using the GUI to select which OSes you want to install? ;-) (i.e. what's the use-case that the current system doesn't solve?)
That is a good point - these are my use-cases, hopefully you can see my reasons (hopefully they make good sense). I am happy to accept NO, since I fully understand NOOBS is there to keep it simple not make it more complicated. Just focusing on ways to improve NOOBS and make it easier to manage setups.
USE-CASE 1 Flavours: The route to this was being able to setup a magic NOOBS install which is configured for whatever purpose needed (workshop setup, automated demo etc etc). That is covered by noobs-config very nicely, the part of the puzzle here is the automated install part which would allow a card to be preconfigured and copied to an SDCard, inserted and powered on (without the need to setup the display or interact with it). So far I've seen schools having a Master SDCard which they clone for everything, which is fine, except you run into issues with needing different versions for different things, with different sized SDCards, when upgrading and if it fails or needs to change, managing multiple 8Gb images etc etc. There are lots of issues related to this which can be avoided. The way I have it working currently, is I can setup a different flavour for each particular setup I want, right down to:
Now you might say this is an advanced use, but this would allow the configuration to be set-up once and shared as a NOOBS setup. As always the hard part is creating the setup, but that would need to be done anyway, the difference is you do it once, rather than each time you do a new install, also you can include/exclude it if you want by selecting the flavour you need. So you end up with one setup which you can manage a lot easier (it would even allow you to manage the setup via Github if you wanted).
The "advanced" part for a typical user is copying it to an SDCard and switching the RPi on, but it allows the complex parts to be done beforehand...as NOOBS should be!
USE-CASE 2 Dual Boot: It is not uncommon to have a RPi dedicated to a specific task, be it a media center or otherwise. It would make sense for these to also be able to have a Raspbian install on (or preferred OS) so that it can be also used for other purposes. My particular case I am thinking of is the RPi which is in a kids bedroom serving video via XBMC, it would make sense to have Raspbian available to switch to by the kid without messing around with the RPi itself. However, you can imagine if that card gets messed up, you want it up and running quickly (typically the time you have available to redo things is when the kids want to use it), so an automated NOOBS install that allows multiple OS selection would be very helpful. Auto-Boot selection, can simply be based on first OS listed. [A side thing here would be boot selection via GPIO...but that is another project I've not looked at yet]
USE-CASE 3 Data-Partition: A side use-case, is the data-partition which is a good way to help manage user files on an SD-Card, it provides a clear place to put files which you know you want to keep/backup etc etc. Also as a common drive between distros (if you have more than one). I like this idea, I use it all the time and it should be encouraged as a general practice (linux is too easy to spread your user data all over the place and it can quickly start to cause issues for beginners not used to it - particularly if they start fighting user permissions etc).
[To play advocate here (not suggesting this should be done)...once NOOBS has been put on a card, there should be no reason for a "NORMAL" user to touch it (and that appears to be generally preferred thing so we "could" move it). Q: How feasible would it be to have the user data partition, as FAT32 and the 1st partition on the card (hiding the Recovery partition away)? I assume it would still need at least one file in the 1st partition to boot initially, but it would make the user data FAR easier to backup and manage (since it would be viewable on any computer including windows). I always prize the user-data as the most important information since everything else can be replaced, so making it easy to backup and keep is paramount. Also by allowing scripts to be edited on any computer means you don't need a Pi setup for every single child, they can edit their code and try it out when ready (even if they end up fine tuning it directly on the Pi after). With the right setup, the Pi could be used headless and used as more of an embedded controller too, without the hassle of switching cables etc.]
I'm not sure it's possible to put a "pure" data partition in Partition 1 because you need the kernel files there to boot from at least, and moving the recovery images to another "hidden" partition would add to the complexity and make customisations of the images (and noobs-config) harder to manage.
However an alternative might be to have an option for NOOBS to reserve a specified amount of space on the recovery partition for a /USR folder before it resizes the partitions. This could be mounted (-o bind) by a distro later. This makes a data folder partition available on the FAT32 recovery partition to be pre-written onto the card easily from Windows beforehand, plus maintains the customisation of the images using noobs-config or by adding/replacing other distro images or flavours as required by advanced users (for the use of noob-users).
Probably best keep the 1st partition User partition as a side comment, as it is outside the original goal of the silent install OS options. I like the idea of the user folder (linux is so flexible!), and that is something I can experiment with without polluting NOOBS with these dark ideas [yet]. Is there a way to specify how what size the recovery partition ends up as, or is that hardcoded (size of files + xMb)?
First off the bat: writing to Partition 1 is never going to happen, because of the NOOBS prime directive https://github.com/raspberrypi/noobs/wiki/NOOBS-partitioning-explained#noobs-partitioning
USE-CASE 1 Flavours:
Sounds like you basically want to be able to silentinstall a particular flavour (i.e. just one), whilst still leaving all other OSes/flavours available on the card for later (re)installation? Sounds like a reasonable request. (and this could possibly also cover the NOOBS-Lite case - perhaps the logic could be that it'll only download from the internet if the OS isn't already available on the SD card.)
USE-CASE 2 Dual Boot: if that card gets messed up, you want it up and running quickly
I still don't see how pre-customising recovery.cmdline is any quicker than just using the regular NOOBS installation GUI in this scenario. It's not as if the NOOBS GUI is particularly complicated...
[A side thing here would be boot selection via GPIO...but that is another project I've not looked at yet]
Yeah that's entirely possible (http://www.raspberrypi.org/forums/viewtopic.php?f=91&t=71107&p=568213#p568213 may give you a starting point) - but if this is a feature you really want please open a separate issue for it.
USE-CASE 3 Data-Partition:
@maxnet has already showed one easy way to add this, but note that @maxnet 's method creates the data-partition as being logically 'owned' by Raspbian, rather than logically being a standalone partition (not that it makes much difference in practical terms!). And don't forget that every time you install/re-install an OS, the Data-Partition gets wiped too.
One thing I've been considering (but not in great depth yet) is some way to add the Data-Partition (if selected) to the partition_setup.sh call, so any OSes installed at the same time could automatically mount it (via /etc/fstab) if available, rather than leaving this as a manual task for the user.
linux is too easy to spread your user data all over the place and it can quickly start to cause issues for beginners not used to it
Hmm? There shouldn't be any reason for them to store files outside of their /home/pi user folder? Unless they get too carried away with using sudo to store files in places they're not supposed to...?
How feasible would it be to have the user data partition, as FAT32 and the 1st partition on the card
Never gonna be the first partition on the card, but you're welcome to switch it (for your local setups) to FAT32 if you want. I wonder if using the Windows "Disk Management" GUI allows you to (manually) mount FAT partitions on an SD card beyond the first one?
In summary: I'm happy to consider options for silent-installing a single flavour/OS out of all those available, and I'm happy to also consider an option for installing non-bootable OSes (i.e. Data Partition) at the same time, but IMHO installing multiple bootable OSes is beyond the scope of silentinstall, and is a level of complexity that should be left to the normal NOOBS GUI.
Agreed. x=100MB, but should be easy to make it settable.
Is there a way to specify how what size the recovery partition ends up as, or is that hardcoded (size of files + xMb)?
https://github.com/raspberrypi/noobs/blob/master/recovery/initdrivethread.cpp#L177
Lol - our posts keep crossing! I was agreeing to Meltwater's side issue :-) and #L177 was where I got 100MB from.
@Lurch - some minor comments:
"I still don't see how pre-customising recovery.cmdline is any quicker than just using the regular NOOBS installation GUI in this scenario. It's not as if the NOOBS GUI is particularly complicated..."
This would be useful if you want to install noobs in an unattended headless situation without requiring the vnc option.
"One thing I've been considering (but not in great depth yet) is some way to add the Data-Partition (if selected) to the partition_setup.sh call, so any OSes installed at the same time could automatically mount it (via /etc/fstab) if available, rather than leaving this as a manual task for the user"
I am half way though writing a script to do this (for Raspbian) already. Can share when it's done.
Given your other comments, would you consider a cmdline option to specify some additional reserved space on the recovery partition i.e. make+xMB settable in #L177?
That is excellent! No problem with the prime directive, I can see good reason for it and would need very careful use if it wasn't followed.
USE-CASE 1: Sounds good. The getting from internet could work, although would depend how it would deal with flavours in that circumstance (since the flavour would be in the directory which you'd need to delete). For NOOBS lite, it could grab the OS automatically as you say. For standard NOOBS an option to update the current packages on the card if different to the latest would be a helpful option (that operation could be independent to the silent install itself - if set to run first, i.e. update_distros). This is something which I'd probably end up keeping enabled so that I never installed an out of date distro (network availability dependent).
USE-CASE 2: The GUI isn't complicated, but quickly finding a screen/keyboard/mouse can be (number one issue at workshops etc). For me, the silent install allows a card to be setup without anything more than a RPi and power (or if being really lazy, a SSH session and reboot). By allowing multiple OS setups to also be done through silent install, you are more likely to use them as they will be setup the same way each time without thinking about the options you need to do. I guess the concern you have is running out of space on a card, or how to handle when it does. I take it the system knows how much space it needs before running, so it could either run and log an error [I've not looked at how the silent install reports failures] that details the space needed verses available or it could install in order given until there is no space.
USE-CASE 3: Yes, independent partition would be better, particularly if we can protect it from being deleted at some point later (which I assume would be an aim at a future point), or if the user wished to dd it. Happy either way with the current option if not.
And don't forget that every time you install/re-install an OS, the Data-Partition gets wiped too.
Could (ab)use the NOOBS settings partition if you want data to survive a re-install. It is only 32 MB by default though.
Not sure if trusting a SD card to keep your data safe long term is a good idea though. Perhaps it is just me being clumsy, but I managed to break off the corners of several SD cards over time, resulting in the Pi no longer recognizing those cards. Think adding some kind of simple "backup /home to cloud storage" application to the Raspbian desktop might be more beneficial.
@maxnet getting the data in one place is a good start. So far my OS has gone kaka, or SDcard slot broke before I've killed a card, but I use microSD cards so that is probably why - the adaptors are probably less sensitive. The cloud backup is on my to-do list, so no problem there, just fitting other parts together at the moment. That is probably the better way to keep hold of data anyway.
@procount
"I still don't see how pre-customising recovery.cmdline is any quicker than just using the regular NOOBS installation GUI in this scenario. It's not as if the NOOBS GUI is particularly complicated..."
This would be useful if you want to install noobs in an unattended headless situation without requiring the vnc option.
And exactly how often does that happen?!
"One thing I've been considering (but not in great depth yet) is some way to add the Data-Partition (if selected) to the partition_setup.sh call, so any OSes installed at the same time could automatically mount it (via /etc/fstab) if available, rather than leaving this as a manual task for the user"
I am half way though writing a script to do this (for Raspbian) already. Can share when it's done.
Cool. Don't forget that depending upon what other OSes you install, the Data Partition can potentially end up on any partition number.
Given your other comments, would you consider a cmdline option to specify some additional reserved space on the recovery partition i.e. make+xMB settable in #L177?
Not gonna happen as that implicitly encourages people to write to the first partition, breaking the prime directive. But NOOBS is Open Source so you're free to make your own fork... ;-)
@meltwater2000
For standard NOOBS an option to update the current packages on the card if different to the latest
The obvious danger with that is if someone creates a custom-distro and leaves it named as Raspbian - they might be surprised if their customisations suddenly disappear just because the online-Raspbian gets updated! But it could be okay as long as it was an explicitly-chosen option as you said.
The GUI isn't complicated, but quickly finding a screen/keyboard/mouse can be (number one issue at workshops etc). For me, the silent install allows a card to be setup without anything more than a RPi and power
In that particular scenario (and also e.g. for schools), would it be more useful to have an "offline-NOOBS" that you could run on your regular Linux PC/laptop, and this would then install NOOBS and selected distros onto an SD card in your PC's SD card-reader?
independent partition would be better, particularly if we can protect it from being deleted at some point later (which I assume would be an aim at a future point)
Yeah that's on the long-term roadmap. Ever noticed that /dev/mmcblk0p4 is currently unused? ;)
@maxnet
Perhaps it is just me being clumsy, but I managed to break off the corners of several SD cards over time, resulting in the Pi no longer recognizing those cards.
Yeah, obviously with all the NOOBS building&testing I have to do I tend to swap SD cards a lot and the corners on one of my 16GB cards has snapped - very annoying. But it still works in the Pi if I use the "peg technique" to apply pressure to it against the Pi's SD pins :) I've found that swapping a MicroSD card in/out of a MicroSD->FullSD adaptor (e.g. http://swag.raspberrypi.org/collections/frontpage/products/noobs-8gb-sd-card) while leaving the SD-adaptor remaining in the Pi, seems to require much less physical effort, and so hopefully makes damage much less likely? Guess it depends how "robust" the pins inside the adaptor are to multiple insertions / removals.
For standard NOOBS an option to update the current packages on the card if different to the latest
The obvious danger with that is if someone creates a custom-distro and leaves it named as Raspbian - they might be surprised if their customisations suddenly disappear just because the online-Raspbian gets updated! But it could be okay as long as it was an explicitly-chosen option as you said.
Assuming you wouldn't clear the entire Raspbian directory (just updates the files), this shouldn't be a big problem - it should be expected if they select to update it that it would be replaced. Doesn't need to be enabled by default.
The GUI isn't complicated, but quickly finding a screen/keyboard/mouse can be (number one issue at workshops etc). For me, the silent install allows a card to be setup without anything more than a RPi and power
In that particular scenario (and also e.g. for schools), would it be more useful to have an "offline-NOOBS" that you could run on your regular Linux PC/laptop, and this would then install NOOBS and selected distros onto an SD card in your PC's SD card-reader?
Yes an offline NOOBS would also be useful, although that would tie up the computer while it is processing the image rather than adjusting the NOOBS image and letting it install on the Pi itself.
independent partition would be better, particularly if we can protect it from being deleted at some point later (which I assume would be an aim at a future point)
Yeah that's on the long-term roadmap. Ever noticed that /dev/mmcblk0p4 is currently unused? ;)
Thats good then. :)
Given your other comments, would you consider a cmdline option to specify some additional reserved space on the recovery partition i.e. make+xMB settable in #L177?
Not gonna happen as that implicitly encourages people to write to the first partition, breaking the prime directive. But NOOBS is Open Source so you're free to make your own fork... ;-)
Well it doesn't have to be documented as an option...:) What would happen if NOOBS needs to download an additional OS if selected for install, I take it would need to fit onto this partition.
I'm still -1 on allowing NOOBS to silentinstall multiple bootable distros. Exactly how many people do you expect to use NOOBS 'headless'? I guess the only way you have of knowing when it's complete, is when the disk-activity LED stops flashing??!
What would happen if NOOBS needs to download an additional OS if selected for install, I take it would need to fit onto this partition.
Remember the prime directive - nothing ever writes to the first partition :-) If Raspbian 2014-01-07 is available on the NOOBS SD card, but Raspbian 2014-06-20 is available online (and assuming the Pi is connected to the internet via Ethernet) then NOOBS downloads, uncompresses and writes the newer files to the SD card, all without touching the first partition. Exactly the same way that NOOBS Lite works. (it uses the os.json files to work out if the version online is newer than the version available locally)
I'm not quite sure what the purpose of the silent install is then? Just seems odd not to allow it to perform the same function as the GUI but automatically. Does silent install have a different intended purpose? I think it would be good to encourage the use of dual OS installs, since that would encourage the use of Raspbian even on systems which are mainly used for something else [and use the RPi for its main goal, even through side-effect if needs be!]. Use case 2 is a very real one in my house - I don't want the kids messing with the SDCard and cables, but I would like them to be able to use scratch as well as stream videos via XBMC (this is a good way to allow both without the hassles of setting it up manually) on the same setup. I don't need to be climbing up into a kids bed in order to re-install the OS, and tying up the TV while I do it...or pulling apart the setup to get the SDCad out and tie up my computer instead. Also, automated install means it is easy to keep it consistent, so it is easy to re-apply settings etc.
Does that mean that you won't allow NOOBS to update itself? In order to ensure that people use the latest software, it would be a very helpful feature. Otherwise, that would also mean that an install which occurred on the network would be newer to an offline-install even though it occurred afterwards (and potentially a problem if the older distro has a fault the other does not)? I would have thought that NOOBS would have the right to write to the partition (and it does do that for the cmdline.txt). NOOBS owns the recovery partition so it makes sense it has control of it, manage it and ensure it is updated as required. [Although I didn't realise that NOOBS-lite does it all on the fly - although makes sense]
PS: I do appreciate the time you are taking to discuss all this!
Does that mean that you won't allow NOOBS to update itself?
Correct. "FAT partition the first, NOOBS no writee..." See also #104
NOOBS only writes to recovery.cmdline when it does the initial resize of the FAT partition, i.e. before any OSes have been installed yet (at which point, it doesn't matter if any corruption occurs).
PS: I do appreciate the time you are taking to discuss all this!
Likewise. It's always great get to get all the "design issues" fully thrashed out, before actually starting on implementation :-)
Humm. Really not sure about the non-upgradability, you end up removing a route by which people can obtain the latest versions of the OS. For those who buy pre-configured SDCards because they are not confident enough to write NOOBS to it, they will always end up with out of date OS images. I don't think many will wipe a pre-configured card and update it manually.
The issue with just network install of the latest is you then remove the possibility of installing the same OS version you had previously if there is a failure and you do not have network available at that particular time (also making the OS files on the card useless).
As with all systems, ensuring users are using the latest version of the software is vital for ensuring things remain stable and secure. This is particularly important when it comes to ensuring support for RAM changes, camera driver updates etc etc [we know this has caused issues in the past, including returning of perfectly good units]. I know the system can be updated in situ but often users will choose to make a clean install, which is pointless if they are starting again with an out of date system. You can't complain that people aren't using the latest release if it isn't easy as possible to grab it directly and use it each time. If any thing, NOOBS should be checking and informing users that there is a newer OS version available, so it actively encourages them to update before installing.
If you take just the OS files as something which can be updated, which are SHA-1 checked, the stability of the system can be maintained. If an image fails the check, the image isn't cleared for use and it will fall back to having to get it from the network or re-downloading it. That is very different to replacing NOOBS itself...which in case of failure you would hit an issue.
Anyway, happy to take this as a separate issue another time, since the implementation of this would be a different task entirely.
@lurch - Here is my script to mount the user Data_Partition in Raspbian, no matter which partition it is on: MountUser. It could probably do with some additional error checking to make it robust, but it works for me.
@procount Cool, looks good.
@meltwater2000 To selectively quote from your comment: "vital for ensuring things remain stable" - the only way to do that is to ensure that nothing writes to the first FAT partition (we obviously have no control over the quality of SD card or PSU people choose to use with their Pi). To allow the OS tarballs to be updated separately, without writing to the first partition, would require the OS tarballs to be stored on a separate (non-first) partition, and that's a level of complexity that IMHO just isn't warranted. (and then what would happen if the "updated OS" you want to download is now suddenly too big to fit on the partition you're storing the distro tarballs on?)
While NOOBS may be less than perfect, it's tried and tested :)
I don't think many will wipe a pre-configured card and update it manually.
Is there something missing from the instructions ?
If any thing, NOOBS should be checking and informing users that there is a newer OS version available
It already does that indirectly by displaying a Network-icon in the OS-install-menu rather than a SD-icon, if the version online is newer than the local version. Or are you saying that this "update check" should be done every time the OS-boot-menu is displayed? I suspect that would very quickly get annoying.
Going back to your earlier example of where you wanted to re-install both XBMC and Raspbian-Scratch on your childrens' Pi using silentinstall - you said both "I don't need to be climbing up into a kids bed" and "[I don't need to be] pulling apart the setup to get the SDCad out" so how exactly are you expecting to trigger the silentinstall? I'm not trying to be argumentative, just trying to understand exactly how you expect this to work :) Is there any way that NOOBS could be made simpler, so that your children are able to use it themselves?
Sorry lack of response on this, I got sidelined with holidays etc. We've picked on multiple issues here, so rather than debating all of them (which I am happy to do else where/another time) the route issue was: a) being able to install a specific os from the command line without needing to delete the other os's from the sdcard. b) A bonus would be to be able to specify a data partition to also be installed c) beyond that to specify more than one os from the command-line. I think even having just the first option would be of huge benefit.
Thanks.
So basically, everything that you can do via the NOOBS GUI, you want to be able to automate from a 'settings file' (whether that's cmdline.txt
or something else) ? ;-)
That would be ideal yes and would make sense, but obviously that would be asking for a much larger change.
@meltwater2000 is this still an issue?
Only a single OS/flavour can be selected for silentinstall, this includes the user data partition. Can this be amended?