GRUB2 update for PUPPY Linux builds

CollaboratorGCM commented 4 years ago

Couple months ago on WOOFCE GIT a discussion emerged on GRUB2 boot manager. GRUB2 distributes in various PUPPY linux distros.

As many of you know, GRUB2 is world-wide open source and has enormous benefit; not to mention the fact that hardware manufacturers the likes of Intel, AMD, Western Digital, Toshiba, RedHat, IBM, etc. contribute support since its inception.

GRUB2 offers ways to boot the various operating systems and ISOs it sees when system starts occur. And a generic one is one that is (or should) be in all of the PUPs; assuming developers can agree to use a common install library structure and modules for filesystem discovery.

Not to offend anyone or discourage choice of boot managers, this thread intends to review if there can be agreement on a common structure and inclusion of GRUB2 in PUPs going forward. I feel we could include its structure to match what we see in other distros as it seems they have adopted a common GRUB2 lib structure.

If agreement, then this opens the door for some more advance uses that can occur, as, those who choose PUPs coming from other Linux distros bring their consistent knowledge from those distros they have used. And any Windows/MAC tire-kickers, too, will see a common structure making it simpler to get onboard in PUPs boot processes.

That said, there is another benefit which could be useful for PUP's GRUB2. A developer has a "MATURE" GRUB2 implementation which automates bootable Linux/Windows discovery with NO USER intervention required. It works for both BIOS as well as UEFI PCs. Though he is very busy, he inidicates a willingness to have PUPPY Linux distros become one of those OSes which would be automatically discovered, too. From my understanding, it merely require a simple file inclusion in the path where grub.cfg is found. Further this file can be universal for PUPs; same as it is for other distro aka. Ubuntu, etc.

I am aware that everyone in WOOFCE is busy too, but, this offer could significantly reduce the learning curve for getting a common GRUB2 implementation that in understandable to all developers/users/reviewers.

I am hopeful that some of us are interested enough to take a moment to review his offer in this GIT thread. My feeling is this is something that is to be included in WOOFCE versus something that is released as an installable.

Please take a moment and look over this GIT exchange.

I am willing to work with anyone to create a wiki or a document which supports PUP's GRUB2 builds to boot DVDs or Frugals or directly from PUP ISOs with no user intervention required. And/or I am willing to generate a user guide to insure an "ease of understanding" as well as use of the boot-manager.

I see this as a one-time addition that will not need continual future developer time; assuming we can come up with something consistent for GRUB2. All future filesystems and storage devices will be addressed by the large GRUB2 community and PUP developers will be relieved.

wdlkmpx commented 4 years ago

I don't understand what they're talking about. But reading this https://github.com/supergrub/supergrub/commit/8b9e0bf3da0eab603d426eda9bd4c88cfa1c20f7 it looks like all the documentation is needed

sc0ttj commented 4 years ago

Does this help?

https://www.supergrubdisk.org/wiki/Loopback.cfg

wdlkmpx commented 4 years ago

This is what I think I understand, and I'm not sure what to think. But it's probably something I'd only use when booting grub2 in uefi mode

grub2 (a flavor or something) is able to recognize ISO's and boot them through loopback.cfg
This is a standard loopback menu entry

menuentry "Ubuntu" {
        set iso_path=/_ISO/Ubuntu64.iso
        export iso_path
        search --set=root --file $iso_path
        loopback loop $iso_path
        set root=(loop)
        configfile /boot/grub/loopback.cfg
        loopback --delete loop
}

loopback.cfg simply contains entries with specific kernel params
the distro init (kernel params) script must implement a way to find and mount the iso file (must find the partition first).. and continue booting.
So, I have to download a 2GB iso just to see a 3kb file and try to understand how things might work.

wdlkmpx commented 4 years ago

I have achieved success... with grub4dos

A new secret boot param (synonyms): findiso | isofrom | find_iso | iso-scan/filename

Level of difficulty: high Requires a deep understanding of the boot process and init script (complex pup init script).

grub4dos:

title Start pup.iso
  find --set-root --ignore-floppies /ISO/pup.iso
  map /ISO/pup.iso (0xff)
  map --hook
  root (0xff)
  kernel (0xff)/vmlinuz find_iso=/ISO/pup.iso
  initrd (0xff)/initrd.gz

possible grub2 menu entry:

menuentry "Start pup.iso" {
    set isofile="/ISO/pup.iso"
    loopback loop $isofile
    linux    (loop)/vmlinuz find_iso=$isofile
    initrd   (loop)/initrd.gz
}

The loopback.cfg file is just a special menu with find_iso entries.. to be created.. and what about loopback.lst?

Problem/BUG: unclean shutdown

I'll continue testing tomorrow, changes will be added one of these days.

gyrog commented 4 years ago

"isobooter" has been directly booting Puppy iso's for years, using grub4dos.

CollaboratorGCM commented 4 years ago

ISObooter REQUIRES knowledge by the user!!! I llike ISObooter, but I cant overlook the obvious: User knowledge is required. Too many steps each time an ISO is presented.

The idea we should/could pursue is one which looks for a boot process that does NOT require user knowledge beyond placement of the PUP ISO.

MY reason for bringing this to our GIT is to make it "TOO SIMPLE" to have our future ISOs to boot directly without expanding it to a device. Merely to place the ISO in a prescribe in a prescribed library and it boots. Unlike ISObooter, no user script knowledge is required and no developer support would be needed.

The SG2D process says NO USER KNOWLEDGE is required. Merely place it "here" and I will find it and make it available for booting. This is TOTALLY different from ISObooter.

It is my understanding from SuperGRUB2 developer, that he is willing to hand-hold if we are willing to document our piece. This make easy for PUP builders to position their work for easy evaluation and use. And, the knowledge of doing this makes simple PUP OSes to work same as Ubuntu, Fedora, CAINE, etc.

Looks like a win-win-win with no-downside, if a short time appears to work either with him or thru him. And it can be done all via GIT.

Once complete, NO USER knowledge is required after ISO placement. ISObooter does NOT work like that.

I understand ISObooter and have written a document in its use. There are too many user steps required. It is easy for us, but new users have too much to learn for merely booting an ISO directly. (Yes, I, personally, know how to boot a PUP using GRUB2, GRUB4DOS and one other unmentionable)

The idea here is that this appear to have a easy, simple, long term benefit derive in a use of the GRUB2 Bootmanager which worldwide developer support that we take advantage for reason I think everyone knows.

SuperGRUB2 would NOT require user knowledge of scripts to run...etc. Merely place it and boot.

Once done for even a single PUP distro we can document for all future PUP developers.

I (and I am sure all of us too) would welcome something simple. Further I am sure all developers would welcome something that allows their ISOs to boot directly; no matter UEFI or BIOS.

Both, WOOFCE developers are very busy and I am acutely aware. But, this seemingly is a short one-time interaction between an GIT author and our community.

If this make sense, of course.

wdlkmpx commented 4 years ago

After spending a whole day on this.. commit 8c77c77e816f276ba014acd175f805bcafa0b0b5 implements support for booting directly from ISO. This seriously screams money.

if [ -f /sbin/isoboot ] ; then
  . /sbin/isoboot # optional, can remove all ocurrences of "isoboot" from this script
fi

It works with grub4dos and grub2 (bios & uefi mode).

An "add-on" was added to the initrd: /sbin/isoboot Delete that file (and a few lines from init) and the whole thing is gone.

init contains a few new conditional hacks to fix stuff for $ISO_LOOP. Easy to delete anytime without causing breakage.

With loopback.cfg new ISOs will (probably) be compatible with Super Grub2 Disk [OOTB] (I'll provide a test iso soon).

This specialized method works for me without super grub2:

menuentry "Start pup.iso" {
    set isofile="/ISO/pup.iso"
    loopback loop $isofile
    set root=(loop)
    linux    (loop)/vmlinuz find_iso=$isofile
    initrd   (loop)/initrd.gz
}

The use of /boot/grub/loopback.cfg is an agnostic method to boot isos, but distro developers must understand that everything must be implemented on their end.. $iso_path is exported so that loopback.cfg can pass it to the init script (kernel params). Difficulty level: over 9000.

Testing loopback.cfg with grub2 from debian, this works for me:

menuentry "Start pup.iso" {
    set iso_path="/ISO/pup.iso"
    export iso_path
    loopback loop $iso_path
    set root=(loop)
    configfile (loop)/boot/grub/loopback.cfg
}

A proper shutdown is achieved only if:

sfs's are copied to ram (1.1gb+ ram or pfix=copy (Copy SFS's to RAM from loopback.cfg menu)
you don't create a pupsave

Persistence is possible (pupsave) and is only allowed on the same partition where the ISO is located. ISO on a DVD = no persistence.

A test ISO will be produced tomorrow probably after downgrading eudev to 3.2.5.. 3.2.8 it seems to break stuff for all older pxps.

CollaboratorGCM commented 4 years ago

Once complete, post the path. I will test, too.

On Sat, Nov 23, 2019 at 9:01 PM wdlkmpx notifications@github.com wrote:

After spending a whole day on this.. commit 8c77c77 https://github.com/puppylinux-woof-CE/woof-CE/commit/8c77c77e816f276ba014acd175f805bcafa0b0b5 implements support for booting directly from ISO. This seriously screams money.

if [ -f /sbin/isoboot ] ; then . /sbin/isoboot # optional, can remove all ocurrences of "isoboot" from this script fi

It works with grub4dos and grub2 (bios & uefi mode).

An "add-on" was added to the initrd: /sbin/isoboot Delete that file (and a few lines from init) and the whole thing is gone.

init contains a few new conditional hacks to fix stuff for $ISO_LOOP. Easy to delete anytime without causing breakage.

With loopback.cfg new ISOs will (probably) be compatible with Super Grub2 Disk [OOTB] (I'll provide a test iso soon).

This specialized method works for me without super grub2:

menuentry "Start pup.iso" { set isofile="/ISO/pup.iso" loopback loop $isofile set root=(loop) linux (loop)/vmlinuz find_iso=$isofile initrd (loop)/initrd.gz }

The use of /boot/grub/loopback.cfg is an agnostic method to boot isos, but distro developers must understand that everything must be implemented on their end.. $iso_path is exported so that loopback.cfg can pass it to the init script (kernel params). Difficulty level: over 9000.

Testing loopback.cfg with grub2 from debian, this works for me:

menuentry "Start pup.iso" { set iso_path="/ISO/pup.iso" export iso_path loopback loop $iso_path set root=(loop) configfile (loop)/boot/grub/loopback.cfg }

A proper shutdown is achieved only if:

sfs's are copied to ram (1.1gb+ ram or pfix=copy (Copy SFS's to RAM from loopback.cfg menu)

you don't create a pupsave

Persistence is possible (pupsave) and is only allowed on the same partition where the ISO is located. ISO on a DVD = no persistence.

A test ISO will be produced tomorrow probably after downgrading eudev to 3.2.5.. 3.2.8 it seems to break stuff for all older pxps.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/puppylinux-woof-CE/woof-CE/issues/1672?email_source=notifications&email_token=ADXYRARVJRCM6MBPH63NQ5LQVHN63A5CNFSM4JQIZF42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFAB5CQ#issuecomment-557850250, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADXYRAUSVWLA5QY3GBYM26TQVHN63ANCNFSM4JQIZF4Q .

--

Here to Help you.

wdlkmpx commented 4 years ago

Is super grub2 able to recognize and boot this iso?

https://github.com/puppylinux-woof-CE/woof-CE/wiki/Builds#dpxp-stretch

CollaboratorGCM commented 4 years ago

Wonderful!

### Here's what I did..

Downloaded "SuperGRUB2's Image Beta", here
Downloaded "Puppy Linux ISO (beta)", here
Used an Image writing tool to write the SG2D image to a USB stick
Using GParted, I expanded the 2nd partition of the USB
On the 2nd partition of the USB I added your Puppy Linux distro to its "/BOOTISOS" folder along with Ubuntu, CAINE, and Rescatux
Unplugged the USB and plugged it into another desktop PC
Booted the USB and hit the enter-key upon SG2D's desktop
Waited until it completed its discovery process to produce its list of boot candidates
Selected pstretch-7.0.0.iso from the list
It booted to the Puppy boot-manager and then booted to the Puppy desktop.

THANK YOU for showing a PUPPY ISO that can become a GRUB2 target thru your efforts.

"I think" this means that if the loopback.cfg you provided could become a universal in all PUPs for GRUB2 use to boot. And it could simplify all future PUP boots with a standard library structure as well.

I would think that this means that PUP developers and users benefit by not having to worry about future hardware thanks, DIRECTLY, to what you are showing us.

Great...just GREAT!

CollaboratorGCM commented 4 years ago

The SG2D author is requesting something I am NOT familiar with. He seemingly wants to participate. Might be worthwhile for PUP Linux's GRUB2 future. He has in excess of 10 years experience with GRUB2

Could/would a PUP developer do this for the GRUB2 effort accomplished? It is here.

wdlkmpx commented 4 years ago

He's asking you to fork the project and edit a file through the github web interface. https://github.com/supergrub/supergrub/blob/loopbackcfg-doc/README.loopback.cfg.md

I've never used super grub2 so I'm not sure what I could write.

You could just copy and paste what I wrote here in previous comments. The documentation is right here waiting. I disclosed all the needed info for developers to know what to do to successfully create a loopback.cfg.

I didn't need any documentation, I only saw 2 things:

a standard grub2 menu entry to call loopback.cfg (export iso_path)
the loopback.cfg file from a ubuntu iso (2gb) (boot=casper iso-scan/filename=${iso_path})

That's all I needed to know to understand that everything must be implemented by the distro developers. Knowledge of how to create a custom grub2 menu with distro specific params is also required.

This thing is not easy to understand and only developers that understand the boot process and init script can implement it and it might require some drastic changes. Distros are different, the same code does not apply to all distros.

I knew Ubuntu & Arch Linux had special params to make them boot from a iso file.. Debian doesn't seem to work that way, it requires either a CD or USB with a dd-ed iso.

wdlkmpx commented 4 years ago

This is /boot/grub/loopback.cfg from ubuntu-18.04.3-desktop-amd64.iso

menuentry "Try Ubuntu without installing" {
    set gfxpayload=keep
    linux   /casper/vmlinuz  file=/cdrom/preseed/ubuntu.seed boot=casper iso-scan/filename=${iso_path} quiet splash ---
    initrd  /casper/initrd
}
menuentry "Install Ubuntu" {
    linux   /casper/vmlinuz  file=/cdrom/preseed/ubuntu.seed boot=casper only-ubiquity iso-scan/filename=${iso_path} quiet splash ---
    initrd  /casper/initrd
}
menuentry "Check disc for defects" {
    linux   /casper/vmlinuz  boot=casper integrity-check iso-scan/filename=${iso_path} quiet splash ---
    initrd  /casper/initrd
}
menuentry "Test memory" {
    linux16 /install/mt86plus
}

I haven't tested that ISO and now am going to deleted it hehe.

CollaboratorGCM commented 4 years ago

In reviewing your loopback.cfg in /boot/grub your lines are identical to munu lines found in /boot/grub/grub.cfg; excepting that each linux line has find_iso=... added.

Curiously, I tried replacing all of your loopback.cfg lines with a single line source /boot/grub/grub.cfg just to see if boot would fail without the find_iso linux parm

It failed (of course)

Next, I replaced the matching lines in grub.cfg with those of your ORIGINAL loopback.cfg to create a replacement grub.cfg. Using the replacement and used my single line loopback. This new ISO booted via SG2D, as expected.

I am in the process of burning the ISO to a CDRW to see if this replacement series will boot. Update: CDRW creates a GRUB4DOS boot manager ... not GRUB2. So cannot test new ISO changes on bootable CD

CollaboratorGCM commented 4 years ago

Changes to your /boot/grub files. Works booting ISO from SG2D.

LOOPBACK.CFG source /boot/grub/grub.cfg GRUB.CFG

insmod part_acorn
insmod part_amiga
insmod part_apple
insmod part_bsd
insmod part_dfly
insmod part_dvh
insmod part_gpt
insmod part_msdos
insmod part_plan
insmod part_sun
insmod part_sunpc

loadfont /boot/grub/font.pf2
# set gfxmode=800x600
set gfxmode=1024x768
set gfxpayload=keep

insmod efi_gop
insmod efi_uga

insmod all_video
insmod video_bochs
insmod video_cirrus
insmod gfxterm
insmod png
insmod jpeg
terminal_output gfxterm

insmod ext2
insmod f2fs
insmod ntfs
insmod exfat

insmod loopback
insmod iso9660
insmod udf

background_image /boot/splash.png
set timeout=10

# https://help.ubuntu.com/community/Grub2/Displays
color_normal=cyan/black
#menu_color_highlight=black/light-gray
menu_color_highlight=yellow/red
menu_color_normal=light-gray/black

menuentry "PLinux Stretch 7.0.0" {
    linux /vmlinuz pfix=fsck pmedia=cd find_iso=${iso_path}
    initrd /initrd.gz
}

menuentry "PLinux Stretch 7.0.0 - Copy SFS files to RAM" {
    linux /vmlinuz pfix=copy,fsck pmedia=cd find_iso=${iso_path}
    initrd /initrd.gz
}

menuentry "PLinux Stretch 7.0.0 - Don't copy SFS files to RAM" {
    linux /vmlinuz pfix=nocopy,fsck pmedia=cd find_iso=${iso_path}
    initrd /initrd.gz
}

menuentry "PLinux Stretch 7.0.0 - RAM only" {
    linux /vmlinuz pfix=ram,fsck pmedia=cd find_iso=${iso_path}
    initrd /initrd.gz
}

menuentry "PLinux Stretch 7.0.0 - No X" {
    linux /vmlinuz pfix=nox,fsck pmedia=cd find_iso=${iso_path}
    initrd /initrd.gz
}

menuentry "PLinux Stretch 7.0.0 - No KMS (Kernel modesetting)" {
    linux /vmlinuz nomodeset pfix=fsck pmedia=cd find_iso=${iso_path}
    initrd /initrd.gz
}

menuentry "PLinux Stretch 7.0.0 - Ram Disk SHell" {
    linux /vmlinuz pfix=rdsh pmedia=cd find_iso=${iso_path}
    initrd /initrd.gz
}

menuentry "Shutdown" {
    halt
}

menuentry "Reboot" {
    reboot
}

CollaboratorGCM commented 4 years ago

Could the WOOFCE development community add GRUB2 Bootmanager config to the PUP Menu subsystem, please?

CollaboratorGCM commented 4 years ago

@JamesBond has a suggestion ISO available on ibiblio that I have tested with his single line loopback.cfg and the grub.cfg that boots the system normally from DVD as well as boots the system when it is discovered by SuperGrub2.

It is CLEVER too. That makes Fatdog and @wdlkmpx tests, both, working in either environment....same a Ubuntu, etc.

These WOOFCE developers have produced a 32bit ISO and a 64bit ISO that have been tested on my 64bit configurations. Both approaches are independently focused, slightly different, in implementation...yet effective.

CollaboratorGCM commented 4 years ago

Maybe the upcoming PUPs will make GRUB2 "OBVIOUS" in the PUP Menu system. Older PUP distros ONLY show GRUB4DOS.

Time to include GRUB2 utility(s) along with the UEFI changes upcoming. It is currently not apparent to users via PUP's menus; change should land in Menu>Settings or Menu>System or whatever PUP's future brings.

CollaboratorGCM commented 4 years ago

An updated report of booting PUP ISOs is here

3guesses commented 4 years ago

Sorry, a little late to this party and I've only skim-read this thread so apologies if I'm slightly wide of the mark with my comments.

I am very keen on being able to boot ISOs from USB flash drive - in native format (ie without extracting the ISO). I have been doing so for years very successfully using the most excellent WinSetUpFromUSB. This tool leverages grub4dos to enable the user to boot from a selection of ISOs stored on a single USB flash drive by mounting the ISO file as if a CD had been inserted and then chainloading it. Naturally, this requires the flash drive to be formatted to the MBR schema and only supports legacy (MBR/BIOS/CSM) booting, ie not UEFI. Grub2 does support UEFI (and GPT), but it does not support chainloading ISOs. This is a major functionality deficiency compared to its predecessor. Since most Linux ISOs support [persistent] live environments (and installers), it really would be incredibly useful functionality if Grub2 were to provide functionality that would allow the mounting and chainloading of ISOs in GPT partitions via UEFI in the same way that grub4dos does via MBR.

Or alternatively (if technically possible), UEFI and GPT support being added to legacy grub/grub4dos (which, given the option, would very much be my preferred solution and I'm sure that of many other users).

Again, apologies if I have missed the exact target of this thread.

CollaboratorGCM commented 4 years ago

Why do you think Chainloading is a feature missing? Hmmm.

This is NOT about taking anything away from GRUB4DOS, as it is very very good.

This is about moving forward with what the manufacturers of hardware are doing as well as the worldwide investment is doing moving forward. YOu must remember, GRUB is a LINUX product built and supported by the Linux community. GRUB2 carries that moving forward.

Lastly, the product mentioned here ONLY attempts to promote a form of booting that does NOT require anything from a user other than to download and boot the ISO file with NO effort or understanding by the user to get to a Puppy desktop. NOTHING!

Users just boot the ISO file.

And this frees the PUppy developers from having to support or entertain boot as GRUB2 is worldwide supported and WOOFCE allows a correct distro generation.

Puppy loses nothing! Everyone gains. In total of this, it makes it all too simple for any user, new/old Windows/Mac/Unix/Linux/etc, to go right to desktop. And they use a booter that is widely available all over the world in many/most mainstream distros already; GRUB2 (look at Distroworld's distro content charts for each distro listed ).

Hope this is helping you "why this started".

3guesses commented 4 years ago

Hi @CollaboratorGCM,

The point I was trying to make was that grub4dos CAN chainload ISOs whereas Grub2 CANNOT - Grub2 can chainload other things, and can boot specially configured ISOs, but it can't chainload them.

The ability to chainload an ISO makes booting one completely trivial and transparent for the user, especially if you use a tool such as the most excellent WinSetUpFromUSB that I mentioned previously. Try it and see - it will change your life. Probably.

So rather than spending time and effort trying to bodge booting ISOs using Grub2, I strongly suggest either adding the "chainload ISO" functionality to Grub2, or else add UEFI and GPT support to grub4dos. Alternatively, add the "chainload ISO" functionality to the rEFInd boot manager, which Roderick Smith seems to think would be a fairly simple task for someone with knowledge of EFI drivers - see rEFInd discussion thread

CollaboratorGCM commented 4 years ago

Again, "Chainloading?"

One of the major points being made here is that NO DEVELOPMENT EFFORT by Puppy developers are required for their distros to boot. NONE! And their work incorporating things has been accomplished for users to boot to desktop...no matter BIOS/UEFI.

GRUB2 is worldwide contribution on GIT. This is just 2 of the prime reason we are here. Many users understand it in the world. Manufacturers contribute to it for hardware needs.

There are NO WINDOWS CONCERNS OR NEEDS, as the ability to boot any Windows without disturbing a Windows booting system is shown here as well.

It appears you have NOT tried any of what is discussed here. Is that true? Did you find something broken? Did the PUPs not boot?

Please share any finding you have, please.

I am not sure what you are getting at???

If you are offering to develop new materials for Puppy evaluation, then you may want to open a GIT thread to accomplish that development and support it....same as is done for world's GRUB2.

CollaboratorGCM commented 4 years ago

@3guesses

Since it appears you have NOT tried any of this, I ask you evaluate and comment on this Puppy Linux forum thread built by another PUPPY linux member. It shows current 2020 PUPs that work! NO media files are generated after downloading PUPs ISO files to the common folder.

These PUP ISO files have been booted and tested on BOTH BIOS & UEFI PCs.

He shows a document which intends to hand-hold any user in a start to finish at a PUP desktop in that forum thread: Try it.

If you find errors, documentation miss-wordings. OR if you find bugs, please post them to that thread for other PUP users to see.

Your contribution could help everyone...if you are willing.

puppylinux-woof-CE / woof-CE

GRUB2 update for PUPPY Linux builds #1672

Changes to your /boot/grub files. Works booting ISO from SG2D.