New branches 'legacy' and 'rationalise'

[01micko]

For more information, I will refer readers to this blog post.

I have created 2 new branches legacy and rationalisation.

Important

As @mrfricks and myself are in development please keep changes to testing as bugfix only or minor updates unless there is a pressing need for change. After release of xenialpup and slacko-63x (bugfix) it is intended that rationalise branch will be merged with testing and deleted. Hopefully this will only be a matter of weeks. Of course all xenialpup changes before and at release should land in testing so that we can merge that with master.

'legacy'

legacy branch is purely in place to preserve old code. Read the README before making changes to that branch - changes are discouraged.

'rationalise'

rationalise branch is for sweeping new changes including but not limited to:

• deletion of un-maintained distros (arch, mageia etc) and older unsupported versions of distros (ubuntu - jaunty, karmic, lucid; debian - squeeze; slackware - 13.1)
• rationalisation of code in the woof build system
• changes in the way fdrv is dealt with. See #785
• corresponding changes to kernel-kit and rationalisation of code
• initrd_progs
• other stuff which may need enhancing. Suggestions welcome.

  Notes

I expect that @zigbert will keep updating his progs in testing. Therefore, before anyone makes a commit or PR to rationalise be sure that you have merged the changes in testing. This will make for minimal conflicts when rationalise is merged back into testing.

Of course any minor bugfixes and enhancements should land in testing first. So if you have a change to make, carefully consider if it is major or minor before committing.

As usual, have fun! :octocat:

[wdlkmpx]

The last 3 "|" are crucial, so I ended up making all pkgs end in |||, like this:

rubix-1.0.6-x86_64|rubix|1.0.6-x86_64||Fun|120||rubix-1.0.6-x86_64.pet||small rubiks cube game||||

findpkgs has a logic that requires consistency in the same base pkg, you won't notice it until the exe pkg is not found, that's what happened to you..

I'm not sure if this is worth pushing, but everything works as expected now..

[01micko]

Hmm.. the specs are written by dir2pet. _DEV _DOC _NLS are usually only for compiled apps anyway so it shouldn't be hard to automate. The Packages-puppy-slacko6414.2-official will change radically soon as I compile packages.

[01micko]

Is it enough just to add it to _DEV packages? After all the others are discarded, except for gtkdialog where doc>dev

[peabee]

Built a new system including: 1a35f76 xorgwizard: remove old cli code

Still have no keyboard or mouse after selecting Option 1: Continue :-(

[peabee]

Why does 3builddistro-Z always download a tarball from 01micko.com making it dependent on an internet connection - 1download also downloads....???

--2016-07-03 09:29:40-- http://01micko.com/initrd_tarballs/initrd_progs-20160630-static.tar.xz Resolving 01micko.com (01micko.com)... 203.170.81.37 Connecting to 01micko.com (01micko.com)|203.170.81.37|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 3042288 (2.9M) [application/x-xz] Saving to: '0sources/initrd_progs-20160630-static.tar.xz'

initrd_progs-201606 100%[===================>] 2.90M 845KB/s in 3.5s

2016-07-03 09:29:47 (845 KB/s) - '0sources/initrd_progs-20160630-static.tar.xz' saved [3042288/3042288]

• Extracting initrd_progs-20160630-static.tar.xz... Arch: i686

[wdlkmpx]

I'll test again peebee... 3builddistro shouldnt download that tarball because it was already downloaded by ./1dowload... i'll look into this..

[wdlkmpx]

the initrd-progs issue should be resolved now..

i downloaded the latest iso, i noticed it was commits behind so i added a ydrv with /usr directory from rootfs-skeleton.

boots to desktop, no xorgwizard-cli dialog (i use pupdeskflag to trigger that dialog at bootup), so there's nothing to worry about i guess, however i fixed a pupdialog bug i introduced a few days ago.. the shutdown dialog does not obey. also tested with DISTRO_XORG_AUTO='no' in initrd... displays cli dialogs including timezone, etc.. obviously commits behind. after the xorgwizard-cli dialog, it continues as usual, no issues.

so i guess you can update, unless you want to use that xorgwizard-cli dialog explicitely

i wonder what exactly are you doing to trigger the xorgwizard dialog at startup... because it should not popup by default...

[wdlkmpx]

The old xorgwizard is gone, i see that gnomeplayershell writes config based on the xorg driver, the dconf,gconf, glib schemas, whatever sometimes just doesn't work, it was a bad choice for gnome-mplayer.

The advantage of mplayer is this: it can bypass hardware limitarions and play HD videos in hardware that does not support them, assuming of course you pass some swithces (i have a script that does this).

Beyond that. mpv surpasses mplayer by a mile.. you don't need to specify a video output. It will automatically find the correct video output, it can interact with youtubedl to play vids from a thousand sites, it's just beyond amazing. And coupled with SMPlayer you just have the best experience ever, if you compile mpv with the latest ffmpeg (statically linked).

[dimkr]

These chanegs already create some confusion (http://murga-linux.com/puppy/viewtopic.php?p=911556&sid=54a7e3c1c91bd398953ca5a1832d4ea2#911556), I think it's time for a blog post that summarizes what has already been done.

[wdlkmpx]

Most stuff has already been done, except... support for newer savefile encryption, which will cause many rants as there is no "light" encryption, the fdrv stuff in kernel-kit. Although 3builddistro-Z could do the same, even offering an option to use a full firmware package (gyro's) as the fdrv. A running puppy should also offer that, in case it's running in a RW partition..

Other things to do, perhaps contact some active devolopers, like the debiandog one, offer people to merge their work to produce official variants of the same puppy, Of course this is just wishful thinking. i certainly dont want to run a rox puppy, but a pcmanfm one.

I also see alpine linux could be a good option, a new gentoo pup should also be cool, and basically anything that works fine with pup-volume-monitor, i couldn't get it to work properly in precise

[wdlkmpx]

Just as I thought, the only issue is xorg. I'm running lxpupSc with Xorg server 1.11.4 and drivers from Ubuntu Precise. A 2003 machine with extremely crappy graphics running fine lxpupSc with kernel 4.6.3. This xorg version was probably the last to feature XAA acceleration. Quite fast actually, the main sfs has no compression at all.

The 32bit puppy should be the "retro" version including an alternative legacy xorg (about 5mb) and greeting the user with xorgwizard in the first boot, offering no changes, xorg vesa and a legacy xorg version for the problematic old hardware. Everyone should upgrade to the latest puppy..

[gyrog]

It would make things a little simpler if any new encrypted save faciity was based on encrypting a directory. ALL save mechanisims (savefile, savefolder, savepartition,....) must end up as a directory for inclusion as an aufs branch. So such a facility could be used on all current save mechanisms, and any future mechanisms.

[wdlkmpx]

I've never used a encrypted savefile, before i get around to it... gyro, do you already have an idea on how to do this? , i mean code lines..

I remember mrfricks having problems with a recent version of util-linux, so besides losetup and savefile encryption... what other issues related to util-linux need to be fixed in woof?

I just finished implementing the xorgwizard cli stuff i was talking about. To trigger the xorgwizard cli at first boot (no other cli dialogs), in DISTRO_SPECS before building: DISTRO_XORG_AUTO='no'

or edit initrd.gz/DISTRO_SPECS

There are 4 options: do nothing, set screen resolution (probably Marv will like this), use vesa driver, use alternative xorg server version

pet for lxpupsc... it works flawlessly in my oldest machine.. additional 3.3MB to the main sfs or another sfs, but this does really makes it support a broader range of hardware xorg-1.11.4-i686-ubuntu-precise-4-lxpupsc.pet: https://www.datafilehost.com/d/6ef6d6f3

[peabee]

Tried the xorg_alt... above pet + latest xorgwizard from git-hub - works fine on my desktop with nvidia graphics

Notice it is actually version 1.11.3 but xorgwizard says it's 1.11.4

Best delivery mechanism would seem to be a ydrv sfs in the iso - that way it can be tested in a pristine frugal install without savefile creation and reboot and is easily omitted from an installation if not required - I've uploaded a ydrv for LxPupSc-16.07.2 to: http://smokey01.com/peebee/downloads/woofce-rationalise/

[wdlkmpx]

Uhmm yes it's 1.11.3.. but i think it really is 1.11.4 (http://packages.ubuntu.com/precise/xserver-xorg-core) maybe a Xorg that reports 1.11.3 + patches = 1.11.4. There's no way to know the actual version if the /opt/xorg_alt Xorg binary is missing dependencies, then the script falls back to read this file: /opt/xorg_alt/usr/lib/xorg/version (manually created to address this), just edit that file and fix the version number..

[gyrog]

@wdlkmpx, I do not have a code suggestion for encrypting a directory. (I've never used an encrypted savefile either.) I just thought that if the encryption facilities provided by puppy are changing, it would be nice to try and find something that would work more universally, rather than just savefiles. I've still got lots of other stuff to work-on in the new init, before I look at support for encrypted save.

[peabee]

Ubuntu has that nasty habit of refusing to increase version numbers.....openssl was a case in point.... All the Puppy tools report 1.11.3 - Sys-Info Video-Info

[wdlkmpx]

I think encryption works in filesystems only, but i'm not sure. The dmcrypt module must also be builtin I guess. So basically i'm not sure if current kernels comply... mine does not (i think).

I think i'll edit my version of kernel-kit to make it more suitable for inclusion. It does some things differently.. the kernels it builds can be later put together in a single SFS, of course it requires changes at bootloader level and renaming vmlinuz of course.

A "standard" kernel config for a specific version should be produced i think, a little bigger. More firmware by default, people are having issues... I have other ideas but ...

[gyrog]

If dmcrypt is our only option, and this only works on filesystems, then we are limited to encrypting only savefiles, just like current puppy.

[gyrog]

I just finished implementing savefile resizing in the new init script, and fould that the file pupsaveresize.txt is in the "wrong" location. It's in the root of the partition when it should be in the PSUBDIR, so it will be seen by only the puppy that it relates to. To support a transition, the new init script looks in both places. It contains a line of code that can be deleted if the file gets moved.

[wdlkmpx]

Thinking about the huge firmware package i have and the script i wrote to download and install/update it...

I would suggest to compile a somewhat full kernel (distros produce a 50mb kernel (xz compressed)) and trim it down to produce the official version, both versions can be uploaded to the same dir... a script can check if there is a more complete version of the same kernel, and offer to download and install it (replace the sfs or actually install it for full installs).

To replace an exisiting sfs, just overwrite the current one and add garbage to BOOTCONFIG (LASTUNIONRECORD) to force updating the layered filesystem, then reboot.

Many users are experiencing problems, most of them don't have a clue of what's happening, i bet would rather install a huge and heavy distro when the solution in puppy is a megabytes away. Some people use an outdated version when a newer one might have the fix or a better way to report issues and these people create a lot of topics. So i think this is a lost 'cause anyway.

I would suggest to offer releases as torrents as well... in the official page.

[gyrog]

An easy way to replace a pupdrv sfs file is to specify it as a boot parameter, e.g. "zdrv=:wonderful.sfs" will use $PSUBDIR/wonderful.sfs as the zdrv. If it works, then rename it to standard name if you want to. Using different names automatically triggers updating. But if you are replacing and fdrv or zdrv, no updating is required. The updating is not realted to the stack itself, since thhis is build new at every boot. The updating is to fix up other things that may be changed byt the presence of different sfs files, e.g. menus, fonts, etc....

[wdlkmpx]

Good points gyro... mmm i see anything that has to do with the bootloader as "advanced usage"

Wow, i can't believe what i see, some dirs in Packages-templates are huge, xsane.. has huge images, mplayer_gui has 1.5MB of fonts. Well to be honest, i think the git repo has to be restarted, there is/was too much junk that should be in pet repos instead, and it will never go away, it will live forever in the git history.

I think i'm burned out, but just like a drug i can't get rid of, i'll continue. But micko and mrfricks should start building alphas using the rationalise branch so i can test as well.

I also want to give mrfricks some private advice,

I think th

[peabee]

Just tried to do a build with: BUILD_FROM_WOOF='rationalise;2e7a990;2016-07-20 09:37:09 +0000'

got error from 3builddistro-Z:

ENTER only for default, or type it now: ...ok, set to en_US.UTF-8

• Extracting initrd_progs-20160630-static.tar.xz... Arch: i686 00_i686/bin/losetup-222 not found mv: cannot stat 'initrd.[gx]z': No such file or directory No initrd... exiting

[wdlkmpx]

it should be fixed now..

[gyrog]

If encryption of the save layer is to be changed, can ecryptfs be used instead of dmcrypt? Ecryptfs can work at the directory level, so it could be used for both savefile and savefolder. Either way, no work can be done on new encryption code in the init script until an appropriate kernel is available.

[wdlkmpx]

maybe, according to this page there are 2 encryption methods: http://www.cyberciti.biz/hardware/howto-linux-hard-disk-encryption-with-luks-cryptsetup-command/

well, regarding the kernel config, it must be easy to produce an appropriate kernel.

jamesbond has already implemented the dm-crypt stuff in fatdog, maybe he can give us some advice..

i'm looking inside a fatdog iso.. i see the cryptsetup binary is quite small, the one i compiled (1.7.2) is 1MB haha, i think i should use --with-crypto_backend=kernel

the code to be copied is in initrd/init and initrd/fd64.sfs/.../savefile.sh i guess

[wdlkmpx]

funny i keep hitting the same error everytime if i use --with-crypto_backend=kernel

as i see in lxpupsc... the current code is stable, even the code in testing is less stable, so i think it's time to merge rationalise into testing, if we want to dig up something we must switch to the legacy branch.

the current init is working fine, i'll make losetup-222 the default losetup. then it's ready to go into testing again.

this is to speed up the testing process for what is already working fine. so stable development can continue in testing

i basically want to address the coreutils/util-linux/busybox mess... people have complained about this since the beginning of time, and i'm part of the angry mob.

rationalise is still the place where really unstable and experimental stuff is uploaded... which is the rest of the roadmap.

[01micko]

I think that it is important that @mrfricks (ping!) has some input as to merging. Maybe testing should get merged to master first?

I'm doing my building out of rationalise lately anyway so it doesn't bother me. Slacko(and 64) both build fine out of rationalise albeit with a few bugs that are nothing to do with the woof process; packaging issues which will be sorted soon.

[wdlkmpx]

Well, i'll merge testing to master, and then after a few hours later i'll add a few commits to rationalise and merge it back to testing..

Well, i think it's better this way... i'd rather hear about bugs than finding them myself... less work and more time to focus on other things. So I'll ask mrfricks to build a couple private test isos until it's ready and i'll be watching the xenialpup thread. Some people may miss the old xorgwizard, but xorg works fine without it, when it doesn't it's because a downgrade is needed or it's a kernel issue (brand new hardware)... i already wrote code to fix this and have xorg 1.11.4 from precise ready to replace xorg 1.18... all problems solved.`

Now building a ubuntu based puppy iso is way faster, it's still heavy in findpkgs, but it's 3 times faster, the checking of existing packages in 0setup takes a few seconds.. it used to take an eternity. For slackware everythings is as fast as a bullet but the processing of some heavy packages (glibc_locales is way faster now).

[wdlkmpx]

Ok it's done. So everybody start building pups and report any issues you find.

[wdlkmpx]

Thanks for the feedback ninaholic.

Well, i'm not sure, when i installed a newer udev in precise, now it takes longer to reach the desktop. Actually like 5 seconds indeed, but loads the joypad driver automatically so i can play Mortal Kombat 3, something it didn't do before.

In this case it must be something else, but ensures a proper behavior even in the slowest hardware, and i have something close to a PIII and therefore to a primitive raspberry pi or something. That's where i tested the MSCW with pci soundcards (the slowest hardware) and the latest kernel working just fine

[01micko]

@ninaholic thanks for testing a 14.2 build!

It is in very early stages as I updated shared-mime-info and that is likely why evince isn't working. There is also a problem with gzipped files not opening with clicking on them. I'll sort these issues out soon. There are bound to be more because we were using an ancient patched up version of shared-mime-info.

I haven't booted it on a laptop yet so don't know about the flsynclient issue. Will check it out soon.

[peabee]

I've rebuilt LxPupSc-16.07.3 using: BUILD_FROM_WOOF='testing;05ce82a;2016-07-26 21:27:28 +0000'

plus procps-ng recreated using the package template

All seems sane - and similar congrats to @wdlkmpx as given by @ninaholic - the speed up in building is impressive and certainly very welcome

[wdlkmpx]

I play it with snes9x (every so often).. ultimate mk3. it's surprisingly good for SNES, just like Donkey Kong 2/3 it's closer in design to Play Station 1. I like how complex it is to perform combos and fatalities. There's also a version for arcade.. i played it in MAME, very 3D.. too heavy and ugly screaming voices... and for PS1 (PCSXR), like the arcade version, but it's playable in slow machines.

The new project for rationalise is basically integrate sfs_load properly (woofce specific) and remove the code in the initrd init that deals with extra sfs's.. pave the way for the new init until it's ready, etc.

[gyrog]

The first thing to do to properly integrate sfs_load, is to fix the facility in "Startup Manager" (System->Boot Manager) that does SFS-packages, so that it calls sfs_load when the "Choose which extra SFS files to load at bootup" button is clicked. Of course the text also needs to be changed. Then sfs_load becomes the sole mechanism for loading extra sfs files. It's this other mechanism that uses BOOTCONFIG and forces sfs_load to fudge a separation.

[wdlkmpx]

That's what i was thinking, i was checking bootmanager... that dialog is triggered in the second boot, i think i'll remove it.

Smithy said that the first boot dialogs are annoying as hell.. should i remove them? and place a quicksetup icon the in the PuppyPin instead. Although i remember there was a mechanism to reset the puppypin, it happened to me once, i can't remember how i triggered... didn't like it.

I've been checking some topics... this guy "oui" is truly lost, he said DebianDog is one of the most optimized puppies, but it's not puppy and has nothing to do with puppy.

I wonder if it's ok to add a ca-certificates "bundle" to rootfs-packages. too much text. i have these files i took from debian and arch linux, these all contain different files and folders and are "noarch" despite the i686 in one of them.

ca-certificates-20160104-all.tar.gz - the ones we know ca-certificates-cacert-20140824-2-any.tar.gz - strange stuff ca-certificates-mozilla-3.22.1-1-i686.tar.gz - mozilla certs.. 1MB uncompressed ca-certificates-utils-20150402-1-any.tar.gz - strange update-ca-trust script patched for puppy

[gyrog]

I don't think we should remove the SFS-packages facility from Boot Manager, just change it to use sfs_load. I'll see if I can produce a patch against rationalise.

Smithy might find the first boot dialogs annoying as hell. I find the quicksetup one quite useful, I usually change the language, timezone and hostname on any first boot. So, I vote no to removing it.

[mrfricks]

bit late, but I'm happy building xenial with the new merge.... nice work!

[gyrog]

The new init is now in rationalise, thanks wdlkmpx.

It still needs support for internationalised messages. Why should it not use gettext like many other puppy scripts?

It still needs crypto. Do I port the stuff from the current init or implement something different? If different, what?

[wdlkmpx]

I'll try to use gettext once i compile it for the initrd..

First we just have to make sure that the current init works like the old one, then we can try to do something else..

[gyrog]

As a start to internationalisation, I've made a text file that contains all the current messages that need to be transalated as variables. But I've not started to modify the init code itself, and I won't until we've decided gettext vs variables.

[peabee]

LxPupSc-16.08.99

BUILD_FROM_WOOF='rationalise;203706e;2016-08-02 12:31:20 +1000'

Seems sane on quick initial test.....

[wdlkmpx]

this is still alpha.. the init script is missing bits of code, there is a problem checking the DISTRO_SPECS in the savefile, apparently the path is wrong.. it always asks whether to upgrade. i have to go out, if gyro doesn't fix this by tomorrow, i'll probably do it.

this is the message i get (not using xenial, just an example): grep: /mnt/dev_save/PUPPYBOOT/xenial/xenialsave.4fs/etc/DISTRO_SPECS: Not a directory

[gyrog]

Thanks peabee for LxPupSc-16.08.99. In my testing using a savefolder, everything worked fine OOTB, including adding 2 extra sfs's and after a few reboots removing them, so the sfs_load connection seems to be working.

@wdlkmpx, I did not see that problem at all, I'll check if savefile makes a difference.

Unfortunately, my latest push missed the build, but it's just some keymap code I had missed.

[gyrog]

@wdlkmpx, I see the problem, it's looking in the filename of the savefile/savefolder instead of the directory it's going to add to the aufs stack. For a savefolder the one is just a symbolic link to the other, so either works. But for a savefile these are completely different. I'll push a fix as soon as I've tested it.

[gyrog]

version check fixed in 87d267e

[wdlkmpx]

ninaholic there is a problem with the synaptics driver.. it's not being loaded. Xorg tried to load it many times.

/usr/lib/xorg/modules/input/synaptics_drv.so is missing this dep: libevdev.so.2

[wdlkmpx]

There is a script i consider essential when building pups, actually i wrote when i first tried to build a puppy back in 2014, it's called checkdeps

just run "checkdeps /" and you'll see what other libs/executables are missing dependencies..

[wdlkmpx]

/sbin/test-eide and /sbin/test-scsi seem to do the same as probedisk.. i think i'll delete them...

puppy is based on large distros, however the way it's designed makes it a bit difficult for a beginner to use and learn about some basic unix commands. so puppy is not fit even as learning tool. one day someone was complaining puppy does not have "unshare".. www.murga-linux.com/puppy/viewtopic.php?t=105435 so i'll add some more apps from coreutils, util-linux.

i think i'll also a busybox petbuild, although it uses only libc, if you compile in precise (or even wary), then it will be universal.

by the way, i think you should close this issue and open a new 1. the page is getting heavy in light firefox