New branches 'legacy' and 'rationalise'

[01micko]

For more information, I will refer readers to this blog post.

I have created 2 new branches legacy and rationalisation.

Important

As @mrfricks and myself are in development please keep changes to testing as bugfix only or minor updates unless there is a pressing need for change. After release of xenialpup and slacko-63x (bugfix) it is intended that rationalise branch will be merged with testing and deleted. Hopefully this will only be a matter of weeks. Of course all xenialpup changes before and at release should land in testing so that we can merge that with master.

'legacy'

legacy branch is purely in place to preserve old code. Read the README before making changes to that branch - changes are discouraged.

'rationalise'

rationalise branch is for sweeping new changes including but not limited to:

• deletion of un-maintained distros (arch, mageia etc) and older unsupported versions of distros (ubuntu - jaunty, karmic, lucid; debian - squeeze; slackware - 13.1)
• rationalisation of code in the woof build system
• changes in the way fdrv is dealt with. See #785
• corresponding changes to kernel-kit and rationalisation of code
• initrd_progs
• other stuff which may need enhancing. Suggestions welcome.

  Notes

I expect that @zigbert will keep updating his progs in testing. Therefore, before anyone makes a commit or PR to rationalise be sure that you have merged the changes in testing. This will make for minimal conflicts when rationalise is merged back into testing.

Of course any minor bugfixes and enhancements should land in testing first. So if you have a change to make, carefully consider if it is major or minor before committing.

As usual, have fun! :octocat:

[wdlkmpx]

today musher0 reported that basically all woofce distros are broken (PKGS_DISTRO_SPECS), so those files need updating asap.

fortunately today i also deleted some code i added to xwin, which at some point began to become a major bug visible in some older hardware.

also rerwin is making pull requests these days, which is very welcome. some past contributors such as sfr, karlgodt might want to test this stuff as well and provide feedback and patches..

[peabee]

this delta

converts 16.08.1 ('testing' build) to 16.08.99

a 'rationalise' branch build with the new init system

db4291cce7e307ea4cd1e2c804c1e429 LxPupSc-16.08.99.iso

BUILD_FROM_WOOF='rationalise;18661ca;2016-08-07 20:25:14 +0000'

[peabee]

I notice that my 'rationalise' on 8-aug and today 10-aug do not have fully populated /root/.packages/builtin_files

Only the rootfs-packages packages are there.

My 'rationalise' build on 2-aug was fully populated.

[wdlkmpx]

I haven't edited the build scripts in a while, i wonder what happened..

[peabee]

I have made a change which could be the cause - I've moved the packages-** directory up a level and symlinked it back into woof-out-** - perhaps the builtin_files logic does not work with a symlink? In all other aspects of the 3builddistro-Z process it seems to work OK.

[wdlkmpx]

so today i wrote a lot of things in the woofce needs you thread. a good read by the way.

it seems there are not enough contributors to keep the dbs and config files updated. so i think it's ok to let debian languish. there was greengeek talking about ubuntu's bloat and encouraging devuan development.. which is also bloat. everything based on debian is bloat. the only solution is to recompile most apps.

hmm it's clear that nobody ---knows how to--- contribute to the project (git), specially taking over relegated distros/version, although tuxx has showed interest in wary... even tiny core linux is using a modern kernel and software.

so i guess devuan is the only other distro to spend some time making it work, there is musher0 remastering a devuan iso using xenial pets and config files provided by mrfricks (not suitable), which confirms my theory... all debian-based distros are the same thing.

[gyrog]

Why do we use a real losetup in initrd.gz, but revert to the busybox one in the running system? Why not have the default losetup in the running system identical to the one used by init?

[wdlkmpx]

"busybox losetup" and "losetup" are a bit different... basically in "losetup -a"

however the build process renames losetup to losetup-FULL and places a symlink to busybox instead. this is one of the annoyances that i intend to fix but this might break or cause a couple scripts to misbehave if not done properly.

all the scripts that use losetup must be checked... inside and outside woofce.

[gyrog]

Yes, I know how it is. I just find it difficult to understand why we symlink the "better" one out of the way. But, unfortunately if a script extracts a filename from the output of busybox losetup, it's likely to fail on the output from the real losetup. I hope you manage to sort it some time.

[gyrog]

Crypto stuff in 'rationalise': I suggest that we immediately remove all crypto stuff from 'rationalise', init, shutdownconfig, rc.shutdown. LxPupSc already contains an losetup that has no support for crypto. We either go for dm-crypt for only savefile, or encfs for everyone.

[peabee]

FYI: losetup-FULL in LxPupSc comes from Slackware-Current package util-linux-2.28-i586-1.txz

[gyrog]

The so called cryptoloop stuff is not going, it's gone. If 'rationalise' is about positioning us for the future, then cryptoloop should not be part of it. dm-crypt only works on block devices, e.g. partitions or loop devices (i.e. a savefile). The fuse based ones look like they could be readily implemented, but would add 1.5 to 2 MiB to initrd.gz, and there could be a question of ongoing support. eCryptfs seems to be the obvious one since it is in the kernel, but it looks like a real pain to implement.

[01micko]

Ok, so kernels should have CONFIG_ECRYPT_FS=y (my current configs are set to 'm') Do we need CONFIG_ECRYPT_FS_MESSAGING?

No luck as yet building gocryptfs.

[gyrog]

I'm testing eCryptfs in xenialpup 7.0.3, and it works in a running system. It's DOTconfig-xenial-4.1.30-120816 has CONFIG_ECRYPT_FS=m, and CONFIG_ECRYPT_FS_MESSAGING is not set. I have found no reason to change this. The init script already contains code to insmod a needed module, (search for PIMOD). My big problem with eCryptfs is that it assumes that it can store stuff outside the encrypted directory which it expects to be available while mounting the encrypted directory. A slight problem when the encrypted directory is a puppy save layer, that gets mounted in init. But I intend to try to get it to work without that writing and /or try storing it back in the initrd.gz at the place where it is expected to be.

Just realised the fuse based ones don't have to be in initrd.gz. With the new init, by the time we want to decrypt the save layer we already have an aufs stack containing all the system sfs's mounted at /pup_new/. So should be able to execute any fuse ones from within that.

@01micko, thanks for trying to compile gocryptfs. While gocryptfs looks nice for a fuse based system, if we can't compile it for 32 bit puppies, then it's a no go. Not ideal to introduce a new thing that only works on 64bit puppies. So for a fuse based solution encfs is currently the top of the list, (It compiles, no problems.)

So still a lot of testing to do.

[01micko]

@gyrog

Just realised the fuse based ones don't have to be in initrd.gz. With the new init, by the time we want to decrypt the save layer we already have an aufs stack containing all the system sfs's mounted at /pup_new/. So should be able to execute any fuse ones from within that.

A drawback there though is that the savelayer is not at (or near as it needs to be) the top of the AUFS stack. It at least needs to be above the main sfs to cover updates etc.

[gyrog]

At the time we would be trying to decrypt the save layer, it is not yet in the aufs stack, only the sfs's are. (The new init builds the aufs stack gradually, one branch at a time, not all at once, and the save layer is the last to be added.)

[01micko]

Ok. Thanks.

So it works a bit differently to BK's model? - How puppy Works

All I was concerned about is that if you say for instance upgrade firefox, to my way of thinking (which could well be flawed) that the save layer had to be above the main sfs in the stack. I haven't had time to read your code but does the save layer get inserted 'above' the main sfs? I'm more than likely missing something completely! Anyway, I plan to do a build soon but ibilbio is down and I have to wait until that is up (NB: down for uploading pets/sfs only. We seem to be locked out).

[jamesbond3142]

One big consideration as you're re-modelling the entire process, is to have a model where you can deconstruct the savelayer cleanly at shutdown. It is one thing to have a corrupted savefile/dir/partition which you can fix with fsck at boot, it's another thing to have a corrupted encrypted savefile/dir/partition that no tools can fix (or worse, silent corruption that goes undetected).

fuse-based filesystems are especially notorious for being very difficult to unmount cleanly once they're stuck in aufs root stack.

On Wed, 24 Aug 2016 23:16:14 -0700 gyrog notifications@github.com wrote:

I'm testing eCryptfs in xenialpup 7.0.3, and it works in a running system. It's DOTconfig-xenial-4.1.30-120816 has CONFIG_ECRYPT_FS=m, and CONFIG_ECRYPT_FS_MESSAGING is not set. I have found no reason to change this. The init script already contains code to insmod a needed module, (search for PIMOD). My big problem with eCryptfs is that it assumes that it can store stuff outside the encrypted directory which it expects to be available while mounting the encrypted directory. A slight problem when the encrypted directory is a puppy save layer, that gets mounted in init. But I intend to try to get it to work without that writing and /or try storing it back in the initrd.gz at the place where it is expected to be.

Just realised the fuse based ones don't have to be in initrd.gz. With the new init, by the time we want to decrypt the save layer we already have an aufs stack containing all the system sfs's mounted at /pup_new/. So should be able to execute any fuse ones from within that.

While gocryptfs looks nice for a fuse based system, if we can't compile it for 32 bit puppies, then it's a no go. Not ideal to introduce a new thing that only works on 64bit puppies. So for a fuse based solution encfs is now the top of the list.

You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/puppylinux-woof-CE/woof-CE/issues/789#issuecomment-242290764

James B jamesbond3142@gmail.com

[01micko]

KISS

My 2c is to allow savefiles optionally encrypted (dmcrypt - block device) and savedirs as they are. Much pain follows otherwise. Everything is a trade off, and our time is valuable.

[wdlkmpx]

I want to create an official semi retro pup. As i've been using really slow hardware (but not extremely slow as some puppy users are to used to use), i know what apps are truly heavy, what apps are more stable.

Basically the most lightweight combination is something similar to slacko, with deadbeef as the audio player and QtWeb as the web browser. To make qtweb work fine, you must properly install ca-certs and create a konqueror symlinkx to rox? i think. It allows total control over the content and features, so it can be almost as fast as dillo.

I think i'll go with precise, the base is already heavily updated as i've never had problems recompiling many apps for my pleasure. and i use this daily, the major problem is that i don't use rox and i can't hide the fact that i despise its design, but i might try again..

although precise is going to be archived soon. so what do you think..

[gyrog]

Re new init: At the end of init, the aufs stack is pretty much identical to the what it is now. The thing that's changed significantly is the way it goes about building the stack. Unfortunately this means that clean shutdowns are about the same as now. Looking at cleanly deconstructing the stack is still on the bucket list.

Re encryption: I'm currently playing with eCryptfs. I have some code that looks promising in a running xenialpup 7.0.3. I now need to workout any easy way to make all the needed stuff available to init. The interesting thing is that my current approach to using it means that the "mount" always works, it's just that if you get the password wrong the files are not decrypted. From my current perspective, dm-crypt simply doesn't give us any flexibility for the future. One thing I do want to do, is to make shutdownconfig more intellegent, i.e. only offer encryption options that the current system supports, so that right now it won't offer cryptoloop encryption on systems whose losetup does not support it.

Re retro pup: Sorry, but I'm simply not interested.

[wdlkmpx]

well, that was what i i was thinkg.. if losetup > 2.22 then don't offer old encryption if no DM_CRYPT module or no crytpsetup... don't offer cryptsetup if no ECRYPTFS or no ecryptfs tools.. then don't offer ecryptfs..

with a couple more edits to sfs_load and shutdownconfig, i say rationalise be merged back to testing again.. unless this causes problems with the rox desktop...

[gyrog]

jamesbond, you've just strengthend the argument against using dm-crypt. With a directory based encryption like eCryptfs, fsck can still do it's thing. While the contents and filenames are encrypted, they are still valid files from the filesystems point of view.

[gyrog]

@wdlkmpx, I was actually thinking of checking the output from something like "losetup-FULL -e 1 .....".

[gyrog]

Why do we have bcrypt and not ccrypt?

[jamesbond3142]

On Thu, 25 Aug 2016 12:09:52 -0700 gyrog notifications@github.com wrote:

jamesbond, you've just strengthend the argument against using dm-crypt. With a directory based encryption like eCryptfs, fsck can still do it's thing. While the contents and filenames are encrypted, they are still valid files from the filesystems point of view.

Here's my thinking:

When using dm-crypt, corruption can happen on two places: a) not cleanly unmount the decrypted partition, and/or b) not cleanly unmounting dm-crypt partition itself.

When using fuse, corruption can happen in two places: a) not cleanly unmounting the fuse partition (which provides decrypted view of the data) b) not cleanly unmounting the partition that fuse uses.

For dm-crypt, corruption in (a) can be fixed by fsck. Corruption in (b) is not obvious, it depends on where it happens, and it cannot be fixed directly. If it happens on data area, you're SOL. If it happens on metadata area, fsck on on (a) will notice it and attempt to fix it (which may or may not be successful).

For fuse, corruption in (a) cannot be fixed by fsck because it either happens in data area of the file (in which you're SOL) or in the proprietary fuse metadata area (which again means SOL). Corruption in (b) can be fixed by fsck, but fixes that involve arbitrarily changing file contents will result in corruption in (a).

The problem is made worse because it is easier to cleanly umount dm-crypt than it is to cleanly unmount any fuse filesystem.

As a comparison: In Fatdog we have two fuse filesystems which can be mounted by init to provide savedir/savefile/savepartition. The first one is ntfs-3g; puppy also has this. This can be unmounted cleanly, as long as we are careful.

The second one, I have a fuse filesystem called "posixovl", which provides POSIX semantic on filesystem that doesn't support it (e.g. on FAT). posixovl enables full POSIX semantics (symlnks, owner, file mode etc) on otherwise dumb FAT filesystem. If you're familiar with linux from the olden days, posixovl provides similar features as the legacy UMSDOS filesystem. posixovl however cannot be unmounted cleanly (at least I haven't found a way to cleanly umount it).

Now I haven't personally try to use savefile based on encfs, so I can't tell which category it would be.

ecryptfs is quite interesting to me (even encfs author says so) so I'm following with interest what you're doing with it, hopefully you can implement it successfully in your new init.

cheers!

[gyrog]

Yes, ecryptfs is interesting. I get the impression that there's no real unencrypted files, it's just a "view" that you get when it's "mounted". I have a method that looks like it could work with a puppy save layer, but it's still all about doing it directly in a running system. Still some large hurdles, i.e. getting it to work init, and the big one, will it play nicely with aufs. But I'm currently looking at a simple file encryption facility, hence my question about ccrypt. Once you add files to an ecryptfs "mounted" directory they can only be decrypted with the original key, to change the key you have to "mount" it with the old key, move the files somewhere else, then "remount" the directory with the new key and move the files back. So the thing to do is to not encrypt the files with the password, but generate a data-key and "mount" the directory with that and store this data-key in a small file encrypted with the password. Then to change the password you decrypt the file containing the data-key with the old password, and encrypt it again with the new password. While I'm waffling; it has the interesting capability to be mounted with any key, it's just that the files inside won't make sense unless they were written when the directory was "mounted" with that key.

[wdlkmpx]

Uhmm ldd .. what are the ecryptfs tools deps? I was creating an ecryptfs-tools static .petbuild, then i found a red hat pkg.. simple to compile. Later i discovered that Mozilla NSS is a hard dependency, there was no way to disable it even though the configure switch was there. Then i thought, i'll do this later.. it's somewhat difficult to compile NSS in a normal setup, it should be way more difficult to cross compile.. everything outside GNU autoconf is pure darkness..

i think it's time to merge rationalise into testing.. my philosophy is shoot first and ask questions later. but if someone builds from it, should label their releases 'alpha', add that name to the file and thread description.. developers should make clear that alpha releases are meant for testing only and will be deleted when a new version appears. there is no better way to do things. development requires a team for kernel config and stuff, a team to create extra wizards and udev related stuff, a team with gui capabilities and a team of dedicated testers.

i've come to the conclusion there should a script to gather all important log/config files into tarball for users to provide developers with interesting info about how their system behaves.. i mean all users, not one or two. since puppy is supposed to just work, devs need all the info needed to enhance wizards, etc.

i reckon the alsa stuff can be incredibly complex where you have to even add kernel parameters and create a special alsa config.. not worthy. overall linux requires knowledge and experience the average unsuspecting user that only wants their stuff to work will never have..

[gyrog]

ecryptfs tools: For testing in xenialpup 7.0.3 I simply installed ecryptfs-utils_111 and libecryptfs1_111 packages via PPM. My plan is to work out what is the minimal I need, and put that in a ydrv so that it is also available to init before loading the save layer. Hm..., to start testing in init, i'll probably just put it all in a ydrv.

If you want to merge rationalise into testing, then go for it. I would say that it's at alpha stage.

[wdlkmpx]

Yes i think it's ready.. but i'll wait a little more while i produce a retro kernel config for tux, meanwhile you might want to test a little more..

[gyrog]

ecryptfs: Seems that in the init environment, executable files that are in a mounted sfs cannot be executed. So to decrypt a save layer in int, all executable code has to be in the kernel or module or in the initrd.gz, correct? For ecryptfs, that means mount.ecryptfs, and either ecryptfs-add-passphrase or bcrypt or ccrypt, would need to be compiled and added to /sbin in initrd.gz. This is becoming a real pain.

Another argument for another significant challange on my bucket list: Too remove all save-layer processing from init and do it first thing in rc.sysinit, init would only and always boot into pupmode=5. The stumbling block could be removing the tmpfs rw layer when the aufs stack is the system /.

More testing..... What else could I possible want to do with my life.

[peabee]

for testing: LxPupSc-16.08.992.iso BUILD_FROM_WOOF='rationalise;024a875;2016-08-27 16:37:00 +1000'

[01micko]

If you want to merge rationalise into testing, then go for it. I would say that it's at alpha stage.

Bear in mind that @mrfricks is trying to push a reasonably stable xenial pup. I would leave testing alone, let him get that out the door then go for it.

[gyrog]

@peabee, Thanks for the iso, downloading now, I'll give it a go.

[gyrog]

What copies rc.sysinit and rc.update to the save-layer? I just did a fresh frugal install of peabee's iso, created a savefolder, changed some configuration, several reboots. And then I noticed that my LxPupScsve/etc/rc.d/ contained copies of rc.sysinit and rc.update that are identical to those in puppy_LxPupSc_16.08.992.sfs. I did not attempt to edit them, so I did not do any thing to induce an aufs copy up. It's not a rogue rc.update doing an unnecessary version update, since there was no rc.services etc... After deleting them from a different puppy, they did not come back, after several reboots. What's happening here? Why does it matter? Because it's a trap waiting to spring when someone tries to test a new rc.sysinit or rc.update by adding it in a ydrv.

[01micko]

Ok, so I'm running slacko64-6.9.5 built from slackware64-14.2 packages and it's all running pretty smooth. loaded libreoffice and @peabee 's latest chromium64 sfs's and no dramas there, even after rebooting a couple of times. I'll upload this for broader testing.

# grep -i 'build' /etc/DISTRO_SPECS 
#The distro whose binary packages were used to build this distribution:
#The version of the distro whose binary packages were used to build this distro:
BUILD_FROM_WOOF='rationalise;f045eca;2016-08-28 09:09:08 +1000'

The very latest (but I have a couple of patches to upload later today - slacko specific)

Probably take a few hours to upload on my brilliant connection.. might even use my phone so it's faster.

[01micko]

Bugger.. nearly as slow on my phone - 4G :confused:

[01micko]

@gyrog I see that bug too. Also functions4puppy4 is copied and rc.country

[wdlkmpx]

I don't see where it happens. in rootfs-skeleton i see greps in sfs_load (to be deleted) and a few other scripts.. but is that enough? Probably something outside woofce or maybe an aufs bug? Does this also happen in tahrpup gyro?

[wdlkmpx]

In the rationalise branch sfs_load does not even grep rc.sysinit. actually nothing touches rc.sysinit, pmodemdiag greps a temp rc_sysinit. There is no evidence..

[01micko]

anyway, slacko64-6.9.5 alpha is released .. http://www.murga-linux.com/puppy/viewtopic.php?p=920437#920437

[gyrog]

The rc.* copyup "bug" is part of first boot in LxPupSc-16.08.992, I'll try tahrpup 6.0.5. Edit: The result is the same in both tahrpup 6.0.5 and LxPupSc-16.08.992, on first boot, at the end of init, booting pfix=rdsh, only BOOTCONFIG and PUPSTATE are there, "exec switch" to desktop, "X" out of the 2 firstime config screens, and now the drifters have taken residence.

[01micko]

looks like they get copied up in rc.update... if they were never copied in the first place then that code wouldn't be needed.

[01micko]

Or maybe not but the fact that that code is in rc.update shows that it is intentional.

[gyrog]

In tahrpup 6.0.5, rc.update only copies rc.sysinit if it thinks it's a version update, and then it also copies lots of other stuff including rc.services.

[01micko]

I just did a pristine boot of my latest slacko and there is a huge clue as to why these files are copied up at initial boot in /tmp/bootsysinit.log

cheching if interface eth0 is alive...Files being translated:
 /etc/rc.d/functions4puppy4
 /etc/rc.d/rc.sysinit
 /etc/rc.d/rc.update
 /etc/rc.d/rc.country
 /usr/local/petget/0setup
Files being translated:
 /root/Choices/ROX-Filer/PuppyPin
 /root/.jwmrc-tray

So I guess it is the SSS or whatever it is that translates files.

[01micko]

So.. when quicksetup is run then SET_LOCALE="yes" .. therefore fixscripts always runs whether the file needs translation or not.. so is copied up.

Ok, we have the cause but the solution may not be so easy.

[gyrog]

But locating the cause is a good start. So if I stop quicksetup from running, there should be no copy-up.

[gyrog]

All DISTRO_IDSTRING processing in Woof-CE rationalise can be removed. It's not used at all in the new init, so all it achieves is a couple of warning messages from losetup in /initrd/tmp/bootinit.log. Everything works fine after I "head -c -32" the offending files.

[gyrog]

I just created a ydrv conatining the flag file in /etc/ that stops quicksetup from running, then did a first boot of LxPupSc-16.08.992, I didn't get any quicksetup screen, and I didn't even close the welcome screen, simply looked in /initrd/pup_rw/etc/rc.d and the 5 squatters were there.