New branches 'legacy' and 'rationalise'

[01micko]

For more information, I will refer readers to this blog post.

I have created 2 new branches legacy and rationalisation.

Important

As @mrfricks and myself are in development please keep changes to testing as bugfix only or minor updates unless there is a pressing need for change. After release of xenialpup and slacko-63x (bugfix) it is intended that rationalise branch will be merged with testing and deleted. Hopefully this will only be a matter of weeks. Of course all xenialpup changes before and at release should land in testing so that we can merge that with master.

'legacy'

legacy branch is purely in place to preserve old code. Read the README before making changes to that branch - changes are discouraged.

'rationalise'

rationalise branch is for sweeping new changes including but not limited to:

• deletion of un-maintained distros (arch, mageia etc) and older unsupported versions of distros (ubuntu - jaunty, karmic, lucid; debian - squeeze; slackware - 13.1)
• rationalisation of code in the woof build system
• changes in the way fdrv is dealt with. See #785
• corresponding changes to kernel-kit and rationalisation of code
• initrd_progs
• other stuff which may need enhancing. Suggestions welcome.

  Notes

I expect that @zigbert will keep updating his progs in testing. Therefore, before anyone makes a commit or PR to rationalise be sure that you have merged the changes in testing. This will make for minimal conflicts when rationalise is merged back into testing.

Of course any minor bugfixes and enhancements should land in testing first. So if you have a change to make, carefully consider if it is major or minor before committing.

As usual, have fun! :octocat:

[01micko]

I have removed some (not referenced above are arch and mageia and slackware/13.1).

I left T2and pet-based for now as they could be used as templates for a possible sabotage branch.

[wdlkmpx]

gyro posted 2 patches in the forum

1) To add help msg for his additions to init (bootloader menu) 2) To make fdrv behave like the other *drv (not tested), but I'll test it myself.

Regarding kernel-kit, I have to offer this as a slightly improved version http://murga-linux.com/puppy/viewtopic.php?mode=attach&id=97710

Check it out and provide your impressions about it.

initrd_progs: how exactly it should be merged? There are replacement for commands and bb-create-symlinks

I'll copy rootfs-skeleton to my system to make sure everything is ok, i'll also start building a pup to make sure I don't break anything.

[01micko]

1. @gyrog 's enhancements should be added - be aware of the newish support for UEFI too regarding the help messages.
2. I'll test your adjustments to kernel kit soon, but probably not until next weekend.
3. I think initrd_progs (for now) should be put straight in like kernel-kit - of course minus the git history. And, for now, it should be (or should that be 'maybe') optional to use it or go with the old method of using the pre-built stuff. Of course there are some hurdles - DISTRO_SPECS should be obtained from the target, plus other stuff that I can't recall right now. The timing will be critical too - in the context of 3builddistro-Z. Cross builds may be an issue. Not so much x86/x86_64 but more for x86/arm. It's a learning curve for us all!

[gyrog]

@01micko, When you say "@gyrog 's enhancements should be added", which branch do you mean? My current plan is to pull-request all my patches against the "rationalise" branch.

I actually have 5 in the pipeline. 1) Update "help2.msg", I'm not happy with the current version in the forum, it needs a bit more thought. The current "help2.msg" in "testing" is not wrong, it just doesn't reflect the extra flexability available now. 2) Change the way "fdrv" works, ready to go, but I think only peabee wants this now. 3) Update "help1.msg" to remove "fw_drv". 3) Tidy "init" by moving the fdrv load code to be with the zdrv load code, definitely minor. 4) Tidy "init" remove some references to tmpfs mountpoints that are no longer used.

[01micko]

My current plan is to pull-request all my patches against the "rationalisation" branch.

Indeed that is the one @gyrog :smile:

2) Change the way "fdrv" works, ready to go, but I think only peabee wants this now.

With some fairly trivial changes to kernel-kit this can be facilitated. I think it is the way to go going forward. I recall you questioning 'why is firmware part of the kernel?' and this addresses that. Also helps 'libre' puppy by keeping firmware out of the kernel sfs (zdrv).

@wdlkmpx has already began adding commits, most notably restoring 'init' and 'rc.sysinit' to the original place and deleting the old 3builddistro.

I look forward to your contributions and if anything is unclear, while I am a bit busy, I'll be as attentive as practical.

Cheers!

[wdlkmpx]

I think the static binaries (initrd_progs) must be in a repo. All of them in a tar.xz archive in this format:

initrd_progs-binaries-$date.tar.xz

I want initrd_progs to be used as a standalone app as well as part of a greater scheme (3builddistro).

Perhaps a new switch: -prebuilt (build.sh will and download use already compiled binaries) and 3builddistro can call initrd_progs/build.sh this way (or another):

(
    cd /path/to/initrd_progs
    rm -f initrd.*
    ./build.sh xz -auto -prebuilt -arch arm -specs /PATH/TO/DISTRO_SPECS &>/dev/null
    mv -f initrd.gz /pathto/maindir
)

if [ ! -f initrd.gz ] ; then
    error!!
fi

[wdlkmpx]

The init script needs a refactor to allow proper translations. The current implementation runs a modified init to do so.. crazy.

Network Wizard has the right approach, it sources the the main english *.po file (actually a script with variables only, each variable contains a message), then it sources other files according to your locale. The same logic is applied in my modified MSCW.. that's the way to go.

This is something that needs to be discussed with many people, specially translators.

And those translations can go directly in the initrd-tree.. the initrd_progs repo can be used to advance this..

[gyrog]

The init script could do with more than just a refactor to facilitate translations. It could use a major rewrite to simplify it, as well. I have some ideas on that, but maybe here is not the place to discuss it.

[wdlkmpx]

gyro I think you should open a thread about the init rewrite and the work should be done in initrd_progs... when initrd_progs lands on woof-CE some directories will be deleted and in fact that would mean a new version of woof

I strongly recommend you to "build" a initrd.gz file with a tahrpup or slacko, use it to test, and inspect the new apps and possibilities. Just run ./build.sh (will download about 150-200MB of stuff) and hit enter at any prompt (2), and then a initrd.gz file is ready.

I'm also simplifying build.sh while adding some new functionality to make it completely independent ( https://github.com/puppylinux-woof-CE/initrd_progs/commit/a627c925f32533932df478e5f94436ad2797649c ), so if you have any issues, just let me know, and do add your changes as soon as you see it's working, if something is broken i'll perhaps notice it and i'll notify you.

[gyrog]

@wdlkmpx, Thanks for the suggestion, unfortunately I haven't girded my loins sufficiently to undertake such a large project as rewriting init, yet. Although I'm open to the possibility of discussing some ideas, so I will consider opening a thread.

[wdlkmpx]

The whole "woof-arch" should disappear or at least most if it. I can add a "pkg" to initrd_progs to compile and archive all the support/boot apps and that can be downloaded afterwards.

The commit history will grow by 15mb for no apparent reason and, i would suggest to restart in a few months but before that put the repo including commit history in a tar.xz archive ready to download for everyone to see what happened before and look for the legacy branch, though nothing of value would be lost, except perhaps Arch compat, but again and i can make it come back, but it has gone the wrong way..

I wonder what .bac files need translation into a more reasonable language, i don't understand baCon though, looks primitive like a Windows shell script

[01micko]

BaCon is structured Basic. It's fairly easy to grasp. It converts the code into C then compiles it, but the C sources produced are rather cryptic. Probably just as easy to go back to shell.

Another thing to consider is updating encryption to support dm-crypt instead of the old DES and XOR, then we can update losetup >= util-linux-2.22 instead of sticking with the ancient version which dropped support -

Util-linux 2.22 Release Notes [Sep 4, 2012]
===========================================

 The cryptoloop support in the commands mount(8) and losetup(8) is DEPRECATED.
 This is the last release where encryption= mount option and -e,-E,--encryption
 losetup options are supported.

https://www.kernel.org/pub/linux/utils/util-linux/v2.22/v2.22-ReleaseNotes

[wdlkmpx]

Yeah that should be a priority. I'm already using util-linux 2.27, the losetup in the new initrd is from util-linux 2.28... looking at the config file of the kernel i'm using, i see this:

CONFIG_DM_CRYPT=m

The agent said that this was implemented in fatdog long ago, so anybody using fatdog paste code here to look at it

As usual, the Arch wiki also provides lots of info to read https://wiki.archlinux.org/index.php/Dm-crypt

I'll look into this

[wdlkmpx]

After commit https://github.com/puppylinux-woof-CE/initrd_progs/commit/f3f2944bb2516cd9973fb11b231cba1286c89279 initrd_progs is ready to be merged so I can actually face a problem that I will solve.

I will create a pull request

[wdlkmpx]

I opened a thread at the forum:

http://murga-linux.com/puppy/viewtopic.php?p=907290#907290

meanwhile micko... i was not able to get the armv6l cross compiler to work. Perhaps it works in your slacko64, type ./build.sh -arch armv6l -auto and see what happens..

[01micko]

busybox built, exited on disktype.

[wdlkmpx]

Well, slacko packages are taking forever to download. If there are 10 mirrors, on each download choose a random mirror (between 1 and 10)..

Obviously this is just me doing and thinking about everything, and it will continue that way until i shut up... so I wonder where a lengthy discussion about all of this can happen..

[wdlkmpx]

I think that by now it's safe to wipe compatibility with Red Hat or RPM based distros such as Mageia and Scientific Linux and just remove any RPM traces from the build scripts... petget should keep supporting that format, some very basic binaries should work on any distro..

It would also be a good idea to create a newer LFS Puppy, like the old one, for those who want to port Puppy by compiling themselves, but this task is huge. I know this is something already done, perhaps by fatdog? (never used it). If there is code, something already usable, we should just use it and try to improve it................

[wdlkmpx]

2createpackages can get very heavy when processing packages with lots of files, reminds me of the new2dir script, the same logic and heavy processing. It tries to strip binaries.. and that happens again in 3builddistro-Z

Half the code in 3builddistro is commented out... should be deleted.

I see repetitive code in all the build scripts, this can be simplified by sourcing a script with functions..

[dimkr]

I think we should start with mass-scale code deletion, before we do any refactoring. Something in the spirit of grep -l magia $(find . -type f).

[wdlkmpx]

Yes of course, time to delete

[dimkr]

I created pull requests for Scientific/RHEL and Arch. Low-hanging fruit.

[wdlkmpx]

gyro has created a thread about the init "rewrite" or "rationalisation". it summarizes very well all the needed changes, so it's a roadmap, he will lead this.

http://murga-linux.com/puppy/viewtopic.php?t=107178

it's unlikely that both of the topics will get comments, so perhaps the init one should be moved to the cutting edge area where he feels more comfortable.

[wdlkmpx]

initrd-progs has been integrated... most likely

[wdlkmpx]

So I saw this line; echo "Removing extra documentation"

And made me think................. this overrides the rules.. completely. Perhaps the packages need only minimal processing doing things at large scale, and then when everything is done remove or move extra documentation, extra locales, extra development files...In fact a sfs can be created of each.

How about that?

[wdlkmpx]

I built slacko with the all changes I made... seems to be ok. Boots to desktop in my toshiba nb100.... The size of main SFS is the same... running with a tahrpup kernel (accidentally chosen).

there's a d-bus issue.. gnome-mplayer complains something's wrong. So I have to run dbus-uuidgen > /etc/machine-id.. i'm not sure if this also happens in 6.3.0

[wdlkmpx]

I think "1choosepackages" and "4quirkybuild" (and all things "quirky") should be deleted, but i need to know what others think. I'm thinking about opening a thread and ask questions.. like this.. should be deleted? Yes or not. Give me a reason why it shouldn't.. something like that.

I would also delete the woof_gui, create 00_build.conf, func.sh or something, but i'm not sure what to do, or how to do it.. this is not as simple as initrd-progs

[dimkr]

I think Quirky-specific stuff and other-distro-packages-to-PET-convertion stuff should be killed with fire.

[wdlkmpx]

Well, iguleder, how about a pull request to see extent of the changes? I could certainly merge it hehe

[dimkr]

OK, I created https://github.com/puppylinux-woof-CE/woof-CE/tree/remove-quirky. Hope I haven't missed anything.

[01micko]

Not mentioned in the start of this thread is crap like xorgwizard and ddcprobe (both legacy crap from pre xorg-7.3 see #583 and also xorg branch). They need to be removed. ddcprobe is ancient and fails if a TV is used as a monitor - sometimes via VGA input but all the time via HDMI. And worse is xorgwizard-automatic. Why try and fake something that the actual program (Xorg) does well enough itself? (With a better success rate - even on old crap hardware).

This should have happened years ago but there has been resistance to include full udev (now eudev) support.

[wdlkmpx]

Well micko, this is the time to include full eudev support then, old puppies are gone.. everything is ready. I don't know much about it, but i'll try to learn and help.

iguleder, it looks right, should be merged directly or through a pull request?

Looking at the woof-distro there is enough for everything, the arm stuff is in a sad state. Need people to update the arm distro compat, debian and gentoo (the main antagonist to systemd alongside slackware and perhaps devuan?)... that's enough work.

People should work with LTS versions only... specially for ubuntu where every year some versions are moved to the old-archive.. raring and saucy are already gone.. crazy.

We should proceed.

[dimkr]

should be merged directly or through a pull request?

Up to you, I'm a CLI guy myself :)

[wdlkmpx]

I merged it!!

I'm looking at the 'remove-legacy-distro' branch, it looks ok too. It removes very old and and abandoned arm distro compat, if it's to happen again, it has to use raspbian template (wheezy) and templates for trusty and later (for ubuntu). So I think it's right. gentoo, i never of any gentoo puppy, perhaps it was left unfinished, just like Arch Compat, but somebody may to take lead and create something that works.

Only slackware 14.1 is supported... i think it's ok, 13.0 is getting old and 14.0 is superseded by 14.1... micko might agree, i'm also testing slacko, so i'll probably provide some fixes for some old hardware.

x86 still has precise... ok, raring and saucy should be deleted. So... I'm merging that one as well.. i take full responsability for this.

[dimkr]

i take full responsability for this.

:laughing:

[wdlkmpx]

Another issue to address is DISTRO_PKGS_SPECS-, it's really ugly having to read a very long line, DISTRO_PET_REPOS looks like a very generic file for a specific arch, so it should not be inside a specific distro.. i'll take care of this

With this reduction, it's easy synchronize the pups a bit, using the same busybox version, bash version, etc.... but that requires cooperation.

[dimkr]

I tried to remove xorgwizard more than once, but way too many things depend on it. The event thingy, pinboard icons, xorg wizard, X startup scripts ... it's a huge pile of spaghetti. Things will break, although I don't think it's a major concern now (after woof-CE lost 12k lines) :smile:

[dimkr]

By the way: if we're already into dropping legacy stuff, we can replace busybox init, syslogd and klogd with https://github.com/dimkr/sdaemons/blob/master/init.c and https://github.com/dimkr/sdaemons/blob/master/syslog.c. I created a modded, ultra-lightweight tahrpup without VT support, that uses these.

[wdlkmpx]

Uhmmm well, i'd suggest to 1st remove support for xvesa and old xservers a few hundred lines will go away... support only Xorg server, then try to beautify the code and fix some stuff, and then finally try to remove it..

Yes we can use your init and syslog, or we can even replace the busybox init, syslogd with yours.. inside busybox.. but is that a good idea? I already put waitmax, vercmp, guess_fstype inside busybox.. that is in initrd-progs (the same config file and patches should go to the petbuilds)

[01micko]

I already rewrote xwin and a few others in the xorg branch. Moved ~/.xinitrc to /etc/X11/xinitrc. It is (was) a work in progress but many machines boot to desktop just fine, and touchscreens work. Still run it on my Acer W500 tablet. Udev does need some fine tuning though. See http://murga-linux.com/puppy/viewtopic.php?t=101827

Feel free to cherry pick that branch or just peruse it for ideas. I broke it trying to move fido (ugh) and haven't had much time for it lately.

[wdlkmpx]

ddcprobe is useless nowadays, xorg wizard can be simplified a bit until it gets removed... basically to choose the correct driver with a few standard choices 800x600, 1024x768, etc, since precise puppy I've been seeing a very smart xorg, that's why I added this commit: https://github.com/puppylinux-woof-CE/woof-CE/commit/7519bab342259d2b9728d3cee9842e5fc11d6b0d

So we must remove ddcprobe, and try to simplify xorgwizard/automatic until the judgement day. Of course the xorg branch must be used a reference, but would like to know the commits done there to see the diff, it's hard to tell when a branch is merged into another. Or at least would to know the files that have changed to make a local diff hehe..

[01micko]

Testing might still merge cleanly.. dunno. I can then do a fairly clean diff.

[01micko]

Ok, its over 22k lines! But most of that is the template xorg_base_new; file attached. It removes xorgwizard-automatic. Note that keyboard changes go to the /etc/X11/xorg.conf.d/ dir. I didn't get as far as synaptics/alps/elantech touchpads but they worked on the 2 laptops I tested on at the time.

testing_xorg-diff.tar.gz

[wdlkmpx]

Ok, then I'll start editing stuff in rootfs-skeleton, 1st remove traces of the old xservers

[wdlkmpx]

My expriment is working locally, needs testing, if it's bad i'll revert some changes..

Since commit https://github.com/puppylinux-woof-CE/woof-CE/commit/1bf49330d9cab47f37191845ddbd8ba6e297a152 there is another way to deal with video hardware... using the current xorg.conf, then falling back to a generic template, and then there's a chance run xorgwizard (of course it may crash during any of these steps). It's very simple..

[peabee]

Built LxPupSc-16.06.103-rationlaise:

https://sourceforge.net/projects/lxpup/files/Other/test/woof-ce-rationalise/

using:

BUILD_FROM_WOOF='rationalise;1bf4933;2016-06-19 23:23:49 +0000'

seems sane so far tested on my desktop with nvidia graphics....

the build definitely went quicker - particularly the built-in .pet build stage

what should I test?

already tried fdrv - OK

Cheers

PeeBee

On 20/06/16 05:34, wdlkmpx wrote:

My expriment is working locally, needs testing, if it's bad i'll revert some changes..

Since commit 1bf4933 https://github.com/puppylinux-woof-CE/woof-CE/commit/1bf49330d9cab47f37191845ddbd8ba6e297a152 there is another way to deal with video hardware... using the current xorg.conf, then falling back to a generic template, and then there's a chance run xorgwizard (of course it may crash during any of these steps). It's very simple..

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/puppylinux-woof-CE/woof-CE/issues/789#issuecomment-227048640, or mute the thread https://github.com/notifications/unsubscribe/AFpv0VZwC8MKoiXGa8EG9STN0uFQWbmlks5qNhhVgaJpZM4IuPH7.

[wdlkmpx]

How to explain this. Puppy uses video hardware profiles and stores them in /etc/X11, now that code is restricted to xorgwizard-cli, but only to produce a working xorg.conf, xwin, xorgwizard-automatic, will ignore it completely (before this puppy would use that profile if it matches the hardware, or to detect hardware changes and do some stuff to cope with it [it never worked if there was no previous profile]).

With this new approach, people should try not to touch xorg.conf, because it actually works and loads the correct driver every single time (i tested 4 ancient video cards in my precise install), but when they really want to mess with xorg.conf and somehow break it, then it will be reverted to the generic template in xwin.. assuming the system does not crash. Of course this is how the code is supposed to work, let's keep it that way until someone complains.. then i'll revert xwin and xorgwizard to a previous state (before i removed the ddcprobe stuff).

Can you upload your iso to an alternative location? Maybe mega, I'm downloading, but it's too slow..

[wdlkmpx]

By the way peebee has posted a message on the forum saying that there's a problem with fdrv @gyrog . Now i'll stuff iguleder's init and syslog into busybox and test

[peabee]

Ive uploaded the iso (and the delta) to:

http://smokey01.com/peebee/downloads/

On 20/06/16 15:22, wdlkmpx wrote:

Can you upload your iso to an alternative location? Maybe mega, I'm downloading, but it's too slow..

[wdlkmpx]

I already downloaded it hehe, it's working quite well, exiting to console, running xwin again....looks ok, this should be the default puppy (but without the whole lxde stuff, just the basic apps and jwm), so i can copy/cut and paste files using shortcuts with the keyboard