Raspberry Pi build broken

[01micko]

I just did a sanity check, doing a No-X build, crashes at 3builddistro stage

[01micko]

Rolling back 3builddistro to one from late 2012 seems to have fixed it. I'll do a diff and find the problem.

[01micko]

Found the problem. Barry changed it so that the .img is mounted. This is an vfat/ext4 image, Busybox mount failed with the vfat and the script closes. Changing to full mount fixed that but mounting the ext4 no journal partition failed. It may be a problem with the image so I'll try another.

[woodenshoe-wi]

I think I found the other problem with the second partition. The offset was being calculated with the value from the first partition. $P1STARTSECTORS instead of $P2STARTSECTORS

Sent pull request #455 EDIT: That one was against master, #456 is against testing.

[mavrothal]

Where do we stand on this one?

[01micko]

out to pasture?

[mavrothal]

Lovely place to be :D

[woodenshoe-wi]

Well... This spring I couldn't resist and I got a Pi2 and over the summer I did succeed in hacking the woof build scripts enough to make a bootable image that would work with either the 512MB model B or the new Pi2. The Pi2 is a quad core and uses a different kernel, so I modified the scripts to load two kernels into the image. I had success treating the kernels and firmware like any other package and loading them from raspbian.org with the rest of the packages, but there was a puppy specific program used by the fixmenus script, and the ARM version is an old version ( I think jwm-xdgmenu, but that's just from memory ) that can't parse the config file as it currently is in woof. I have the modified woof script files somewhere, but haven't gotten around to making the changes into git commits yet, and am kind of busy getting ready for winter outside.

Short answer: Probably still broken, at least for Pi2, but I know it can be fixed. (but it may take me a while to get around to it.)

[woodenshoe-wi]

I found back some of my notes... ... xdg_puppy-0.7.6-15-binaries_only-armv6.pet is too old to handle the "MENHEIGHT" option in /usr/sbin/fixmenus ...

[01micko]

You can compile the newer version. It is at http://distro.ibiblio.org/puppylinux/sources/x

[woodenshoe-wi]

I may be missing something, but the only version I could find was xdg_puppy-0.7.6-9.tar.bz2 Is this newer than 0.7.6-15?

[woodenshoe-wi]

I tried it and it worked!? I don't understand the version numbering.

[woodenshoe-wi]

Assuming you wanted the pet, I uploaded it to Drop Box. https://www.dropbox.com/s/hmk012wqld98w8f/xdg_puppy_jwm-0.7.6-16-armv6.pet?dl=0 I'm guessing at the version number, but I figured that if it worked it must be newer than 0.7.6-15.

I also used petbuilds to compile JWM. https://www.dropbox.com/s/l34rhvxmw8ogevu/jwm-2.3.2-armv6l_hf.pet?dl=0

Both of these were compiled in a test woof build based on Raspbian wheezy on my model B Pi. I think they would go in Packages-puppy-wheezy-official?

I haven't finished making all the changes into commits, but I did finish a few and pushed them up to my git-hub account. They have only been tested with builds for raspberry pi, would it be OK to issue pull requests or do they need to be tested on an x86 build first?

[mavrothal]

The one commit I found in your repo at ( a4808b3 ) appears safe, but is always a good idea to run a test build for x86.

Regarding the pets, why not in Packages-puppy-armv6-official. Is there any compile option that would break them in other flavours (that support hard float of course)? Or maybe open a Packages-puppy-armv6hf repo? Not that makes a huge difference but if there are more to come is better to get them together I think.

[woodenshoe-wi]

I didn't use any special compile options, but I'm assuming they're dynamically linked against system libs from Raspbian. Some pets from Packages-puppy-armv6-official didn't run right in my Raspbian based builds so I assumed my pets wouldn't run right in other flavors either. If it is a problem to compile new pets for each flavor I could try compiling them statically soft float, then they could run on any armv6, but as they are now they may or may not work depending on what library versions the other flavors use.

I'm still tracking down dependencies so I haven't made my changes to DISTRO_PKGS_SPECS-raspbian-wheezy into a commit yet, but I didn't know if you wanted me to dump all my changes on you at once, or as I finished testing them. a4808b3 735b39f 412006c

[mavrothal]

Pulls with many commits have the risk to get rejected because of some little issue in one of the commits, but makes more sense to get them in one pull if they are related as in this case. Re-issuing a pull with one commit or many does not make a lot of difference really so I would rather say go with one, but it depends on how comfortable you are with git and pull requests.

[woodenshoe-wi]

The three commits I finished should allow a build that will boot to the command line, but JWM won't start and jwm-xdgmemu can't parse the menu properly. I'm still tracking down dependencies of dependencies, and if I have to do a x86 test build I would rather wait until I am all the way done.

~~The two issues that I can't solve myself are JWM and xdg_puppy. One option is to use the dynamically linked pets in Drop Box, the other option is to try to compile xdg_puppy statically against uclibc and take the files from petbuilds and make a template to use JWM from Raspbian.~~

Until this is figured out, my "working" builds won't be reproducible by others.

[woodenshoe-wi]

I did a test build last night with just the commits in https://github.com/woodenshoe-wi/woof-CE/tree/fix-pi-build and the problems with JWM and xdgmenu somehow went away by themselves.

I still need to do a x86 test build, but I think I am close to being finished.

[mavrothal]

:+1:

[woodenshoe-wi]

Well, I've been running into snags with my x86 build. First I tried the slacko 14.1 build running woof-CE in precise but ran into problems in 2createpackages, so I installed the latest slacko on my parent's computer and then 2createpackages worked but when I tried to run 3builddistro it said I had to run 3builddistro-Z instead which is not the script I edited so that won't work for a test.

I also was working on getting rpi-update to work and I think I have to add one more line to fstab for /boot which is done in 3builddistro I think.

I want to try to finish the fix for rpi-update before I try another x86 build, but I can't recommend actually running rpi-update because I think it breaks the arrow keys on my keyboard.

I guess I'm not as close to being finished as I hoped :-(

[mavrothal]

I guess I'm not as close to being finished as I hoped :-(

You are closer than anybody else :+1:

[01micko]

I ordered the new 'el cheapo' pi. Still set me back AUD$32.50 :scream: but it does come with hdmi adapter and USB adapter. We should be able to make use of FatDogARM repo by @jamesbond3142 :wink:

[dimkr]

It's ARMv6 like Raspberry Pi 1 :sleeping:

[01micko]

Doesn't matter.. we can build up a repo.. :+1: petbuilds should help.

Even better really. We can revisit GTK 1 and Dillo and all the good old small stuff.. run it all in RAM and show those other distros that Pi can Fly. :grin:

[mavrothal]

But let's have something that builds first :wink: (have to do something about it...)

[01micko]

TBH I don't like the idea of downloading an image. It should be created on the fly with builder's choice of ext2, or ext3,4 (no journal)., with builder's choice of size.

Actually, if you make the small FAT partition, then swap, then the Linux partition it becomes trivial to resize the thing, so a default of fitting on a 2 GB media is easy to resize (with gparted or tool of choice).

[woodenshoe-wi]

For those of you that have a Pi or are buying one, I have my work uploaded to git-hub and you can start playing with it now if you would like. https://github.com/woodenshoe-wi/woof-CE.git If you are running anything older than Slacko 6.3.0 you will need to update debdb2pupdb or you will have errors finding the raspberrypi-bootloader-nokernel package. See https://github.com/woodenshoe-wi/woof-CE#raspbian-build

On the subject of fixing the the Pi build in woof-CE, I tried a slacko 14.0 build and when I tried to boot the CD on my laptop I got this

[wdlkmpx]

A lot of changes have happened since the fire nation attacked. I wonder if we can port some of the commits from woodenshoe. I hope his pet pkgs are in the ibiblio arm repo or something

[woodenshoe-wi]

I thought I would try merging https://github.com/woodenshoe-wi/woof-CE/tree/fix-pi-build and do a test build for my pi. It booted and everything seemed to work until I realized that I had merged master and not testing.

When I tried to merge testing instead of master I found that 3builddistro had disappeared... From 3builddistro-Z in testing ...

if [ "$DISTRO_KERNEL_PET" != "Huge_Kernel" ];then
    echo "You can not use this script. Use 3builddistro instead or click the button"
    echo "named 'BUILD DISTRO (normal build)' in the Woof GUI"
    echo "Press enter to exit."
    read normal_build
    exit 0
fi

Back when I had been working on this, it seemed that 3builddistro-Z was for "Huge_Kernel" and 3builddistro was for everything else. Is the 3builddistro-Z script in rationalise going to support alternatives other than "Huge_Kernel"?

I was using the kernels from Raspbian and have no plans to try to compile a kernel on my pi.

[wdlkmpx]

You're raising good points, many things have changed while woof was being simplified. I guess someone has to compile a huge kernel or cross-compile. For arm boards people usually cross compile in a x86 host.

The rationalise branch has many more changes, so i think everything has to be restarted. But we're lacking devs. probably it's all dead now according to a vision i had..

I'm not sure, but this can be the solution http://autobuild.buildroot.org/toolchains/tarballs/

This stuff runs in x86_64 systems and cross-compiles for armv6 armv7, etc. Assuming everything is properly set up, that's something to be added to the kernel-kit. Something similar to initrd-progs, which uses the tarballs from aboriginal linux i guess...

For this maybe we just have to copy one of the existing kernel configs and tweak it for an arm build, or stole a proper config and just use it, the latter seems to be most reasonable solution.

[woodenshoe-wi]

The other option I was going to suggest was to have a separate 3pibuild script like the 4babybuild and 4quirkybuild scripts, but I see that they have been removed too.

I do have a little experience with buildroot, and was trying to make a kind of cross between buildroot and the Puppy Linux idea of a devx sfs, but I don't plan to compile a kernel unless I am compiling everything else from source. If I am using debs from Raspbian, I want to use the kernel from Raspbian also.

I was also able to use rpi-update https://github.com/Hexxeh/rpi-update to update the kernel and firmware from a running pi. I don't know how normal it is to compile a kernel for a pi yourself.

Maybe someone could pack up the Raspbian or github kernel into the format expected for a "Huge_Kernel", but it would not stay up to date automatically.

[wdlkmpx]

Good points, where can i download a pi kernel, just to see. Maybe i can do something. Although you will encounter new problems as the old .pet kernels stuff doesn't exist anymore.

Well, i think you're right, 3pibuild makes sense actually.

I've never used an arm board, so this is unknown territory for me. And it's unlikely i'll buy one, i read it's slow. i already have and am struggling with slow hardware due my masochistic personality traits.

So, a pi puppy is basically a full install right,,, no sfs's, etc? The there should be no problem using a raspbian kernel..

Well, at least in x86 the kernel is always compiled for puppy, and it works in basically any woofce puppy, the only restriction that can be found is probably when compiling new modules... haven't tried.

A couple weeks ago i was looking at the slackware website, and found they also provide an arm port, but they link to fatdog.eu. I wonder if the fatdog team has anything to do with it. Might be of interest.

http://sarpi.fatdog.eu/index.php?p=downloads

Overall slackware, arch, gentoo and a few other distros don't split their packages, so everything is cleaner and less confusing.

[wdlkmpx]

The build process is being automated so that only a couple of questions are asked, everything else is automatic, most options can be specified through a config file (_00build.conf). Well, for x86 at the moment.

The arm stuff remains in the darkness, but hopefully it will be sorted out starting from 2017 haha

This is where the action is happening: https://github.com/puppylinux-woof-CE/woof-CE/commits/rationalise

Everything in that branch will go to testing soon

[woodenshoe-wi]

The Raspbian kernels are located at http://archive.raspbian.org/raspbian/pool/main/l/linux-latest/ According to the wheezy package list linux-image-rpi-rpfv_3.18+46+rpi4_armhf.deb is the recommended version for the pi 1 and linux-image-rpi2-rpfv_3.18+46+rpi4_armhf.deb is the recommended version for the pi 2. I have not tried to do a build with jessie.

A raspberry pi also needs boot-loader / firmware files.

The Raspbian ones are at http://archive.raspbian.org/raspbian/pool/firmware/r/raspberrypi-firmware-nokernel/ According to the wheezy package list raspberrypi-bootloader-nokernel_1.20150212-1~nokernel1_armhf.deb

When I was first getting my pi to boot alpha4 http://murga-linux.com/puppy/viewtopic.php?p=665256#665256 I was grabbing kernels and firmware from https://github.com/Hexxeh/rpi-firmware

I was treating the Raspbian debs as regular packages and the woof scripts would figure out which versions were specified in the package lists, so only the package names had to be specified in DISTRO_PKGS_SPECS-raspbian-wheezy, eg. yes|raspberrypi-bootloader-nokernel|raspberrypi-bootloader-nokernel|doc>null Since a raspberry pi build is a bit different than a normal puppy build, installing two kernels and a bunch of firmware files, using a separate script could just download the packages like I am doing now and not bother with a "Huge_Kernel" package. If a separate 3pibuild script would be OK I could start with the last 3builddistro script, strip out anything not pi related, and try to implement any _00build.conf features using 3builddistro-Z as a reference?

Currently a raspberry pi build is kind of like a full install. The kernels and firmware are installed into the first partition of a filesystem image with a FAT filesystem (the boot-loader needs FAT), then there is a swap partition in the image and a linux type partition where everything else is installed. I avoid loading the swap partition because I am worried it could wear out the SD card. The Raspbian kernels include squashfs, and a devx sfs is made during the build process that can be loaded on the pi to allow compiling. The Raspbian kernels also include overlayfs, but I don't know if it is feasible to get puppy to work with overlayfs.

[wdlkmpx]

OK, i believe you have the complete idea. so it will be better for you step in and join.

Well, puppy uses aufs, but it doesn't matter when the filesystem is complete, i.e. full install. although the code always assumes a layered filesystem, but works nonetheless (i think).

The initrd init used in the x86 builds always assumes aufs and sfs files. it's for a modular approach and fancy handy stuff.

Overlayfs is the winner indeed - in this battle against aufs. but it's not supported in puppy.

when it comes to packages, i can help get everything ready, i mean editing distro_pkgs_specs and uploading files to the repo so the build works.,. if you have any files ready, just open a pull request or post the files somewhere.

regarding 3pibuild, i guess you can do it, but the whole process was meant to unify eveything, applying the same patches and tweaks for a puppy build.

in rationalise, the scripts are, let's say, quite small compared to what they used to be and more changes are coming, so i think you might want to do your stuff directly in rationalise after it's merged to testing. that branch is for experimental stuff, but once there is a number of commits, everything else tends go there, just like it's happening now.

my thinking is try and do something even if that breaks something else (will be fixed), rather than do nothing

[woodenshoe-wi]

I looked at the 3builddistro-Z script in rationalise and recognized a lot of it from 3builddistro, I had thought that 3builddistro-Z was a complete rewrite and all of that would be gone. I will try to make my changes to 3builddistro-Z.

Currently a raspberry pi build does not use an initrd, although I think the pi boot-loader does support one.

Unless there has been a format change, I am familiar with all the distro_pkgs_specs files, and the two pets I mentioned earlier may or may not be needed. Most of the fixing had to do with finding dependencies and fixing templates, I can't think of anything else I had to compile.

[wdlkmpx]

A complete rewrite is in the woof-next branch, it's a different approach and makes puppy part of the parent distro or something like that. it discards many things, this is what probably would have been the right direction as in woofce there is not an agreement to follow a certain path as i'm seeing in the jwm stuff.

I'm building a precise pup testing the changes i make and i'm quite familiar with the package dependency in the ubuntu/debian/devuan/trisquel configs.. because they all follow debian's scheme. all of them have cdrkit instead of cdrtools, all of them have libav instead of ffmpeg... the latter choice made compile ffmpeg and recompile all the apps that use libav. this makes puppy a truly alien guest in the grand scheme of things.

i just compiled jwm and rox, and removed jwmconfig and jwm themes from the distro_pkgs_specs... to use the woofce jwm config

i think many old packages can be removed to make the configs look cleaner and easily maintainable. i deleted about 300 entries and everything seems more reasonable. and it still has almost all the stuff.

[woodenshoe-wi]

900 Not polished, but something to start with.

[wdlkmpx]

Merged. To make your steps reproducible, you might want to suggest changes or make them yourself if you wish. try to build starting from 0setup and then identify what's needed, raspbian-specific stuff.

[woodenshoe-wi]

I think it should be reproducible. I haven't started a build with an empty local-repositories directory and re-downloaded everything, but if I remember right when I was working on this a year ago I was avoiding using anything that was not in one of the repos.

11ff0ff stops 3builddistro-Z from installing a "Huge_Kernel" when using Raspbian kernels, If you want me to do a test x86 build could you suggest a particular version that you are familiar with building?

Regarding 01micko's comment I get nervous partitioning and formatting but I did make a 1GB filesystem image without a swap partition. Raspbian uses a swap file and the puppy swap file maker seems to work fine. Without the firmware files that were in the other filesystem images it compresses down to 155K. The only problem is that gparted won't resize the partition from a running pi because it is mounted.

This isn't a zip file, its an xz compressed filesystem image, but GitHub didn't accept it so I added a .zip to the filename. raspi-sd-1gb-skeleton-ext4_nojournal-8nov2016.img.xz.zip

Do you know a packages-templates expert who could review adb4a21 ? It would affect all builds, not just Raspbian builds.

[wdlkmpx]

I did build a puppy after merging your changes and nothing in the x86 build process was affected. puppies i'm familiar with... i built a slacko slackware 14.2 twice, so that one is working quite fine. the change your proposing looks fine too.

don't worry about paritioning and formatting, i guess that's not that hard to implement. meanwhile micko or peebee might want to upload your img (with a md5 sum) to here: http://distro.ibiblio.org/quirky/arm/sd-skeleton-images/

as that's the url in 1download..

galculator... only raspbian wheeze is using the template, all the other compat distros pick the a pet file (no templates), so it's ok. templates affect all puppies that pick that specific package from the compat distro repos, when everything seems ok, it stays for all. if there are possible issues/conflicts.. then a template for a specific distro or variant is created (dbus_slack for example (maybe))

[peabee]

micko or peebee might want to upload your img (with a md5 sum) to here: http://distro.ibiblio.org/quirky/arm/sd-skeleton-images/

I don't have access to quirky - only puppylinux - would imagine only BarryK can upload to quirky....

[woodenshoe-wi]

If you think they are OK,

902 #903

At this point resizing the partition from a running pi is a problem, so the 1GB image isn't that useful right now.

[woodenshoe-wi]

I did get a Pi3. Found out that the Pi3 wouldn't boot my wheeze based Puppy images so I started working on a Jessie build. #908

There is a problem with debdb2pupdb that is triggered by the main Packages file in Jessie,

Processing Packages-raspbian-jessie-main into a standard format...
./0setup: line 352: 28338 Segmentation fault      ${DEBDB2PUPDB} ${DISTRO_BINARY_COMPAT} ${DISTRO_COMPAT_VERSION} > /tmp/${ONE_PKGLISTS_COMPAT}temp
Processing Packages-raspbian-jessie-nonfree into a standard format...
Processing Packages-raspbian-jessie-contrib into a standard format...
Processing Packages-raspbian-jessie-firmware into a standard format...

I was able to work around it by making a Packages-raspbian-jessie-main in an old version of woof-CE that still used the old slow bacon debdb2pupdb. https://www.dropbox.com/s/rot55z9zbgmlzz6/Packages-raspbian-jessie-main.xz?dl=0

I tested the build on my Pi1 Pi2 and Pi3, and it runs on all of them. I don't have a Pi zero but it should run on that too.

Currently it is using JWM from Raspbian and jwmconfig3. I was ending up with an empty menu trying to build without jwmconfig3, maybe its xdg_puppy again. JWM appears to hang when shutting down, just click the desktop or scroll the scroll wheel and jwm will shut down. I had to test the devx anyway so I compiled jwm-2.3.6 https://www.dropbox.com/sh/49y6kup5q6w88hi/AAA_HZmkwwD0QZkC8g50kRbZa?dl=0 The new JWM does fix the hanging on shutdown.

If you want to load the devx, before running sfs_load you have to open a terminal and run modprobe loop and before using git to clone https repos you have to run update-ca-certificates

Midori won't download files, I just use wget, and it appears /var/run is directly on the SD card so it is probably writing to the SD card way too much, but at least it is a start :grin:

[wdlkmpx]

Good job.. If raspbian jessie works fine in all rpi versions.. do you think it's ok to delete the raspian wheeze configs/build recipes?

iguleder aka dimkr actually fixed debdb2pupdb several times.. so the last version should be working ok... are you cross building or building in a pi?

i updated woof-arch/arm/build/support/debdb2pupdb .. it was never updated since the first commit, at least according to this:

https://github.com/puppylinux-woof-CE/woof-CE/commits/rationalise/woof-arch/arm/build/support/debdb2pupdb

[woodenshoe-wi]

I basically started with grep -v '^no' DISTRO_PKGS_SPECS-raspbian-wheezy > DISTRO_PKGS_SPECS-raspbian-jessie and then started updating what was left. It would still be in the git history if anyone did need to reference anything that I removed, and I don't plan on doing anything more with the wheeze build, so I think it is OK to delete them.

I do the woof-CE builds on my laptop, mostly x86 host arm target but I did try an x86_64 host arm target build and I still got the segfault with the debdb2pupdb from rationalise.

I compiled JWM natively on my Pi1 to test the devx. I also got network swap to work so I could get around xz running out of RAM on my Pi1.

I don't know that the PPM works on the Pis but I can confirm that downloading debs from the raspbian.org website and clicking on them to install works, and clicking on the JWM pet that I made worked also.

[wdlkmpx]

peebeeeeee create 'pet_packages-jessie' in http://distro.ibiblio.org/puppylinux/arm/ and upload these files to that dir: https://www.dropbox.com/sh/49y6kup5q6w88hi/AAA_HZmkwwD0QZkC8g50kRbZa?dl=0 also add this file (in arm/): https://github.com/puppylinux-woof-CE/woof-CE/commit/738e8df16a241f44f8125510639db8f5a5a086c5

i can confirm debdb2pupdb segfaults, so i'll look into this..

[peabee]

Done

[dimkr]

@wdlkmpx , make sure debdb2pupdb is built from the latest debdb2pupdb.c

[woodenshoe-wi]

I noticed in the git log that you moved the SD card images to the puppylinux repo.

I am able to boot the initrd that woof-CE generates with the attached cmdline.txt and config.txt

Of course it errors out and drops me to the command line, but I am then able to resize the second partition with busybox's fdisk and the resize2fs program in the initrd.

Except for the fact that I don't know how to run fdisk non-interactively, it should be possible to make a special initrd that would automatically resize the second partition if the initrd is booted.

[wdlkmpx]

Interesting. well everything can be done, however you are (not) alone in this..

It's possible to automate everything you propose. fdisk -> http://superuser.com/questions/332252/creating-and-formating-a-partition-using-a-bash-script

And creating a special and very simple initrd init script for arm makes sense too, we have to have a technical discussion about this, although only those who have rpi's have the last word. Meanwhile @KarlGodt might want to post his ideas and code for a full install inidtrd.gz init..

I fixed debdb2pupdb and updated the binaries. I just noticed 3builddistro copies support/debdb2pupdb and support/find_cat binaries to rootfs-complete... these 2 run in the host system.. it cross builds it may not be possible to update the dbs with the ppm, and scripts calling find_cat might fail..