Failed ABI checks

[kmoore134]

I'm trying to do a build of 10.3-BETA1 package and keep getting failed ABI checks whenever we start a new build. (Which of course requires rebuilding pkgng, and all other packages)

root@bellicose:/synth/world # file bin/sh bin/sh: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 10.2 (1002509), stripped root@bellicose:/synth/world # ./bin/freebsd-version 10.3-BETA1

How is the ABI determined? Anything else I can run to figure out why this fails?

[jrmarino]

i promise you it won't. :)

[kmoore134]

lol ok, you are the expert, I'll let you find the way around it ;)

[jrmarino]

your log is full of errors. The make.conf suffix is missing and everything else has "discovery error"

[jrmarino]

basically the code probably worked, but something else is in play.

[kmoore134]

Anything I can look at? What exactly is a discovery error?

[kmoore134]

also, check the second log, no discovery error there

[jrmarino]

discovery error means the executed command failed (the one in the chroot)

[kmoore134]

Ahh wait, that one may be because it was building while I manually killed synth. My bad, take a look at the lang_gcc log instead

[kmoore134]

err, no, wait on that:

root@avenger:/jenkins # synth everything Builder mounts detected; attempting to remove them automatically ... Dismounting successful! Scanning entire ports tree. Scanning existing packages. 00:06:48 => [02] Builder launched 00:06:48 => [01] Builder launched 00:06:48 => [03] Builder launched 00:06:48 => [06] Builder launched 00:06:48 => [04] Builder launched 00:06:48 => [07] Builder launched 00:06:48 => [05] Builder launched 00:06:48 => [10] Builder launched 00:06:48 => [08] Builder launched 00:06:48 => [11] Builder launched 00:06:48 => [12] Builder launched 00:06:48 => [13] Builder launched 00:06:48 => [09] Builder launched 00:06:48 => [14] Builder launched 00:06:48 => [15] Builder launched 00:06:48 => [16] Builder launched 00:07:23 => [04] 00:00:00 Failure multimedia/ffmpeg 00:07:23 => [16] 00:00:00 Failure devel/libc++ 00:07:25 => [01] 00:00:00 Failure print/cups-client 00:07:27 => [08] 00:00:00 Failure lang/gnustep-base 00:07:41 => [07] 00:00:18 Success x11-toolkits/gnome-pty-helper 00:07:43 => [14] 00:00:22 Failure lang/sbcl 00:08:04 => [15] 00:00:52 Success textproc/iksemel 00:08:14 => [06] 00:01:00 Success net/glib-networking 00:08:24 => [05] 00:01:10 Success textproc/redland

[jrmarino]

HAVE_COMPAT_IA32_KERN=YES
CONFIGURE_MAX_CMD_LEN=262144
_SMP_CPUS=16
UID=0
ARCH=amd64
OPSYS=FreeBSD
OSVERSION=100000
_OSRELEASE=1.0-SYNTH
[ /etc/make.conf TAIL ]

Looks better except for OSVERSION and _OSRELEASE. (That's a hell of a custom makefile)

[kmoore134]

ffmpeg and a few others failed almost immediately, right at the beginning. (Those are the ones with discovery errors).

And yes, we have quite a custom make.conf, lots of things specific to desktop packages ;)

[jrmarino]

any chance mounts are failing? This is new to me.

[kmoore134]

maybe? Not seeing anything in dmesg or other obvious mount failures. It cleans up and moves onto the next builder before I can look ;)

This is with 16 concurrent builders defined, want me to drop it down?

[jrmarino]

the number shouldn't be a problem. You can if you want but I think the same thing will happen. I've run with 10 before without issue.

[jrmarino]

is there anything strange about what synth is mounting to? NFS or prior mounts?

At some point I can tell you what to modify to make it print out the actual error.

[kmoore134]

No NFS, just a local ZFS filesystem. It is cleaning up prior mounts first though, leme manually clean first and test again.

[jrmarino]

hold on a sec

[jrmarino]

can you try hash a8ed62b ? It should at least fix the OSRELEASE (I'm guessing) and maybe other stuff with luck.

[kmoore134]

ok, give me a few min, will report back

[kmoore134]

Ok, looks like it fixes the issue with OSRELEASE:

OPSYS=FreeBSD OSVERSION=1001000 _OSRELEASE=10.1-SYNTH

I saw the new status line go by on screen, nice!

Still seeing a couple of random builders fail when it starts up, with the discovery errors though. Only happens at the initial startup, maybe when its busy spinning up all the builders at once?

[jrmarino]

if that's the case, it's an OS issue. Maybe freebsd isn't concurrent safe? What filesystem is being used? maybe now is the time to halve the builders and see if anything changes.

[kmoore134]

Ok, I just kicked off another (with full 16 builders), and didn't get any of those failures this time... I'll keep an eye on it, see if I can figure out out reproducible it is. If I find something, I'll open a new bug report.

Marking as closed though, the original issue seems to be fixed.

[jrmarino]

ok, but it would be interesting to know if this is a ZFS system. I haven't seen anything like this on DragonFly HAMMER or DragonFly UFS

I suspect you'll see it again, randomly.

[kmoore134]

Yes, this is ZFS system. (100% ZFS on root, nothing else)

[jrmarino]

i have a fairly strong suspicion that the custom env definitions are responsible for the discovery errors.