All new packages on Illumos broken

[lschweiss-wustl]

On or about February 6th, 2022. All packages that get updated from pkgsrc will not run. They fail with a libc error:

/opt/local/bin# ./openssl
ld.so.1: openssl: fatal: libc.so.1: version 'ILLUMOS_0.37' not found (required by file openssl)
ld.so.1: openssl: fatal: libc.so.1: open failed: No such file or directory
Killed

/opt/local/bin# ./mosh
ld.so.1: perl: fatal: libc.so.1: version 'ILLUMOS_0.39' not found (required by file /opt/local/lib/perl5/5.34.0/x86_64-solaris-thread-multi-64/CORE/libperl.so)
ld.so.1: perl: fatal: libc.so.1: open failed: No such file or directory
Killed

This is happening on older (20200124) with packages that get updated and new installations (20220208) with most packages.

[lschweiss-wustl]

I forgot to mention this is on OmniOS CE r151030 and r151038.

[lschweiss-wustl]

This is a serious problem. Pkgsrc is now unusable on OmniOS.

It appears to be a build problem for the library binaries starting around the time of the 2022Q1 release.

[jfqd]

For SmartOS the solution was to use a newer platform image. I am too far away from OmniOS, but is this an issue too with latest CE version?

[lschweiss-wustl]

I will test the latest CE, however, it is a problem on both active LTS versions.

Upgrading to the latest CE is not an option for production systems.

[lschweiss-wustl]

I just upgraded an r151038 to r151040 and the problem goes away. Looks like something is compiling against too new of a library for anything older.

If this cannot be resolved quickly, are there some docs on building this for myself so I can make a working repository? I have many broken systems right now.

[jfqd]

@jperkin mentioned it here: https://smartos.topicbox.com/groups/smartos-discuss/Tb9044eab41e370b0/heads-up-trunk-migrating-to-20210826-gcc-10

You may use 2020Q4 instead of trunk with older OmiOS systems.

[lschweiss-wustl]

I am using 2020Q4 on most of my systems. This problem started on those systems when some packages were updated last week.

Installing a new OmniOS with 038 and 2020Q4 is now broken.

[jfqd]

Oh shit. I can feel your pain. For us it was only a reboot with a newer SmartOS platform image.

[jperkin]

Hey, I'm really sorry this broke your install. This is completely my fault, I was under the mistaken impression that OmniOS systems (and other illumos distributions) were generally a lot more up-to-date than SmartOS, and so didn't factor that into my decision and subsequent announcements to bump our minimum platform requirement.

As for 2020Q4 installs, nothing has changed there. Only trunk and 2021Q4 have been bumped up to requiring joyent_20210826T002459Z, all other releases are still locked to joyent_20161222T003450Z. If you are really seeing some issues with 2020Q4 packages then please send me the details and I'll look urgently. I have a number of checks to ensure that each branch is built on the correct platform, so I would be very surprised if this is the case.

[lschweiss-wustl]

I've realized this morning I'm partially at fault here. I thought I had 2020Q4 on most of my systems. In fact, I had 20201019 on them due to some problems I had at the time of building my deployment stack.

Am I safe to wipe out /opt/local and reinstall 2020Q4 from scratch? I'm not worried about any of my configurations, Puppet will restore them.

[jperkin]

Yep, clear out /opt/local and /var/db/pkgin and then you should be able to unpack a fresh 2020Q4 bootstrap and go from there.