Offer EL9 builds

[fdr]

a.k.a. Enterprise Linux 9, the Red Hat/Rocky/AlmaLinux family of distributions.

https://packagecloud.io/citusdata/community only offers EL8 builds and below, it'd be nice if EL9 were offered as well; as-is EL8 and below binaries are provided.

On top of that, the GPG key used to sign https://packagecloud.io/citusdata/community uses a SHA-1 digest, which has been prohibited in EL9. It will probably become a problem in other distributions sooner or later. So you might have to talk to packagecloud about how to rotate into a new key without breaking your other package downloaders.

If you need an EL9 distribution on Azure, I suggest AlmaLinux: https://wiki.almalinux.org/cloud/Azure.html

[gurkanindibay]

While building citus from source in Centos 9 stream, I'm getting the error below

/usr/bin/gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection  -std=gnu99 -Wall -Wextra -Wno-unused-parameter -Wno-sign-compare -Wno-missing-field-initializers -Wno-clobbered -Wno-declaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wmissing-declarations -Wmissing-prototypes -Wshadow -Werror=vla -Werror=implicit-int -Werror=implicit-function-declaration -Werror=return-type -fstack-clash-protection -fstack-protector-strong -D_FORTIFY_SOURCE=2 -O2 -z noexecstack -fpic -shared -Wl,-z,relro -Wl,-z,now -Wformat -Wformat-security -Werror=format-security -I '//citus-rpm-build/citus-11.2.0.citus/src/include' -I'../../../src/include' -I/usr/pgsql-15/include -I//citus-rpm-build/citus-11.2.0.citus/vendor/safestringlib/include -I. -I./ -I/usr/pgsql-15/include/server -I/usr/pgsql-15/include/internal  -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o safeclib/ignore_handler_s.o safeclib/ignore_handler_s.c
lto1: fatal error: bytecode stream in file '/usr/pgsql-15/lib/libpgcommon.a' generated with LTO version 11.2 instead of the expected 11.3
compilation terminated.
lto-wrapper: fatal error: /usr/bin/gcc returned 1 exit status
compilation terminated.
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
make[1]: *** [/usr/pgsql-15/lib/pgxs/src/makefiles/pgxs.mk:475: pg_send_cancellation] Error 1
make[1]: Leaving directory '/citus-rpm-build/citus-11.2.0.citus/src/bin/pg_send_cancellation'
/usr/bin/gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection  -std=gnu99 -Wall -Wextra -Wno-unused-parameter -Wno-sign-compare -Wno-missing-field-initializers -Wno-clobbered -Wno-declaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wmissing-declarations -Wmissing-prototypes -Wshadow -Werror=vla -Werror=implicit-int -Werror=implicit-function-declaration -Werror=return-type -fstack-clash-protection -fstack-protector-strong -D_FORTIFY_SOURCE=2 -O2 -z noexecstack -fpic -shared -Wl,-z,relro -Wl,-z,now -Wformat -Wformat-security -Werror=format-security -I '//citus-rpm-build/citus-11.2.0.citus/src/include' -I'../../../src/include' -I/usr/pgsql-15/include -I//citus-rpm-build/citus-11.2.0.citus/vendor/safestringlib/include -I. -I./ -I/usr/pgsql-15/include/server -I/usr/pgsql-15/include/internal  -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include  -c -o safeclib/mem_primitives_lib.o safeclib/mem_primitives_lib.c
make: *** [Makefile:49: pg_send_cancellation] Error 2

When I execute as below by adding -fno-lto flag, build completed successfully

gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -std=gnu99 -Wall -Wextra -Wno-unused-parameter -Wno-sign-compare -Wno-missing-field-initializers -Wno-clobbered -Wno-declaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wmissing-declarations -Wmissing-prototypes -Wshadow -Werror=vla -Werror=implicit-int -Werror=implicit-function-declaration -Werror=return-type -fstack-clash-protection -DGIT_VERSION=\""main(sha: 3f66f3d9d)"\" /citus//src/bin/pg_send_cancellation/pg_send_cancellation.o -L/usr/pgsql-15/lib -lpgcommon -lpgport -L/usr/pgsql-15/lib -lpq -L/usr/pgsql-15/lib -Wl,--as-needed -L/usr/lib64  -L/usr/lib64 -Wl,--as-needed -Wl,-rpath,'/usr/pgsql-15/lib',--enable-new-dtags -L/usr/pgsql-15/lib -Wl,--as-needed -L/usr/lib64  -L/usr/lib64 -Wl,--as-needed -Wl,-rpath,'/usr/pgsql-15/lib',--enable-new-dtags -lzstd -llz4 -lcurl     -lpgcommon -lpgport -lselinux -lzstd -llz4 -lxslt -lxml2 -lssl -lcrypto -lgssapi_krb5 -lm **-fno-lto** -o pg_send_cancellation

[fdr]

I don't know how compatible CentOS Stream is as compared to EL9.X. I think it's its own thing, kinda?

[luss]

Generally speaking Centos 9 Stream is very, very compatible, but, it's a little ahead of EL9 for updates. My suggestion is to use Rocky Linux 9 as your baseline for building and maintaining an EL9 distro.

On Wed, Nov 2, 2022 at 6:25 PM Daniel Farina @.***> wrote:

I don't know how compatible CentOS Stream is as compared to EL9.X. I think it's its own thing, kinda?

— Reply to this email directly, view it on GitHub https://github.com/citusdata/citus/issues/6467#issuecomment-1301422842, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMWOHTUC3GDU62ZK6RWCVTWGLS5TANCNFSM6AAAAAART6YSSU . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[fdr]

Generally speaking Centos 9 Stream is very, very compatible, but, it's a little ahead of EL9 for updates. My suggestion is to use Rocky Linux 9 as your baseline for building and maintaining an EL9 distro. … On Wed, Nov 2, 2022 at 6:25 PM Daniel Farina @.> wrote: I don't know how compatible CentOS Stream is as compared to EL9.X. I think it's its own thing, kinda? — Reply to this email directly, view it on GitHub <#6467 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMWOHTUC3GDU62ZK6RWCVTWGLS5TANCNFSM6AAAAAART6YSSU . You are receiving this because you are subscribed to this thread.Message ID: @.>

Maybe this is water under the bridge by now, but when I was evaluating, AlmaLinux was much more on top of builds following RHEL releases, had their SecureBoot paperwork long since figured out before I needed to move from CentOS (Rocky did not, and would not for months afterwards), and provided images on all major Cloud platforms, with Azure being a particularly vexing omission for Rocky. Even on GCP, the Alma image was properly flagged into using gvnic, and Rocky (though maintained by Google) was not. In the end, I had to eliminate Rocky Linux, and it wasn't even a close comparison.

[gurkanindibay]

I'm a little confused right now. I haven't tried all Rpm based distros recently, but we have been using Centos based images for all rpm based distros. AFAIU, from your comments, we may need to change this strategy and create their own packages in their dedicated environment (docker images) for all distros such as Rocky Linux, AlmaLinux, EL9, etc. What do you think @fdr @luss ?

[luss]

You only need one EL9 build environment. Don't use CentOS 9 Streams, it won't be available long term and it's EL9 plus. I am familiar with using Rocky Linux and others are familiar with (and prefer) AlmaLinux. Both are well known and (I think) either are good choices for maintaining binary compatibility with RHEL 9 for the long term.

On Fri, Nov 11, 2022 at 2:55 AM Gürkan İndibay @.***> wrote:

I'm a little confused right now. I haven't tried all Rpm based distros recently, but we have been using Centos based images for all rpm based distros. AFAIU, from your comments, we may need to change this strategy and create their own packages in their dedicated environment (docker images) for all distros such as Rocky Linux, AlmaLinux, EL9, etc. What do you think @fdr https://github.com/fdr @luss https://github.com/luss ?

— Reply to this email directly, view it on GitHub https://github.com/citusdata/citus/issues/6467#issuecomment-1311353558, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMWOHWPTHXHYVR33UGYZSLWHX3WPANCNFSM6AAAAAART6YSSU . You are receiving this because you were mentioned.Message ID: @.***>

[gurkanindibay]

Thanks @luss , now using AlmaLinux as build environment. I will update the issue whenever I finish baking and testing packages.

[fdr]

I'm a little confused right now. I haven't tried all Rpm based distros recently, but we have been using Centos based images for all rpm based distros. AFAIU, from your comments, we may need to change this strategy and create their own packages in their dedicated environment (docker images) for all distros such as Rocky Linux, AlmaLinux, EL9, etc.

What do you think @fdr @luss ?

Well, the issue is that "CentOS" and "CentOS Stream" don't really have the same relation to Red Hat releases at all, and "CentOS" was killed by Red Hat. That's why there's no CentOS 9. I decided (though I considered the alternative) that I wanted a release-structured distribution rather than a rolling-release that is more or less strictly newer than Red Hat releases, so I had to evaluate alternatives to CentOS that shared the same general principle.

Rocky Linux was an early community favorite and heir apparent, but to my surprise it was not getting important project management out of the way (e.g. secure boot) even after quite a while.

There's also NavyLinux. I think that's all the "try to be fully compatible with Red Hat releases" projects I know of. I did not evaluate NavyLinux at the time.

Finally there's what I settled on, which was AlmaLinux. Having used it and occasionally asking questions in their chat area, I've been generally impressed with how well organized they seem to have been, even with (or maybe because of) very few personnel.

[gurkanindibay]

Have an open PR to fix the build problem. Waiting for the merge right now https://github.com/citusdata/citus/pull/6499

[Vonng]

Any update on this?

Since pgdg official repo does not provide el7.x86_64 citus rpm anymore. (citus_15*)

Citus official repo could be the only hope for unifying citus installation on el7, el8, el9. (citus111_15*)

[rajkumartreads]

Any update on this? We would like to evaluate Citus 12.1 on AlmaLinux 9.x.