error: implicit declaration of function ‘wolfSSL_DH_get0_key’

Netatalk / netatalk

Netatalk is a Free and Open Source AFP fileserver. A *NIX or BSD system running Netatalk is capable of serving many Macintosh clients simultaneously as an AppleShare file server.

https://netatalk.io

GNU General Public License v2.0

332 stars 85 forks source link

error: implicit declaration of function ‘wolfSSL_DH_get0_key’ #1296

Closed knight-of-ni closed 1 month ago

knight-of-ni commented 2 months ago

Describe the bug Pardon my learning curve. I am in the process of converting Fedora/EPEL specfile from Autotools -> Meson, when I came across the implicit function declaration below.

Perhaps an #include wolfssl/ssl.h is needed?

Note that building with Autotools succeeds as well as building against openssl.

To Reproduce Mock build Netatalk on Fedora using meson. Here is how I am building it:

%meson \
        -Dwith-manual=true                                                     \
        -Dwith-docbook-path=%{_datadir}/sgml/docbook/xsl-stylesheets-%{xslver} \
        -Dwith-overwrite=true                                                  \
        -Dwith-pgp-uam=true                                                    \
        -Dwith-lockfile-path=%{_rundir}/lock/netatalk/netatalk                 \
        -Dwith-tcp-wrappers=false                                              \
        -Dwith-tests=true                                                      \
        -Dwith-dbus-sysconf-path=%{_sysconfdir}/dbus-1/system.d                \
        -Dwith-pkgconfdir-path=%{_sysconfdir}/netatalk                         \
        -Dwith-init-style=redhat-systemd                                       \
        -Dwith-embedded-ssl=true                                              \
        -Dwith-uams-path=%{_libdir}/netatalk

Expected behavior build succeeds

Environment

Server OS: Fedora 40
Netatalk Version 3.2.4

Logs

[208/304] gcc -Ietc/uams/uams_dhx_passwd.a.p -Ietc/uams -I../etc/uams -I. -I.. -Iinclude -I../include -Ietc/afpd -I../etc/afpd -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Wpedantic -std=c11 -DHAVE_CONFIG_H '-D_U_=__attribute__((unused))' -Wno-pedantic -Wno-extra -Wno-all -Wno-deprecated-declarations -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fPIC -MD -MQ etc/uams/uams_dhx_passwd.a.p/uams_dhx_passwd.c.o -MF etc/uams/uams_dhx_passwd.a.p/uams_dhx_passwd.c.o.d -o etc/uams/uams_dhx_passwd.a.p/uams_dhx_passwd.c.o -c ../etc/uams/uams_dhx_passwd.c
FAILED: etc/uams/uams_dhx_passwd.a.p/uams_dhx_passwd.c.o 
gcc -Ietc/uams/uams_dhx_passwd.a.p -Ietc/uams -I../etc/uams -I. -I.. -Iinclude -I../include -Ietc/afpd -I../etc/afpd -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Wpedantic -std=c11 -DHAVE_CONFIG_H '-D_U_=__attribute__((unused))' -Wno-pedantic -Wno-extra -Wno-all -Wno-deprecated-declarations -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fPIC -MD -MQ etc/uams/uams_dhx_passwd.a.p/uams_dhx_passwd.c.o -MF etc/uams/uams_dhx_passwd.a.p/uams_dhx_passwd.c.o.d -o etc/uams/uams_dhx_passwd.a.p/uams_dhx_passwd.c.o -c ../etc/uams/uams_dhx_passwd.c
In file included from ../etc/uams/uams_dhx_passwd.c:34:
../etc/uams/uams_dhx_passwd.c: In function ‘pwd_login’:
../include/wolfssl/openssl/dh.h:100:25: error: implicit declaration of function ‘wolfSSL_DH_get0_key’; did you mean ‘wolfSSL_DH_set0_pqg’? [-Wimplicit-function-declaration]
  100 | #define DH_get0_key     wolfSSL_DH_get0_key
      |                         ^~~~~~~~~~~~~~~~~~~
../etc/uams/uams_dhx_passwd.c:136:5: note: in expansion of macro ‘DH_get0_key’
  136 |     DH_get0_key(dh, &pub_key, NULL);
      |     ^~~~~~~~~~~
ninja: build stopped: subcommand failed.

Additional context none

rdmark commented 2 months ago

@knight-of-ni Thanks for working on the rpm packaging! So it looks like you're building without PAM support. Is this a deliberate choice for Fedora packaging?

Anyhow, at a glance I think you are correct. The wolfssl/openssl/ssl.h header, that we do include, actually includes wolfssl/ssl.h itself but it's behind a #ifndef OPENSSL_EXTRA_SSL_GUARD preprocessor macro, so perhaps this is where your compiler trips up?

FWIW on my own Fedora 40 test VM this problem does not reproduce, so something must be different from your setup...

I have an experimental PR that should hopefully fix this: https://github.com/Netatalk/netatalk/pull/1299

Would you be able to test this in your environment?

knight-of-ni commented 2 months ago

Thank you for the patch. I'll test and provide feedback. I may not return to working on Netatalk today, but tomorrow should be good.

To make the specfile easier to read, I am starting with only the parameters I need to change from their default. Since PAM support defaults to true, I did not include it. I am still verifying I didn't break or exclude something during the migration to meson, however.

I believe PAM support was indeed enabled. Header files found during the configure step:

Has header "pam/pam_appl.h" : NO 
Has header "security/pam_appl.h" : YES 
.
.
.
Library pam found: YES
Has header "security/pam_appl.h" : YES (cached)

PAM support shows for UAMs (not sure why it's in parenthesis):

    UAMs:
      Kerberos V               : YES
      PGP                      : YES
      Randnum                  : (afppasswd)
      clrtxt                   : (PAM SHADOW)
      guest                    : YES
      DHX                      : (PAM SHADOW)
      DHX2                     : (PAM SHADOW)

Found the PAM config dir:

    Paths:
      dbus daemon path         : /bin/dbus-daemon
      dbus system directory    : /etc/dbus-1/system.d
      init directory           : /usr/lib/systemd/system
      Netatalk lockfile        : /run/lock/netatalk
      PAM config directory     : /usr/etc/pam.d

Let me know if your test build shows something different.

rdmark commented 2 months ago

Ah, apologies. I momentarily forgot that the build system builds both the shadow and PAM modules when PAM is enabled (uams_dhx_passwd.c in your build log snippet.) So never mind!

PAM support shows for UAMs (not sure why it's in parenthesis):

This is just a convention that we ported from Autotools. We literally add parentheses for display.

uams_using_options = '(' + uams_options + ')'

With Meson we actually have more flexibility on how to format the summary, so if you have a suggestion how to restructure those lines to make them look better, please do tell!

knight-of-ni commented 1 month ago

Hmmm.... I applied #1299 as a patch and confirmed rpmbuild applied it successfully during build.

However, the build still fails in the same spot:

[204/302] gcc -Ietc/uams/uams_dhx_passwd.so.p -Ietc/uams -I../etc/uams -I. -I.. -Iinclude -I../include -Ietc/afpd -I../etc/afpd -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Wpedantic -std=c11 -DHAVE_CONFIG_H '-D_U_=__attribute__((unused))' -Wno-pedantic -Wno-extra -Wno-all -Wno-deprecated-declarations -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fPIC -MD -MQ etc/uams/uams_dhx_passwd.so.p/uams_dhx_passwd.c.o -MF etc/uams/uams_dhx_passwd.so.p/uams_dhx_passwd.c.o.d -o etc/uams/uams_dhx_passwd.so.p/uams_dhx_passwd.c.o -c ../etc/uams/uams_dhx_passwd.c
FAILED: etc/uams/uams_dhx_passwd.so.p/uams_dhx_passwd.c.o 
gcc -Ietc/uams/uams_dhx_passwd.so.p -Ietc/uams -I../etc/uams -I. -I.. -Iinclude -I../include -Ietc/afpd -I../etc/afpd -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Wpedantic -std=c11 -DHAVE_CONFIG_H '-D_U_=__attribute__((unused))' -Wno-pedantic -Wno-extra -Wno-all -Wno-deprecated-declarations -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fPIC -MD -MQ etc/uams/uams_dhx_passwd.so.p/uams_dhx_passwd.c.o -MF etc/uams/uams_dhx_passwd.so.p/uams_dhx_passwd.c.o.d -o etc/uams/uams_dhx_passwd.so.p/uams_dhx_passwd.c.o -c ../etc/uams/uams_dhx_passwd.c
In file included from ../etc/uams/uams_dhx_passwd.c:34:
../etc/uams/uams_dhx_passwd.c: In function ‘pwd_login’:
../include/wolfssl/openssl/dh.h:100:25: error: implicit declaration of function ‘wolfSSL_DH_get0_key’; did you mean ‘wolfSSL_DH_set0_pqg’? [-Wimplicit-function-declaration]
  100 | #define DH_get0_key     wolfSSL_DH_get0_key
      |                         ^~~~~~~~~~~~~~~~~~~
../etc/uams/uams_dhx_passwd.c:137:5: note: in expansion of macro ‘DH_get0_key’
  137 |     DH_get0_key(dh, &pub_key, NULL);
      |     ^~~~~~~~~~~
[205/302] gcc -Ietc/uams/uams_passwd.so.p -Ietc/uams -I../etc/uams -I. -I.. -Iinclude -I../include -Ietc/afpd -I../etc/afpd -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Wpedantic -std=c11 -DHAVE_CONFIG_H '-D_U_=__attribute__((unused))' -Wno-pedantic -Wno-extra -Wno-all -Wno-deprecated-declarations -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fPIC -MD -MQ etc/uams/uams_passwd.so.p/uams_passwd.c.o -MF etc/uams/uams_passwd.so.p/uams_passwd.c.o.d -o etc/uams/uams_passwd.so.p/uams_passwd.c.o -c ../etc/uams/uams_passwd.c
[206/302] gcc -Ietc/uams/uams_passwd.a.p -Ietc/uams -I../etc/uams -I. -I.. -Iinclude -I../include -Ietc/afpd -I../etc/afpd -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Wpedantic -std=c11 -DHAVE_CONFIG_H '-D_U_=__attribute__((unused))' -Wno-pedantic -Wno-extra -Wno-all -Wno-deprecated-declarations -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fPIC -MD -MQ etc/uams/uams_passwd.a.p/uams_passwd.c.o -MF etc/uams/uams_passwd.a.p/uams_passwd.c.o.d -o etc/uams/uams_passwd.a.p/uams_passwd.c.o -c ../etc/uams/uams_passwd.c
[207/302] gcc -Ietc/cnid_dbd/dbd.p -Ietc/cnid_dbd -I../etc/cnid_dbd -I/usr/include -I. -I.. -Iinclude -I../include -Ietc/afpd -I../etc/afpd -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Wpedantic -std=c11 -DHAVE_CONFIG_H '-D_U_=__attribute__((unused))' -Wno-pedantic -Wno-extra -Wno-all -Wno-deprecated-declarations -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer '-DVERSION="3.2.4"' -MD -MQ etc/cnid_dbd/dbd.p/dbif.c.o -MF etc/cnid_dbd/dbd.p/dbif.c.o.d -o etc/cnid_dbd/dbd.p/dbif.c.o -c ../etc/cnid_dbd/dbif.c
ninja: build stopped: subcommand failed.

If it helps, here is the prototype specfile I am using: https://github.com/knight-of-ni/specfiles/blob/master/netatalk.spec

rdmark commented 1 month ago

@knight-of-ni I have one more idea off the top of my head what could resolve this: import wolfssl/ssl.h directly in wolfssl/openssl/dh.h ... I pushed an update to the same PR, if you are able to give it a go.

If this also fails, then I might have to set up my own Fedora specfile environment to do some trial and error.

knight-of-ni commented 1 month ago

Shoot. No Dice:

[207/302] gcc -Ietc/uams/uams_dhx_passwd.a.p -Ietc/uams -I../etc/uams -I. -I.. -Iinclude -I../include -Ietc/afpd -I../etc/afpd -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Wpedantic -std=c11 -DHAVE_CONFIG_H '-D_U_=__attribute__((unused))' -Wno-pedantic -Wno-extra -Wno-all -Wno-deprecated-declarations -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fPIC -MD -MQ etc/uams/uams_dhx_passwd.a.p/uams_dhx_passwd.c.o -MF etc/uams/uams_dhx_passwd.a.p/uams_dhx_passwd.c.o.d -o etc/uams/uams_dhx_passwd.a.p/uams_dhx_passwd.c.o -c ../etc/uams/uams_dhx_passwd.c
FAILED: etc/uams/uams_dhx_passwd.a.p/uams_dhx_passwd.c.o 
gcc -Ietc/uams/uams_dhx_passwd.a.p -Ietc/uams -I../etc/uams -I. -I.. -Iinclude -I../include -Ietc/afpd -I../etc/afpd -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Wpedantic -std=c11 -DHAVE_CONFIG_H '-D_U_=__attribute__((unused))' -Wno-pedantic -Wno-extra -Wno-all -Wno-deprecated-declarations -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fPIC -MD -MQ etc/uams/uams_dhx_passwd.a.p/uams_dhx_passwd.c.o -MF etc/uams/uams_dhx_passwd.a.p/uams_dhx_passwd.c.o.d -o etc/uams/uams_dhx_passwd.a.p/uams_dhx_passwd.c.o -c ../etc/uams/uams_dhx_passwd.c
In file included from ../etc/uams/uams_dhx_passwd.c:34:
../etc/uams/uams_dhx_passwd.c: In function ‘pwd_login’:
../include/wolfssl/openssl/dh.h:101:25: error: implicit declaration of function ‘wolfSSL_DH_get0_key’; did you mean ‘wolfSSL_DH_set0_pqg’? [-Wimplicit-function-declaration]
  101 | #define DH_get0_key     wolfSSL_DH_get0_key
      |                         ^~~~~~~~~~~~~~~~~~~
../etc/uams/uams_dhx_passwd.c:136:5: note: in expansion of macro ‘DH_get0_key’
  136 |     DH_get0_key(dh, &pub_key, NULL);
      |     ^~~~~~~~~~~
[208/302] gcc -Ietc/cnid_dbd/dbd.p -Ietc/cnid_dbd -I../etc/cnid_dbd -I/usr/include -I. -I.. -Iinclude -I../include -Ietc/afpd -I../etc/afpd -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Wpedantic -std=c11 -DHAVE_CONFIG_H '-D_U_=__attribute__((unused))' -Wno-pedantic -Wno-extra -Wno-all -Wno-deprecated-declarations -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer '-DVERSION="3.2.4"' -MD -MQ etc/cnid_dbd/dbd.p/dbif.c.o -MF etc/cnid_dbd/dbd.p/dbif.c.o.d -o etc/cnid_dbd/dbd.p/dbif.c.o -c ../etc/cnid_dbd/dbif.c
ninja: build stopped: subcommand failed.
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /usr/bin/ninja -C /builddir/build/BUILD/netatalk-3.2.4/redhat-linux-build -j 4 -v
error: Bad exit status from /var/tmp/rpm-tmp.kkFWua (%build)

However, I do have a new data point.

This builds just fine on el9, with or without the patch... so this may have something to do with newer gcc 14.1.1

rdmark commented 1 month ago

That's a good lead.

However, my Fedora 40 test VM also runs gcc 14.1.1 and I cannot see the problem there...

I wonder if your build environment is setting some special compiler flags or other ENV variables?

What happens if you setup and build it directly from the source tree (e.g. meson setup build && meson compile -C build) without the rpm packaging harness?

knight-of-ni commented 1 month ago

Below are the contents on the meson rpm macros on Fedora. Looks like they mostly just set file paths. I previously tried forcing the number of jobs to 1, and that did not help.

I've been searching for similar bug reports on wolfssl github site and did not find any reported against wolfSSL_DH_get0_key ....another clue this issue has got something to do specifically with the Fedore environment in which I am attempting to build it. I will continue to troubleshoot.

meson rpm macro

$ rpm --eval %meson

  CFLAGS="${CFLAGS:--O2 -g}" ; export CFLAGS ; 
  CXXFLAGS="${CXXFLAGS:--O2 -g}" ; export CXXFLAGS ; 
  FFLAGS="${FFLAGS:--O2 -g }" ; export FFLAGS ; 
  FCFLAGS="${FCFLAGS:--O2 -g }" ; export FCFLAGS ; 
  LDFLAGS="${LDFLAGS:-}" ; export LDFLAGS 
    /usr/bin/meson setup --buildtype=plain --prefix=/usr --libdir=/usr/lib64 --libexecdir=/usr/libexec --bindir=/usr/bin --sbindir=/usr/sbin --includedir=/usr/include --datadir=/usr/share --mandir=/usr/share/man --infodir=/usr/share/info --localedir=/usr/share/locale --sysconfdir=/etc --localstatedir=/var --sharedstatedir=/var/lib --wrap-mode=nodownload --auto-features=enabled %{_vpath_srcdir} %{_vpath_builddir}

meson_build rpm macro

$ rpm --eval %meson_build

    /usr/bin/meson compile -C %{_vpath_builddir} -j 3 --verbose

knight-of-ni commented 1 month ago

What happens if you setup and build it directly from the source tree (e.g. meson setup build && meson compile -C build) without the rpm packaging harness?

This succeeds.

Since that worked, I copied the meson commands mock was using, verbatim, and executed those from the command line:

/usr/bin/meson setup --buildtype=plain --prefix=/usr --libdir=/usr/lib64 --libexecdir=/usr/libexec --bindir=/usr/bin --sbindir=/usr/sbin --includedir=/usr/include --datadir=/usr/share --mandir=/usr/share/man --infodir=/usr/share/info --localedir=/usr/share/locale --sysconfdir=/etc --localstatedir=/var --sharedstatedir=/var/lib --wrap-mode=nodownload --auto-features=enabled . redhat-linux-build --localstatedir=/var/lib -Dwith-manual=true -Dwith-docbook-path=/usr/share/sgml/docbook/xsl-stylesheets-3.2.4 -Dwith-overwrite=true -Dwith-pgp-uam=true -Dwith-lockfile-path=/run/lock/netatalk -Dwith-tcp-wrappers=false -Dwith-tests=true -Dwith-dbus-sysconf-path=/etc/dbus-1/system.d -Dwith-pkgconfdir-path=/etc/netatalk -Dwith-init-style=redhat-systemd -Dwith-init-hooks=false -Dwith-uams-path=/usr/lib64/netatalk

Then

/usr/bin/meson compile -C redhat-linux-build -j 3 --verbose

And THAT succeeded. My mind is blown.

While I have seen builds fail in mock that run cleanly when called from commandline, usually that is caused by the build trying to do something that only a privileged account can do or it is trying to pull something down from the Internets... neither appears to be the case here.

knight-of-ni commented 1 month ago

Whoa hang on. Think I figured out the root cause.

I noticed mariadb-connector-c-devel was pulling in openssl-devel. After removing mariadb-connector-c-devel as a build requirement, the mock build just succeeded.

I thought Netatalk needed a mysql compatible devel package, but now I'm not sure. I see this with or without mariadb-connector-c-devel included:

Run-time dependency mysqlclient found: NO (tried pkgconfig and cmake)

Will dig into this next.

UPDATE: So whether I build from source or try to mock build it, in both cases the build cannot find embedded wolfssl:

with-embedded-ssl is default (true) with-ssl-override is true

Run-time dependency libcrypto found: NO (tried pkgconfig, system and cmake)
Run-time dependency libtls found: YES 3.8.1
Run-time dependency wolfssl found: NO (tried pkgconfig and cmake)

If I exclude mriadb/mysql devel packages (which pull in openssl-devel), the build succeeds, but the relevant crypto features are disabled.

If I pull in openssl-devel and set with-embedded-ssl to false (and do not set with-ssl-override), openssl is detected and the build succeeds (just w/o DHCAST128 support I presume).

If I pull in mariadb devel package, the build cannot find mysql.h. It looks like in order to support mariadb's mysq.h, we would need to do something more complicated during the meson configuration process like this: https://github.com/chhuang-one/sqxclib/blob/master/meson.build#L49

Note that I also tried to set with-mysql-config, but that did not help. It is unclear to me what setting this flag does.

And since maria devel package pulls in openssl-devel, the build fails with the implicit declaration error (unless I disable wolfssl support by setting with-embedded-ssl to false).

If I pull in mysql devel package instead of mariadb, the build does find mysql.h (yea). However, this package also pulls in openssl-devel, which cause the same problem as the mariadb devel package.

So in summary, we are looking at two problems

I have been unsuccessful getting the build to detect the bundled wolfssl library under any circumstances.

The second is the inability to detect mariadb devel headers instead of mysql.
I'm fine with leaving this one be for now, and just pulling in mysql-devel package.

Second UPDATE:

It looks like the Fedora CI workflow does not find bundled woflssl either:

2024-07-27T13:14:45.8256464Z Run-time dependency libcrypto found: YES 3.2.1
2024-07-27T13:14:45.8256862Z Did not find CMake 'cmake'
2024-07-27T13:14:45.8257144Z Found CMake: NO
2024-07-27T13:14:45.8257558Z Run-time dependency libtls found: NO (tried pkgconfig and cmake)
2024-07-27T13:14:45.8258162Z Run-time dependency wolfssl found: NO (tried pkgconfig and cmake)

Phew. My brain hurts. Going to give it a rest for the night.

rdmark commented 1 month ago

Holy moly, this is quite some detective work! Digging into these dependency trees (webs?) make my head spin too.

In 3.2, the openssl devel headers are actually required for DHX/DHCAST128 even with embedded wolfssl, because wolfssl lacks the actual Diffie–Hellman algorithm.

In the next stable release, we are replacing the openssl dependency with libnettle once and for all, which is a less opinionated library. :)

I think the confusion here is caused by the fact that the bundled WolfSSL library is built when with-embedded-ssl is enabled, even when the openssl devel headers are missing...

Your comment about with-mysql-config made me dig into this a bit, and realized it's a stub in Meson and not used. Raised https://github.com/Netatalk/netatalk/issues/1330 to sort this out.

rdmark commented 1 month ago

FYI it turned out to be quite easy to add support for mariadb. This will be in the next release: https://github.com/Netatalk/netatalk/pull/1341

rdmark commented 1 month ago

It looks like the Fedora CI workflow does not find bundled woflssl either:

What you show here isn't actually a check for the bundled WolfSSL, but for the system WolfSSL dependency. This is quite confusing, but we do support either one or the other. However, during the last release cycle the only OS that supplied a WolfSSL library with all the symbols that we need is Debian. So the WolfSSL run-time dependency check will fail everywhere else.

Hence on Fedora, the expected configuration in 3.2.x should be bundled WolfSSL that depends on the system OpenSSL 3.x library for a handful of symbols.

knight-of-ni commented 1 month ago

Yeah, you are right. I realized that after I made that last comment.

I still get an implicit declaration error when I try to build against bundled wolfssl. Full build output is here if you are interested: https://kojipkgs.fedoraproject.org//work/tasks/1934/121241934/build.log

For now I am going to build against crpyto and libretls libraries, rather than wolfssl. That seems to build just fine, presumable w/o DHCAST128 support. I'm not sure how many users would be affected by this anyways.

rdmark commented 1 month ago

Darn... I really want to get to the bottom of this.

Yeah it will build fine with libcrypto v3.x (with deprecation warnings) but the DHX/DHCAST128 library will bail out at runtime with a cryptic error message (pun not intended.) You may want to consider patching out DHX from being loaded by default to avoid user confusion when they attempt to log in from Classic Mac OS. This is what I did previously for the Debian deb.

May I see the configuration file where you define which dependency packages that will be used for the build process?

One thing that stood out looking at your logs, is Run-time dependency libtls found: YES 3.8.1 whereas I'm not using this library when building locally or in the GitHub CI job. Alas installing libretls-devel did not lead to a reproduction of the issue...

rdmark commented 1 month ago

I'm getting libcrypto 3.2.1 while you have 3.2.2, which I doubt has any influence...

knight-of-ni commented 1 month ago

You can see what packages I pull into the build from my latest specfile prototype here: https://github.com/knight-of-ni/specfiles/blob/master/netatalk.spec

Note that openssl-devel gets pulled in, indirectly, via mariadb-connector-c-devel.

If you want to see a complete list of all packages that were present during the build, you can see that here: https://kojipkgs.fedoraproject.org/work/tasks/1934/121241934/root.log

I added libretls-devel as a buildrequires when I noticed this. At the time, this was an attempt to try to avoid pulling openssl into the build.

The example build I linked earlier was based off fedora rawhide. Some dependent packages will be slightly newer than fedora 40.

Note also I am using the latest branch-netatalk-3-2 (3.2.5dev).

I also noticed that the main Netatalk branch does not appear to make use of crypto or libtls. Indeed, looking at the root meson.build, it looks like the future of Netatalk will be to only support wolfSSL. Is that true? If that is the case, then perhaps I should start a new package request to get wolfSSL added to Fedora repos.

rdmark commented 1 month ago

I added libretls-devel as a buildrequires when I noticed this. At the time, this was an attempt to try to avoid pulling openssl into the build.

This logic is meant for systems that completely lack OpenSSL and only have LibreSSL. Since Fedora does provide OpenSSL libraries, I suggest you remove libretls-devel and explicitly add openssl-devel to dependencies, for clarity.

I also noticed that the main Netatalk branch does not appear to make use of crypto or libtls. Indeed, looking at the root meson.build, it looks like the future of Netatalk will be to only support wolfSSL. Is that true? If that is the case, then perhaps I should start a new package request to get wolfSSL added to Fedora repos.

Kind of! We have shifted to using Nettle as the crypto library with WolfSSL (bundled or system) as the provider. So in the short term (4.0) nothing should break for Fedora. Just swap out OpenSSL for Nettle, and keep using bundled WolfSSL. But in the long term, when all of the major platforms start shipping a WolfSSL library then we can remove the bundled WolfSSL code. So yes, it would be great if you could put in a package request!

Edit: It should be noted that WolfSSL is a highly modular library. Every platform that ships WolfSSL provides a different set of headers and symbols, which is a bit of a pain. Debian is the gold standard here, shipping exactly the capabilities that we require, if you need a reference point.

knight-of-ni commented 1 month ago

Ok, so I finally have some good news. I managed to build a wolfssl library that Netatalk will detect and build against.

The build succeeded for all Fedoras and EPEL 9, across all supported architectures: https://copr.fedorainfracloud.org/coprs/kni/wolfssl/build/7810737/

I checked the build output, and Netatalk did in fact detect the system installed wolfssl library and build against that:

Run-time dependency wolfssl found: YES 5.7.0
.
.
.
    Options:
      ACL support              : YES
      AFP stats via dbus       : YES
      Cracklib support         : YES
      dtrace probes            : YES
      GSSAPI support           : YES
      Kerberos support         : YES
      LDAP support             : YES
      Quota support            : YES
      Quota provider           : libtirpc
      SSL provider             : WolfSSL

Yeah, I can already see the number of wolfssl build options are going to be a pain. When I submit it for package approval, I'm going to have to find a general set of options that plays well with most other packages that might need it. Good idea "borrowing" from the Debian source package. I was thinking the same thing. :-)

rdmark commented 1 month ago

Wohoo! Great job. This is encouraging progress.

So let me go ahead and finalize the 3.2.5 release today since it looks like we don’t have any critical blocker in netatalk code for Fedora at this point.

knight-of-ni commented 1 month ago

Certainly. Don't hold this up for Fedora.

Note, I will build Netatalk 3.2.5 against openssl, for the time being, but I'll only build it for Fedora rawhide and 40. It might take a while to get wolfSSL added as a new package. The process can drag on sometimes.

Let's close this one out....

rdmark commented 1 month ago

Sounds reasonable to me. We can figure out the details as we go!