search

apache / arrow-nanoarrow

Helpers for Arrow C Data & Arrow C Stream interfaces
https://arrow.apache.org/nanoarrow
Apache License 2.0
149 stars 34 forks source link

0.5.0 fails to build where `aligned_alloc` is not supplied by the OS #493

Closed barracuda156 closed 1 month ago

barracuda156 commented 1 month ago 
--->  Building R-nanoarrow
xinstall: mkdir /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_R_R-nanoarrow/R-nanoarrow/work/build
Executing:  cd "/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_R_R-nanoarrow/R-nanoarrow/work/nanoarrow" && /opt/local/bin/R CMD INSTALL . --library=/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_R_R-nanoarrow/R-nanoarrow/work/build --install-tests 
* installing *source* package ‘nanoarrow’ ...
** package ‘nanoarrow’ successfully unpacked and MD5 sums checked
** using staged installation
** libs
using C compiler: ‘gcc-mp-13 (MacPorts gcc13 13.2.0_4+stdlib_flag) 13.2.0’
using C++ compiler: ‘g++-mp-13 (MacPorts gcc13 13.2.0_4+stdlib_flag) 13.2.0’
Warning in system2("xcrun", "--show-sdk-path", TRUE, TRUE) :
  running command ''xcrun' --show-sdk-path 2>&1' had status 64
using SDK: ‘NA’‘NA’‘NA’‘NA’‘NA’‘NA’
/opt/local/bin/gcc-mp-13 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I../src  -isystem/opt/local/include/LegacySupport -I/opt/local/include    -fPIC  -pipe -Os -arch ppc  -c altrep.c -o altrep.o
/opt/local/bin/gcc-mp-13 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I../src  -isystem/opt/local/include/LegacySupport -I/opt/local/include    -fPIC  -pipe -Os -arch ppc  -c array.c -o array.o
/opt/local/bin/gcc-mp-13 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I../src  -isystem/opt/local/include/LegacySupport -I/opt/local/include    -fPIC  -pipe -Os -arch ppc  -c array_stream.c -o array_stream.o
/opt/local/bin/gcc-mp-13 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I../src  -isystem/opt/local/include/LegacySupport -I/opt/local/include    -fPIC  -pipe -Os -arch ppc  -c array_view.c -o array_view.o
/opt/local/bin/gcc-mp-13 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I../src  -isystem/opt/local/include/LegacySupport -I/opt/local/include    -fPIC  -pipe -Os -arch ppc  -c as_array.c -o as_array.o
/opt/local/bin/gcc-mp-13 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I../src  -isystem/opt/local/include/LegacySupport -I/opt/local/include    -fPIC  -pipe -Os -arch ppc  -c buffer.c -o buffer.o
/opt/local/bin/gcc-mp-13 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I../src  -isystem/opt/local/include/LegacySupport -I/opt/local/include    -fPIC  -pipe -Os -arch ppc  -c convert.c -o convert.o
/opt/local/bin/gcc-mp-13 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I../src  -isystem/opt/local/include/LegacySupport -I/opt/local/include    -fPIC  -pipe -Os -arch ppc  -c convert_array.c -o convert_array.o
/opt/local/bin/gcc-mp-13 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I../src  -isystem/opt/local/include/LegacySupport -I/opt/local/include    -fPIC  -pipe -Os -arch ppc  -c convert_array_stream.c -o convert_array_stream.o
/opt/local/bin/gcc-mp-13 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I../src  -isystem/opt/local/include/LegacySupport -I/opt/local/include    -fPIC  -pipe -Os -arch ppc  -c flatcc.c -o flatcc.o
In file included from flatcc/flatcc_flatbuffers.h:29,
                 from flatcc/flatcc_builder.h:68,
                 from flatcc.c:19:
flatcc.c: In function 'flatcc_builder_finalize_aligned_buffer':
../src/flatcc/flatcc_alloc.h:116:36: warning: implicit declaration of function 'aligned_alloc' [-Wimplicit-function-declaration]
  116 | #define FLATCC_ALIGNED_ALLOC(a, n) aligned_alloc(a, n)
      |                                    ^~~~~~~~~~~~~
flatcc/flatcc_builder.h:203:44: note: in expansion of macro 'FLATCC_ALIGNED_ALLOC'
  203 | #define FLATCC_BUILDER_ALIGNED_ALLOC(a, n) FLATCC_ALIGNED_ALLOC(a, n)
      |                                            ^~~~~~~~~~~~~~~~~~~~
flatcc.c:1995:14: note: in expansion of macro 'FLATCC_BUILDER_ALIGNED_ALLOC'
 1995 |     buffer = FLATCC_BUILDER_ALIGNED_ALLOC(align, size);
      |              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
flatcc.c:21:1: note: include '<stdlib.h>' or provide a declaration of 'aligned_alloc'
   20 | #include "flatcc/flatcc_emitter.h"
  +++ |+#include <stdlib.h>
   21 | 
../src/flatcc/flatcc_alloc.h:116:36: warning: incompatible implicit declaration of built-in function 'aligned_alloc' [-Wbuiltin-declaration-mismatch]
  116 | #define FLATCC_ALIGNED_ALLOC(a, n) aligned_alloc(a, n)
      |                                    ^~~~~~~~~~~~~
flatcc/flatcc_builder.h:203:44: note: in expansion of macro 'FLATCC_ALIGNED_ALLOC'
  203 | #define FLATCC_BUILDER_ALIGNED_ALLOC(a, n) FLATCC_ALIGNED_ALLOC(a, n)
      |                                            ^~~~~~~~~~~~~~~~~~~~
flatcc.c:1995:14: note: in expansion of macro 'FLATCC_BUILDER_ALIGNED_ALLOC'
 1995 |     buffer = FLATCC_BUILDER_ALIGNED_ALLOC(align, size);
      |              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/flatcc/flatcc_alloc.h:116:36: note: include '<stdlib.h>' or provide a declaration of 'aligned_alloc'
  116 | #define FLATCC_ALIGNED_ALLOC(a, n) aligned_alloc(a, n)
      |                                    ^~~~~~~~~~~~~
flatcc/flatcc_builder.h:203:44: note: in expansion of macro 'FLATCC_ALIGNED_ALLOC'
  203 | #define FLATCC_BUILDER_ALIGNED_ALLOC(a, n) FLATCC_ALIGNED_ALLOC(a, n)
      |                                            ^~~~~~~~~~~~~~~~~~~~
flatcc.c:1995:14: note: in expansion of macro 'FLATCC_BUILDER_ALIGNED_ALLOC'
 1995 |     buffer = FLATCC_BUILDER_ALIGNED_ALLOC(align, size);
      |              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
flatcc.c: In function 'flatcc_builder_aligned_alloc':
../src/flatcc/flatcc_alloc.h:116:36: warning: incompatible implicit declaration of built-in function 'aligned_alloc' [-Wbuiltin-declaration-mismatch]
  116 | #define FLATCC_ALIGNED_ALLOC(a, n) aligned_alloc(a, n)
      |                                    ^~~~~~~~~~~~~
flatcc/flatcc_builder.h:203:44: note: in expansion of macro 'FLATCC_ALIGNED_ALLOC'
  203 | #define FLATCC_BUILDER_ALIGNED_ALLOC(a, n) FLATCC_ALIGNED_ALLOC(a, n)
      |                                            ^~~~~~~~~~~~~~~~~~~~
flatcc.c:2014:12: note: in expansion of macro 'FLATCC_BUILDER_ALIGNED_ALLOC'
 2014 |     return FLATCC_BUILDER_ALIGNED_ALLOC(alignment, size);
      |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/flatcc/flatcc_alloc.h:116:36: note: include '<stdlib.h>' or provide a declaration of 'aligned_alloc'
  116 | #define FLATCC_ALIGNED_ALLOC(a, n) aligned_alloc(a, n)
      |                                    ^~~~~~~~~~~~~
flatcc/flatcc_builder.h:203:44: note: in expansion of macro 'FLATCC_ALIGNED_ALLOC'
  203 | #define FLATCC_BUILDER_ALIGNED_ALLOC(a, n) FLATCC_ALIGNED_ALLOC(a, n)
      |                                            ^~~~~~~~~~~~~~~~~~~~
flatcc.c:2014:12: note: in expansion of macro 'FLATCC_BUILDER_ALIGNED_ALLOC'
 2014 |     return FLATCC_BUILDER_ALIGNED_ALLOC(alignment, size);
      |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/local/bin/gcc-mp-13 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I../src  -isystem/opt/local/include/LegacySupport -I/opt/local/include    -fPIC  -pipe -Os -arch ppc  -c infer_ptype.c -o infer_ptype.o
/opt/local/bin/gcc-mp-13 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I../src  -isystem/opt/local/include/LegacySupport -I/opt/local/include    -fPIC  -pipe -Os -arch ppc  -c init.c -o init.o
/opt/local/bin/gcc-mp-13 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I../src  -isystem/opt/local/include/LegacySupport -I/opt/local/include    -fPIC  -pipe -Os -arch ppc  -c ipc.c -o ipc.o
/opt/local/bin/gcc-mp-13 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I../src  -isystem/opt/local/include/LegacySupport -I/opt/local/include    -fPIC  -pipe -Os -arch ppc  -c materialize.c -o materialize.o
/opt/local/bin/gcc-mp-13 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I../src  -isystem/opt/local/include/LegacySupport -I/opt/local/include    -fPIC  -pipe -Os -arch ppc  -c nanoarrow.c -o nanoarrow.o
/opt/local/bin/g++-mp-13 -std=gnu++17 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I../src  -isystem/opt/local/include/LegacySupport -I/opt/local/include    -fPIC  -pipe -Os -arch ppc   -c nanoarrow_cpp.cc -o nanoarrow_cpp.o
/opt/local/bin/gcc-mp-13 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I../src  -isystem/opt/local/include/LegacySupport -I/opt/local/include    -fPIC  -pipe -Os -arch ppc  -c nanoarrow_ipc.c -o nanoarrow_ipc.o
/opt/local/bin/gcc-mp-13 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I../src  -isystem/opt/local/include/LegacySupport -I/opt/local/include    -fPIC  -pipe -Os -arch ppc  -c pointers.c -o pointers.o
/opt/local/bin/gcc-mp-13 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I../src  -isystem/opt/local/include/LegacySupport -I/opt/local/include    -fPIC  -pipe -Os -arch ppc  -c schema.c -o schema.o
/opt/local/bin/gcc-mp-13 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I../src  -isystem/opt/local/include/LegacySupport -I/opt/local/include    -fPIC  -pipe -Os -arch ppc  -c util.c -o util.o
/opt/local/bin/gcc-mp-13 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I../src  -isystem/opt/local/include/LegacySupport -I/opt/local/include    -fPIC  -pipe -Os -arch ppc  -c vctr.c -o vctr.o
/opt/local/bin/gcc-mp-13 -I"/opt/local/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I../src  -isystem/opt/local/include/LegacySupport -I/opt/local/include    -fPIC  -pipe -Os -arch ppc  -c version.c -o version.o
/opt/local/bin/g++-mp-13 -std=gnu++17 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/opt/local/Library/Frameworks/R.framework/Resources/lib -Wl,-headerpad_max_install_names -L/opt/local/lib -lMacportsLegacySupport -Wl,-rpath,/opt/local/lib/libgcc -arch ppc -o nanoarrow.so altrep.o array.o array_stream.o array_view.o as_array.o buffer.o convert.o convert_array.o convert_array_stream.o flatcc.o infer_ptype.o init.o ipc.o materialize.o nanoarrow.o nanoarrow_cpp.o nanoarrow_ipc.o pointers.o schema.o util.o vctr.o version.o -F/opt/local/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
installing to /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_R_R-nanoarrow/R-nanoarrow/work/build/00LOCK-nanoarrow/00new/nanoarrow/libs
** R
** inst
** tests
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded from temporary location
Error: package or namespace load failed for ‘nanoarrow’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_R_R-nanoarrow/R-nanoarrow/work/build/00LOCK-nanoarrow/00new/nanoarrow/libs/nanoarrow.so':
  dlopen(/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_R_R-nanoarrow/R-nanoarrow/work/build/00LOCK-nanoarrow/00new/nanoarrow/libs/nanoarrow.so, 6): Symbol not found: _aligned_alloc
  Referenced from: /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_R_R-nanoarrow/R-nanoarrow/work/build/00LOCK-nanoarrow/00new/nanoarrow/libs/nanoarrow.so
  Expected in: dynamic lookup

Error: loading failed
Execution halted
ERROR: loading failed

What gcc suggests won't work here, though a missing header is still wrong, of course, since aligned_alloc is not provided by the compiler but relies on the OS, and it is not supported in many macOS versions (and likely other systems).

We need a fallback implementation with a better cross-platform support here.

barracuda156 commented 1 month ago

@paleolimbot Could you please help with this?

Usual fallbacks for this are malloc and posix_memalign.

barracuda156 commented 1 month ago

Wait, looks like flatcc_alloc.h already has everything needed, but nanoarrow does not use needed macros to pick the right implementation.

barracuda156 commented 1 month ago

So, passing -DFLATCC_USE_GENERIC_ALIGNED_ALLOC via Makevars fixes the bug.

I.e., what is needed is a configure script with a check for aligned_alloc support; if it is not present, then add the flag to Makevars. Problem solved.

paleolimbot commented 1 month ago

I think aligned_alloc() is C11 also, so perhaps compiling with explicit C99 will work?

barracuda156 commented 1 month ago

I think aligned_alloc() is C11 also, so perhaps compiling with explicit C99 will work?

@paleolimbot Assuming nanoarrow does not need C11, that might work, but AFAIK R bakes in flags used to build R itself, I have no idea if we could manually cancel them out for a single package and whether this is desirable at all.

paleolimbot commented 1 month ago

Assuming nanoarrow does not need C11

We do test with explicit c99; however, that's just a compiler flag and I can't see where it would disable this based on the C11 or C99-ness.

So, passing -DFLATCC_USE_GENERIC_ALIGNED_ALLOC via Makevars fixes the bug.

Great! I may actually just add this to Makevars because generating protobuf is not a bottleneck for the types of things that we are doing with it 🙂 and because I don't think R even has the ability to force a C standard (in the same way that it cares about the C++ standard).