search

bnprks / BPCells

Scaling Single Cell Analysis to Millions of Cells
https://bnprks.github.io/BPCells
Other
166 stars 17 forks source link

Issue when installing BPCells inside docker container #133

Closed oliviaellis65 closed 1 month ago

oliviaellis65 commented 1 month ago

Hi! I'm getting a weird error when I try to install BPCells inside of a docker container:

g++ -std=gnu++17 -I"/usr/local/lib/R/include" -DNDEBUG  -I'/usr/local/lib/R/site-library/Rcpp/include' -I'/usr/local/lib/R/site-library/RcppEigen/include' -I/usr/local/include   -I/usr/include/hdf5/serial -Ibpcells-cpp -I../tools/highway/include -Ivendor -std=c++17 -DRCPP_EIGEN -DEIGEN_PERMANENTLY_DISABLE_STUPID_WARNINGS -Wno-ignored-attributes -Wno-unknown-pragmas  -fPIC  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g   -c bpcells-cpp/simd/bp128/d1_maxbits.cpp -o bpcells-cpp/simd/bp128/d1_maxbits.o
In file included from bpcells-cpp/simd/bp128/d1_maxbits.cpp:21,
                 from ../tools/highway/include/hwy/foreach_target.h:174,
                 from bpcells-cpp/simd/bp128/d1_maxbits.cpp:19:
bpcells-cpp/simd/bp128/bp128-helper-inl.h: In function ‘void BPCells::simd::bp128::N_SVE::unpack(const uint32_t*, uint32_t*, F&&)’:
bpcells-cpp/simd/bp128/bp128-helper-inl.h:93:48: error: capture by copy of SVE type ‘Vec’ {aka ‘__SVUint32_t’}
   93 |     unroll32([&in, &out, &unpack_transform, d, mask, &InReg, &OutReg](auto i) {
      |                                                ^~~~
bpcells-cpp/simd/bp128/bp128-helper-inl.h: In lambda function:
bpcells-cpp/simd/bp128/bp128-helper-inl.h:109:30: error: ‘mask’ is not captured
  109 |         OutReg = And(OutReg, mask);
      |                              ^~~~
bpcells-cpp/simd/bp128/bp128-helper-inl.h:93:69: note: the lambda has no capture-default
   93 |     unroll32([&in, &out, &unpack_transform, d, mask, &InReg, &OutReg](auto i) {
      |                                                                     ^
bpcells-cpp/simd/bp128/bp128-helper-inl.h:91:9: note: ‘Vec mask’ declared here
   91 |     Vec mask = Set(d, B < 32 ? (1U << B) - 1 : 0xffffffff);
      |         ^~~~
bpcells-cpp/simd/bp128/bp128-helper-inl.h: In function ‘void BPCells::simd::bp128::N_SVE::pack(const uint32_t*, uint32_t*, F&&)’:
bpcells-cpp/simd/bp128/bp128-helper-inl.h:146:63: error: capture by copy of SVE type ‘Vec’ {aka ‘__SVUint32_t’}
  146 |     unroll32([&in, &out, &pack_transform, d, &InReg, &OutReg, mask](auto i) {
      |                                                               ^~~~
bpcells-cpp/simd/bp128/bp128-helper-inl.h: In lambda function:
bpcells-cpp/simd/bp128/bp128-helper-inl.h:154:32: error: ‘mask’ is not captured
  154 |             InReg = And(InReg, mask);
      |                                ^~~~
bpcells-cpp/simd/bp128/bp128-helper-inl.h:146:67: note: the lambda has no capture-default
  146 |     unroll32([&in, &out, &pack_transform, d, &InReg, &OutReg, mask](auto i) {
      |                                                                   ^
bpcells-cpp/simd/bp128/bp128-helper-inl.h:144:9: note: ‘Vec mask’ declared here
  144 |     Vec mask = Set(d, B < 32 ? (1U << B) - 1 : 0xffffffff);
      |         ^~~~
In file included from bpcells-cpp/simd/bp128/d1_maxbits.cpp:21,
                 from ../tools/highway/include/hwy/foreach_target.h:185,
                 from bpcells-cpp/simd/bp128/d1_maxbits.cpp:19:
bpcells-cpp/simd/bp128/bp128-helper-inl.h: In function ‘void BPCells::simd::bp128::N_SVE2::unpack(const uint32_t*, uint32_t*, F&&)’:
bpcells-cpp/simd/bp128/bp128-helper-inl.h:93:48: error: capture by copy of SVE type ‘Vec’ {aka ‘__SVUint32_t’}
   93 |     unroll32([&in, &out, &unpack_transform, d, mask, &InReg, &OutReg](auto i) {
      |                                                ^~~~
bpcells-cpp/simd/bp128/bp128-helper-inl.h: In lambda function:
bpcells-cpp/simd/bp128/bp128-helper-inl.h:109:30: error: ‘mask’ is not captured
  109 |         OutReg = And(OutReg, mask);
      |                              ^~~~
bpcells-cpp/simd/bp128/bp128-helper-inl.h:93:69: note: the lambda has no capture-default
   93 |     unroll32([&in, &out, &unpack_transform, d, mask, &InReg, &OutReg](auto i) {
      |                                                                     ^
bpcells-cpp/simd/bp128/bp128-helper-inl.h:91:9: note: ‘Vec mask’ declared here
   91 |     Vec mask = Set(d, B < 32 ? (1U << B) - 1 : 0xffffffff);
      |         ^~~~
bpcells-cpp/simd/bp128/bp128-helper-inl.h: In function ‘void BPCells::simd::bp128::N_SVE2::pack(const uint32_t*, uint32_t*, F&&)’:
bpcells-cpp/simd/bp128/bp128-helper-inl.h:146:63: error: capture by copy of SVE type ‘Vec’ {aka ‘__SVUint32_t’}
  146 |     unroll32([&in, &out, &pack_transform, d, &InReg, &OutReg, mask](auto i) {
      |                                                               ^~~~
bpcells-cpp/simd/bp128/bp128-helper-inl.h: In lambda function:
bpcells-cpp/simd/bp128/bp128-helper-inl.h:154:32: error: ‘mask’ is not captured
  154 |             InReg = And(InReg, mask);
      |                                ^~~~
bpcells-cpp/simd/bp128/bp128-helper-inl.h:146:67: note: the lambda has no capture-default
  146 |     unroll32([&in, &out, &pack_transform, d, &InReg, &OutReg, mask](auto i) {
      |                                                                   ^
bpcells-cpp/simd/bp128/bp128-helper-inl.h:144:9: note: ‘Vec mask’ declared here
  144 |     Vec mask = Set(d, B < 32 ? (1U << B) - 1 : 0xffffffff);
      |         ^~~~
In file included from bpcells-cpp/simd/bp128/d1_maxbits.cpp:21,
                 from ../tools/highway/include/hwy/foreach_target.h:196,
                 from bpcells-cpp/simd/bp128/d1_maxbits.cpp:19:
bpcells-cpp/simd/bp128/bp128-helper-inl.h: In function ‘void BPCells::simd::bp128::N_SVE_256::unpack(const uint32_t*, uint32_t*, F&&)’:
bpcells-cpp/simd/bp128/bp128-helper-inl.h:93:48: error: capture by copy of SVE type ‘Vec’ {aka ‘__SVUint32_t’}
   93 |     unroll32([&in, &out, &unpack_transform, d, mask, &InReg, &OutReg](auto i) {
      |                                                ^~~~
bpcells-cpp/simd/bp128/bp128-helper-inl.h: In lambda function:
bpcells-cpp/simd/bp128/bp128-helper-inl.h:109:30: error: ‘mask’ is not captured
  109 |         OutReg = And(OutReg, mask);
      |                              ^~~~
bpcells-cpp/simd/bp128/bp128-helper-inl.h:93:69: note: the lambda has no capture-default
   93 |     unroll32([&in, &out, &unpack_transform, d, mask, &InReg, &OutReg](auto i) {
      |                                                                     ^
bpcells-cpp/simd/bp128/bp128-helper-inl.h:91:9: note: ‘Vec mask’ declared here
   91 |     Vec mask = Set(d, B < 32 ? (1U << B) - 1 : 0xffffffff);
      |         ^~~~
bpcells-cpp/simd/bp128/bp128-helper-inl.h: In function ‘void BPCells::simd::bp128::N_SVE_256::pack(const uint32_t*, uint32_t*, F&&)’:
bpcells-cpp/simd/bp128/bp128-helper-inl.h:146:63: error: capture by copy of SVE type ‘Vec’ {aka ‘__SVUint32_t’}
  146 |     unroll32([&in, &out, &pack_transform, d, &InReg, &OutReg, mask](auto i) {
      |                                                               ^~~~
bpcells-cpp/simd/bp128/bp128-helper-inl.h: In lambda function:
bpcells-cpp/simd/bp128/bp128-helper-inl.h:154:32: error: ‘mask’ is not captured
  154 |             InReg = And(InReg, mask);
      |                                ^~~~
bpcells-cpp/simd/bp128/bp128-helper-inl.h:146:67: note: the lambda has no capture-default
  146 |     unroll32([&in, &out, &pack_transform, d, &InReg, &OutReg, mask](auto i) {
      |                                                                   ^
bpcells-cpp/simd/bp128/bp128-helper-inl.h:144:9: note: ‘Vec mask’ declared here
  144 |     Vec mask = Set(d, B < 32 ? (1U << B) - 1 : 0xffffffff);
      |         ^~~~
In file included from bpcells-cpp/simd/bp128/d1_maxbits.cpp:21,
                 from ../tools/highway/include/hwy/foreach_target.h:207,
                 from bpcells-cpp/simd/bp128/d1_maxbits.cpp:19:
bpcells-cpp/simd/bp128/bp128-helper-inl.h: In function ‘void BPCells::simd::bp128::N_SVE2_128::unpack(const uint32_t*, uint32_t*, F&&)’:
bpcells-cpp/simd/bp128/bp128-helper-inl.h:93:48: error: capture by copy of SVE type ‘Vec’ {aka ‘__SVUint32_t’}
   93 |     unroll32([&in, &out, &unpack_transform, d, mask, &InReg, &OutReg](auto i) {
      |                                                ^~~~
bpcells-cpp/simd/bp128/bp128-helper-inl.h: In lambda function:
bpcells-cpp/simd/bp128/bp128-helper-inl.h:109:30: error: ‘mask’ is not captured
  109 |         OutReg = And(OutReg, mask);
      |                              ^~~~
bpcells-cpp/simd/bp128/bp128-helper-inl.h:93:69: note: the lambda has no capture-default
   93 |     unroll32([&in, &out, &unpack_transform, d, mask, &InReg, &OutReg](auto i) {
      |                                                                     ^
bpcells-cpp/simd/bp128/bp128-helper-inl.h:91:9: note: ‘Vec mask’ declared here
   91 |     Vec mask = Set(d, B < 32 ? (1U << B) - 1 : 0xffffffff);
      |         ^~~~
bpcells-cpp/simd/bp128/bp128-helper-inl.h: In function ‘void BPCells::simd::bp128::N_SVE2_128::pack(const uint32_t*, uint32_t*, F&&)’:
bpcells-cpp/simd/bp128/bp128-helper-inl.h:146:63: error: capture by copy of SVE type ‘Vec’ {aka ‘__SVUint32_t’}
  146 |     unroll32([&in, &out, &pack_transform, d, &InReg, &OutReg, mask](auto i) {
      |                                                               ^~~~
bpcells-cpp/simd/bp128/bp128-helper-inl.h: In lambda function:
bpcells-cpp/simd/bp128/bp128-helper-inl.h:154:32: error: ‘mask’ is not captured
  154 |             InReg = And(InReg, mask);
      |                                ^~~~
bpcells-cpp/simd/bp128/bp128-helper-inl.h:146:67: note: the lambda has no capture-default
  146 |     unroll32([&in, &out, &pack_transform, d, &InReg, &OutReg, mask](auto i) {
      |                                                                   ^
bpcells-cpp/simd/bp128/bp128-helper-inl.h:144:9: note: ‘Vec mask’ declared here
  144 |     Vec mask = Set(d, B < 32 ? (1U << B) - 1 : 0xffffffff);
      |         ^~~~
make: *** [/usr/local/lib/R/etc/Makeconf:204: bpcells-cpp/simd/bp128/d1_maxbits.o] Error 1

My flags for building were:

Retrying with pkg-config flags...
Error running 'pkg-config hdf5 --cflags --libs'

Searching for config information with h5cc -showconfig and h5cc -show...
Found working hdf5
HDF5_CFLAGS=' -I/usr/include/hdf5/serial'
HDF5_LIBS=' -lcrypto -lcurl -lpthread -lsz -lz -ldl -lm -L/usr/lib/aarch64-linux-gnu/hdf5/serial -Wl,-rpath -Wl,/usr/lib/aarch64-linux-gnu/hdf5/serial -lhdf5'

I don't get the error when installing BPCells locally. My docker machine has 16GB RAM and 2 CPUs, and I am using g++ 11.4.0.

bnprks commented 1 month ago

Hi @oliviaellis65, thanks for your question. It looks like there was a build error on ARM computers that we hadn't previously noticed.

The good news is that there's an upcoming change in #131 that will fix this problem as a side effect of making some other performance improvements. That will probably be merged into the main branch within a week or two, but in the mean time you can install a version that will fix your problem with remotes::install_github("bnprks/BPCells/r", ref="bp/bitpacking-perf")

oliviaellis65 commented 1 month ago

Thank you so much! That fix worked for me. Unrelated question-- do you happen to know how much RAM is required to install BPCells with g++ 11.3? I'm downloading BPCells on a VM and want to use the least amount of RAM as possible, so if you happen to know this spec, that would be very helpful.

bnprks commented 1 month ago

Glad that worked for you! And again, it should be fixed in the main branch soon so keep an eye out for #131 to be merged so you can remove the workaround.

In a brief test I saw g++ 13.2.0 use up to about 1.8 GB of RAM while compiling (though you can also set MAKEFLAGS=--jobs=4 or something to 4x the memory usage but also speed up compilation by 4x). So I'd say 2 GB of RAM has a decent chance of working, though you could boost to e.g. 4 GB for safety