search

matrix-org / rust-synapse-compress-state

A tool to compress some state in a Synapse instance's database
https://pypi.org/project/synapse-auto-compressor/
Apache License 2.0
142 stars 32 forks source link

synapse_auto_compressor fails build due to missing file #80

Closed PaarthShah closed 2 years ago

PaarthShah commented 2 years ago

Describe the bug cargo build for synapse_auto_compressor fails.

To Reproduce Steps to reproduce the behavior:

  1. git clone rust-synapse-compress-state
  2. cd synapse_auto_compressor
  3. cargo build
  4. See error: error: failed to run custom build command fortikv-jemalloc-sys v0.4.2+5.2.1-patched.2``

Surface-level cause: thread 'main' panicked at 'failed to execute command: No such file or directory (os error 2)', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tikv-jemalloc-sys-0.4.2+5.2.1-patched.2/build.rs:329:19

Logs Attached error.log

   Compiling tikv-jemalloc-sys v0.4.2+5.2.1-patched.2
error: failed to run custom build command for `tikv-jemalloc-sys v0.4.2+5.2.1-patched.2`

Caused by:
  process didn't exit successfully: `/root/rust-synapse-compress-state/target/debug/build/tikv-jemalloc-sys-180bbcfbef4d1b30/build-script-build` (exit status: 101)
  --- stdout
  TARGET=x86_64-unknown-linux-gnu
  HOST=x86_64-unknown-linux-gnu
  NUM_JOBS=8
  OUT_DIR="/root/rust-synapse-compress-state/target/debug/build/tikv-jemalloc-sys-6412eaaaa5ad7037/out"
  BUILD_DIR="/root/rust-synapse-compress-state/target/debug/build/tikv-jemalloc-sys-6412eaaaa5ad7037/out/build"
  SRC_DIR="/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tikv-jemalloc-sys-0.4.2+5.2.1-patched.2"
  cargo:rustc-cfg=prefixed
  OPT_LEVEL = Some("0")
  TARGET = Some("x86_64-unknown-linux-gnu")
  HOST = Some("x86_64-unknown-linux-gnu")
  CC_x86_64-unknown-linux-gnu = None
  CC_x86_64_unknown_linux_gnu = None
  HOST_CC = None
  CC = None
  CFLAGS_x86_64-unknown-linux-gnu = None
  CFLAGS_x86_64_unknown_linux_gnu = None
  HOST_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("true")
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
  CC="cc"
  CFLAGS="-O0 -ffunction-sections -fdata-sections -fPIC -g -fno-omit-frame-pointer -m64 -Wall"
  JEMALLOC_REPO_DIR="jemalloc"
  --with-jemalloc-prefix=_rjem_
  running: "sh" "/root/rust-synapse-compress-state/target/debug/build/tikv-jemalloc-sys-6412eaaaa5ad7037/out/build/configure" "--disable-cxx" "--with-jemalloc-prefix=_rjem_" "--with-private-namespace=_rjem_" "--host=x86_64-unknown-linux-gnu" "--build=x86_64-unknown-linux-gnu" "--prefix=/root/rust-synapse-compress-state/target/debug/build/tikv-jemalloc-sys-6412eaaaa5ad7037/out"
  checking for xsltproc... false
  checking for x86_64-unknown-linux-gnu-gcc... cc
  checking whether the C compiler works... yes
  checking for C compiler default output file name... a.out
  checking for suffix of executables... 
  checking whether we are cross compiling... no
  checking for suffix of object files... o
  checking whether we are using the GNU C compiler... yes
  checking whether cc accepts -g... yes
  checking for cc option to accept ISO C89... none needed
  checking whether compiler is cray... no
  checking whether compiler supports -std=gnu11... yes
  checking whether compiler supports -Wall... yes
  checking whether compiler supports -Wextra... yes
  checking whether compiler supports -Wshorten-64-to-32... no
  checking whether compiler supports -Wsign-compare... yes
  checking whether compiler supports -Wundef... yes
  checking whether compiler supports -Wno-format-zero-length... yes
  checking whether compiler supports -pipe... yes
  checking whether compiler supports -g3... yes
  checking how to run the C preprocessor... cc -E
  checking for grep that handles long lines and -e... /usr/bin/grep
  checking for egrep... /usr/bin/grep -E
  checking for ANSI C header files... yes
  checking for sys/types.h... yes
  checking for sys/stat.h... yes
  checking for stdlib.h... yes
  checking for string.h... yes
  checking for memory.h... yes
  checking for strings.h... yes
  checking for inttypes.h... yes
  checking for stdint.h... yes
  checking for unistd.h... yes
  checking whether byte ordering is bigendian... no
  checking size of void *... 8
  checking size of int... 4
  checking size of long... 8
  checking size of long long... 8
  checking size of intmax_t... 8
  checking build system type... x86_64-unknown-linux-gnu
  checking host system type... x86_64-unknown-linux-gnu
  checking whether pause instruction is compilable... yes
  checking number of significant virtual address bits... 48
  checking for x86_64-unknown-linux-gnu-ar... no
  checking for ar... ar
  checking for x86_64-unknown-linux-gnu-nm... no
  checking for nm... nm
  checking for gawk... gawk
  checking malloc.h usability... yes
  checking malloc.h presence... yes
  checking for malloc.h... yes
  checking whether malloc_usable_size definition can use const argument... no
  checking for library containing log... -lm
  checking whether __attribute__ syntax is compilable... yes
  checking whether compiler supports -fvisibility=hidden... yes
  checking whether compiler supports -fvisibility=hidden... no
  checking whether compiler supports -Werror... yes
  checking whether compiler supports -herror_on_warning... no
  checking whether tls_model attribute is compilable... yes
  checking whether compiler supports -Werror... yes
  checking whether compiler supports -herror_on_warning... no
  checking whether alloc_size attribute is compilable... yes
  checking whether compiler supports -Werror... yes
  checking whether compiler supports -herror_on_warning... no
  checking whether format(gnu_printf, ...) attribute is compilable... yes
  checking whether compiler supports -Werror... yes
  checking whether compiler supports -herror_on_warning... no
  checking whether format(printf, ...) attribute is compilable... yes
  checking whether compiler supports -Werror... yes
  checking whether compiler supports -herror_on_warning... no
  checking whether format(printf, ...) attribute is compilable... yes
  checking for a BSD-compatible install... /usr/bin/install -c
  checking for x86_64-unknown-linux-gnu-ranlib... no
  checking for ranlib... ranlib
  checking for ld... /usr/bin/ld
  checking for autoconf... false
  checking for memalign... yes
  checking for valloc... yes
  checking whether compiler supports -O3... yes
  checking whether compiler supports -O3... no
  checking whether compiler supports -funroll-loops... yes
  checking configured backtracing method... N/A
  checking for sbrk... yes
  checking whether utrace(2) is compilable... no
  checking whether a program using __builtin_unreachable is compilable... yes
  checking whether a program using __builtin_ffsl is compilable... yes
  checking whether a program using __builtin_popcountl is compilable... yes
  checking LG_PAGE... 12
  checking pthread.h usability... yes
  checking pthread.h presence... yes
  checking for pthread.h... yes
  checking for pthread_create in -lpthread... yes
  checking dlfcn.h usability... yes
  checking dlfcn.h presence... yes
  checking for dlfcn.h... yes
  checking for dlsym... no
  checking for dlsym in -ldl... yes
  checking whether pthread_atfork(3) is compilable... yes
  checking whether pthread_setname_np(3) is compilable... yes
  checking for library containing clock_gettime... none required
  checking whether clock_gettime(CLOCK_MONOTONIC_COARSE, ...) is compilable... yes
  checking whether clock_gettime(CLOCK_MONOTONIC, ...) is compilable... yes
  checking whether mach_absolute_time() is compilable... no
  checking whether compiler supports -Werror... yes
  checking whether syscall(2) is compilable... yes
  checking for secure_getenv... yes
  checking for sched_getcpu... yes
  checking for sched_setaffinity... yes
  checking for issetugid... no
  checking for _malloc_thread_cleanup... no
  checking for _pthread_mutex_init_calloc_cb... no
  checking for TLS... yes
  checking whether C11 atomics is compilable... yes
  checking whether GCC __atomic atomics is compilable... yes
  checking whether GCC 8-bit __atomic atomics is compilable... yes
  checking whether GCC __sync atomics is compilable... yes
  checking whether GCC 8-bit __sync atomics is compilable... yes
  checking whether Darwin OSAtomic*() is compilable... no
  checking whether madvise(2) is compilable... yes
  checking whether madvise(..., MADV_FREE) is compilable... yes
  checking whether madvise(..., MADV_DONTNEED) is compilable... yes
  checking whether madvise(..., MADV_DO[NT]DUMP) is compilable... yes
  checking whether madvise(..., MADV_[NO]HUGEPAGE) is compilable... yes
  checking for __builtin_clz... yes
  checking whether Darwin os_unfair_lock_*() is compilable... no
  checking whether glibc malloc hook is compilable... yes
  checking whether glibc memalign hook is compilable... yes
  checking whether pthreads adaptive mutexes is compilable... yes
  checking whether compiler supports -D_GNU_SOURCE... yes
  checking whether compiler supports -Werror... yes
  checking whether compiler supports -herror_on_warning... no
  checking whether strerror_r returns char with gnu source is compilable... yes
  checking for stdbool.h that conforms to C99... yes
  checking for _Bool... yes
  configure: creating ./config.status
  config.status: creating Makefile
  config.status: creating jemalloc.pc
  config.status: creating doc/html.xsl
  config.status: creating doc/manpages.xsl
  config.status: creating doc/jemalloc.xml
  config.status: creating include/jemalloc/jemalloc_macros.h
  config.status: creating include/jemalloc/jemalloc_protos.h
  config.status: creating include/jemalloc/jemalloc_typedefs.h
  config.status: creating include/jemalloc/internal/jemalloc_preamble.h
  config.status: creating test/test.sh
  config.status: creating test/include/test/jemalloc_test.h
  config.status: creating config.stamp
  config.status: creating bin/jemalloc-config
  config.status: creating bin/jemalloc.sh
  config.status: creating bin/jeprof
  config.status: creating include/jemalloc/jemalloc_defs.h
  config.status: creating include/jemalloc/internal/jemalloc_internal_defs.h
  config.status: creating test/include/test/jemalloc_test_defs.h
  config.status: executing include/jemalloc/internal/public_symbols.txt commands
  config.status: executing include/jemalloc/internal/private_symbols.awk commands
  config.status: executing include/jemalloc/internal/private_symbols_jet.awk commands
  config.status: executing include/jemalloc/internal/public_namespace.h commands
  config.status: executing include/jemalloc/internal/public_unnamespace.h commands
  config.status: executing include/jemalloc/jemalloc_protos_jet.h commands
  config.status: executing include/jemalloc/jemalloc_rename.h commands
  config.status: executing include/jemalloc/jemalloc_mangle.h commands
  config.status: executing include/jemalloc/jemalloc_mangle_jet.h commands
  config.status: executing include/jemalloc/jemalloc.h commands
  ===============================================================================
  jemalloc version   : 5.2.1-2-g172143a2979d9d948035423ce347e35cd1388fc3
  library revision   : 2

  CONFIG             : --disable-cxx --with-jemalloc-prefix=_rjem_ --with-private-namespace=_rjem_ --host=x86_64-unknown-linux-gnu --build=x86_64-unknown-linux-gnu --prefix=/root/rust-synapse-compress-state/target/debug/build/tikv-jemalloc-sys-6412eaaaa5ad7037/out build_alias=x86_64-unknown-linux-gnu host_alias=x86_64-unknown-linux-gnu CC=cc 'CFLAGS=-O0 -ffunction-sections -fdata-sections -fPIC -g -fno-omit-frame-pointer -m64 -Wall' 'LDFLAGS=-O0 -ffunction-sections -fdata-sections -fPIC -g -fno-omit-frame-pointer -m64 -Wall' 'CPPFLAGS=-O0 -ffunction-sections -fdata-sections -fPIC -g -fno-omit-frame-pointer -m64 -Wall'
  CC                 : cc
  CONFIGURE_CFLAGS   : -std=gnu11 -Wall -Wextra -Wsign-compare -Wundef -Wno-format-zero-length -pipe -g3 -fvisibility=hidden -O3 -funroll-loops
  SPECIFIED_CFLAGS   : -O0 -ffunction-sections -fdata-sections -fPIC -g -fno-omit-frame-pointer -m64 -Wall
  EXTRA_CFLAGS       : 
  CPPFLAGS           : -O0 -ffunction-sections -fdata-sections -fPIC -g -fno-omit-frame-pointer -m64 -Wall -D_GNU_SOURCE -D_REENTRANT
  CXX                : 
  CONFIGURE_CXXFLAGS : 
  SPECIFIED_CXXFLAGS : 
  EXTRA_CXXFLAGS     : 
  LDFLAGS            : -O0 -ffunction-sections -fdata-sections -fPIC -g -fno-omit-frame-pointer -m64 -Wall
  EXTRA_LDFLAGS      : 
  DSO_LDFLAGS        : -shared -Wl,-soname,$(@F)
  LIBS               : -lm  -pthread -ldl
  RPATH_EXTRA        : 

  XSLTPROC           : false
  XSLROOT            : 

  PREFIX             : /root/rust-synapse-compress-state/target/debug/build/tikv-jemalloc-sys-6412eaaaa5ad7037/out
  BINDIR             : /root/rust-synapse-compress-state/target/debug/build/tikv-jemalloc-sys-6412eaaaa5ad7037/out/bin
  DATADIR            : /root/rust-synapse-compress-state/target/debug/build/tikv-jemalloc-sys-6412eaaaa5ad7037/out/share
  INCLUDEDIR         : /root/rust-synapse-compress-state/target/debug/build/tikv-jemalloc-sys-6412eaaaa5ad7037/out/include
  LIBDIR             : /root/rust-synapse-compress-state/target/debug/build/tikv-jemalloc-sys-6412eaaaa5ad7037/out/lib
  MANDIR             : /root/rust-synapse-compress-state/target/debug/build/tikv-jemalloc-sys-6412eaaaa5ad7037/out/share/man

  srcroot            : 
  abs_srcroot        : /root/rust-synapse-compress-state/target/debug/build/tikv-jemalloc-sys-6412eaaaa5ad7037/out/build/
  objroot            : 
  abs_objroot        : /root/rust-synapse-compress-state/target/debug/build/tikv-jemalloc-sys-6412eaaaa5ad7037/out/build/

  JEMALLOC_PREFIX    : _rjem_
  JEMALLOC_PRIVATE_NAMESPACE
                     : _rjem_je_
  install_suffix     : 
  malloc_conf        : 
  documentation      : 1
  shared libs        : 1
  static libs        : 1
  autogen            : 0
  debug              : 0
  stats              : 1
  experimetal_smallocx : 0
  prof               : 0
  prof-libunwind     : 0
  prof-libgcc        : 0
  prof-gcc           : 0
  fill               : 1
  utrace             : 0
  xmalloc            : 0
  log                : 0
  lazy_lock          : 0
  cache-oblivious    : 1
  cxx                : 0
  ===============================================================================
  running: "make" "-j" "8"

  --- stderr
  thread 'main' panicked at 'failed to execute command: No such file or directory (os error 2)', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tikv-jemalloc-sys-0.4.2+5.2.1-patched.2/build.rs:329:19
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Desktop (please complete the following information):

reivilibre commented 2 years ago

Do you have make installed? On Ubuntu you generally want to sudo apt install build-essential to get things like make and a C compiler.

PaarthShah commented 2 years ago

Do you have make installed? On Ubuntu you generally want to sudo apt install build-essential to get things like make and a C compiler.

Yep, build-essentials, libssl-dev, pkg-config, and cargo are all at the latest.

reivilibre commented 2 years ago

I was going to ask you to run with RUST_BACKTRACE=1 to get a backtrace, but I notice you're one step ahead and included one in the log file — thanks :). It's genuinely failing when trying to run make — if you run make yourself do you see that the command actually exists?

As a workaround for now, I think you can use cargo build --no-default-features and then it won't compile or use jemalloc. The effect you might see (as far as I know) is slightly reduced memory allocation performance, but I don't think it would be too major. I would still like to know why this happens so we can sort it out, but it's quite mysterious.

PaarthShah commented 2 years ago

So, I don't know why this worked, but running

apt install --reinstall cargo

ended up fixing it, and it was able to build immediately with no other changes/reboots. Very strange, especially considering I fresh-installed cargo on this machine specifically to be able to build this :^) Closing this now, hopefully it might be relevant for someone else. Thank you for your help!