search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.03k stars 3.79k forks source link

dev: `gen cgo` does not make `jemalloc.h` available #85362

Open erikgrinaker opened 2 years ago

erikgrinaker commented 2 years ago

I'm using gopls via VSCode on a gceworker (Linux). Often, it complains about not finding jemalloc.h, despite having run dev gen cgo go. If I run make buildshort, the problem goes away. Any idea what's going on here?

# github.com/cockroachdb/cockroach/pkg/server/status
pkg/server/status/runtime_jemalloc.go:21:11: fatal error: jemalloc/jemalloc.h: No such file or directory
 // #include <jemalloc/jemalloc.h>
           ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
# github.com/cockroachdb/cockroach/pkg/server/status
pkg/server/status/runtime_jemalloc.go:21:11: fatal error: jemalloc/jemalloc.h: No such file or directory
 // #include <jemalloc/jemalloc.h>
           ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
# github.com/cockroachdb/cockroach/pkg/server/status
pkg/server/status/runtime_jemalloc.go:21:11: fatal error: jemalloc/jemalloc.h: No such file or directory
 // #include <jemalloc/jemalloc.h>
           ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
# github.com/cockroachdb/cockroach/pkg/server/status
pkg/server/status/runtime_jemalloc.go:21:11: fatal error: jemalloc/jemalloc.h: No such file or directory
 // #include <jemalloc/jemalloc.h>
           ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
# github.com/cockroachdb/cockroach/pkg/server/status
pkg/server/status/runtime_jemalloc.go:21:11: fatal error: jemalloc/jemalloc.h: No such file or directory
 // #include <jemalloc/jemalloc.h>
           ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
# github.com/cockroachdb/cockroach/pkg/server/status
pkg/server/status/runtime_jemalloc.go:21:11: fatal error: jemalloc/jemalloc.h: No such file or directory
 // #include <jemalloc/jemalloc.h>
           ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
# github.com/cockroachdb/cockroach/pkg/server/status
pkg/server/status/runtime_jemalloc.go:21:11: fatal error: jemalloc/jemalloc.h: No such file or directory
 // #include <jemalloc/jemalloc.h>
           ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
: packages.Load error

Jira issue: CRDB-18204 Epic CRDB-17171

rickystewart commented 2 years ago

@erikgrinaker Can you cat pkg/server/status/zcgo_flags.go (after dev gen cgo)? There should be an -I flag in that file, can you verify that the directory exists and contains a file jemalloc/jemalloc.h?

cucaroach commented 2 years ago

I'll try to repro this and see if I can make any hay.

erikgrinaker commented 2 years ago

I can reproduce this by running make clean and then running a test via go test, both in VSCode and a regular shell:

$ go test -timeout 5m -run ^TestJemalloc$ github.com/cockroachdb/cockroach/pkg/server/status -count 1 -v -p 1
# github.com/cockroachdb/cockroach/pkg/server/status
pkg/server/status/runtime_jemalloc.go:21:11: fatal error: jemalloc/jemalloc.h: No such file or directory
 // #include <jemalloc/jemalloc.h>
           ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.

There should be an -I flag in that file, can you verify that the directory exists and contains a file jemalloc/jemalloc.h?

// #cgo CPPFLAGS: -I/home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libjemalloc_linux/include -I/home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libkrb5_linux/include
// #cgo LDFLAGS: -L/home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libjemalloc_linux/lib -L/home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libproj_linux/lib -L/home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libkrb5_linux/lib

The include directory does not exist:

$ ls -l /home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libjemalloc_linux/include
ls: cannot access '/home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libjemalloc_linux/include': No such file or directory

It looks like the hashed directory does not exist:

$ ls /home/erik/.cache/bazel/_bazel_erik/
cache  install

In fact, there is no jemalloc.h anywhere in ~/.cache/bazel:

$ find ~/.cache/bazel -name jemalloc.h
$

Now, after doing this, I pulled master again, ran dev doctor, and then did another dev gen cgo, and now it seems to be working:

// #cgo CPPFLAGS: -I/home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libjemalloc_linux/include -I/home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libkrb5_linux/include
// #cgo LDFLAGS: -L/home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libjemalloc_linux/lib -L/home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libproj_linux/lib -L/home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libkrb5_linux/lib
$ ls -l /home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libjemalloc_linux/include
total 4
drwxrwxr-x 2 erik erik 4096 Aug  1 19:01 jemalloc

$ go test -timeout 5m -run ^TestJemalloc$ github.com/cockroachdb/cockroach/pkg/server/status -count 1 -v -p 1
=== RUN   TestJemalloc
--- PASS: TestJemalloc (0.00s)
PASS
ok      github.com/cockroachdb/cockroach/pkg/server/status  0.053s

I don't know specifically how it gets into this broken state, but when it does, running dev gen cgo does not help. Not sure what made it work this time around. I'll poke around a bit the next time it happens, it tends to crop up every now and then.

cucaroach commented 2 years ago

I've seen this and looked at it but can't for the life of me figure it out, one "workaround" is to put "build --force_build_cdeps" in your .bazelrc and build and it will build it instead of downloading it from gs. What I haven't figured out is how the include gets dropped in the first place or how to repair it from the gs archive.

erikgrinaker commented 2 years ago

Thanks for looking into this @cucaroach. I wouldn't spend too much time on it, I can ping you again the next time it happens and we can poke at it live. I should be able to get it working again anyway.