badger: tests failing

[L1Cafe]

I accidentally stumbled upon your project, I've spent a few hours looking into it, and I also saw Manish is quitting Dgraph Labs and moving to outcaste. Best of luck with that!

In regards to this project, I'm super intersted. I'm by no means an experienced software engineer, but I would learn to help and contribute where I can. However, I'm having problems running the tests. I tried skimming at the test.sh file but couldn't find anything obviously wrong, perhaps you can let me know what's going on?

[dev@arch-dev badger]$ ./test.sh 
go version go1.17.7 linux/amd64
~/Documents/github.com/badger/badger ~/Documents/github.com/badger
github.com/outcaste-io/ristretto/z
# github.com/outcaste-io/ristretto/z
/home/dev/go/pkg/mod/github.com/outcaste-io/ristretto@v0.2.0/z/calloc_jemalloc.go:73:9: could not determine kind of name for C.je_calloc
/home/dev/go/pkg/mod/github.com/outcaste-io/ristretto@v0.2.0/z/calloc_jemalloc.go:103:3: could not determine kind of name for C.je_free
/home/dev/go/pkg/mod/github.com/outcaste-io/ristretto@v0.2.0/z/calloc_jemalloc.go:143:2: could not determine kind of name for C.je_mallctl
/home/dev/go/pkg/mod/github.com/outcaste-io/ristretto@v0.2.0/z/calloc_jemalloc.go:170:2: could not determine kind of name for C.je_malloc_stats_print

Some system information:

[dev@arch-dev badger]$ cat /etc/*release
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
LOGO=archlinux-logo
[dev@arch-dev badger]$ go version
go version go1.17.7 linux/amd64
[dev@arch-dev badger]$ sudo pacman -Syu
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 multilib-testing is up to date
 multilib is up to date
:: Starting full system upgrade...
 there is nothing to do

[manishrjain]

Hey @danielmai , could you please have a look at this?

CC: @ahsanbarkati

[ahsanbarkati]

It is working fine for me. This error generally shows up if there is a blank line between import C and the C code that CGO needs to run. @L1Cafe did you change any of the ristretto code by any chance?

[danielmai]

The tests work for me as well running on a similar setup as @L1Cafe (go 1.17.7 and Arch Linux).

@L1Cafe did you change any of the ristretto code by any chance?

You can check this by running go mod verify. You should get the message all modules verified.

[manishrjain]

It seems like OP might not have jemalloc installed. Do tests check for that?

[manishrjain]

@danielmai -- could you double check on a system without jemalloc installed? And then conclude this issue. Thanks!

[danielmai]

The error message is different when jemalloc isn't installed:

[badger]$ go build -tags jemalloc
# github.com/outcaste-io/ristretto/z
/home/dev/go/pkg/mod/github.com/outcaste-io/ristretto@v0.2.0/z/calloc_jemalloc.go:12:10: fatal error: jemalloc/jemalloc.h: No such file or directory
   12 | #include <jemalloc/jemalloc.h>
      |          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.

The reported error message by @L1Cafe suggests the "blank line import C" error that @ahsanbarkati mentioned. It sounds like the error is in the specific workstation setup, and there's nothing to fix on our end.

I'll go ahead and close this issue. @L1Cafe Feel free to re-open if you're still having this same issue.

[L1Cafe]

Hi, thanks for your patience while I was looking at this.

No, I didn't modify anything.

[dev@arch-dev badger]$ git pull
Already up to date.
[dev@arch-dev badger]$ git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean
[dev@arch-dev badger]$ git reflog
a4486a4 (HEAD -> main, origin/main, origin/HEAD) HEAD@{0}: clone: from https://github.com/outcaste-io/badger
[dev@arch-dev badger]$ git rev-parse HEAD
a4486a4cf33ec972dcd48c85110c2365e23760c1
[dev@arch-dev badger]$ go mod verify
all modules verified

FWIW, jemalloc is installed:

[dev@arch-dev badger]$ pacman -Ss jemalloc
extra/jemalloc 1:5.2.1-6 [installed]
    General-purpose scalable concurrent malloc implementation

Sorry this is a bit difficult to reproduce. I'm not sure how to give you guys more information. Perhaps it would help if I were able to reproduce this on a Docker instance and then provide you with this Dockerfile? Would you want me to do that?

Otherwise, let me know what you need from me.

[danielmai]

A Dockerfile would help. Thanks, @L1Cafe.

[manishrjain]

I wonder if this is because you have jemalloc installed, but not with the right prefix of "je_".

[danielmai]

@manishrjain Ah, I think you're right. Ristretto's calloc does expect that prefix.

[danielmai]

@manishrjain Ah, I think you're right. Ristretto's calloc does expect that prefix.

[L1Cafe]

I'm not totally sure what Ristretto is, to be honest. Is there a way I can fix this?

[danielmai]

@L1Cafe I don’t think you can set --with-jemalloc-prefix=“je_” from pacman. You can download the jemalloc release tarball and then install jemallloc from there. Here’s a small script with the recommended configuration options we set for the jemalloc installation:

mkdir -p /tmp/jemalloc-temp && cd /tmp/jemalloc-temp
curl -s -L https://github.com/jemalloc/jemalloc/releases/download/5.2.1/jemalloc-5.2.1.tar.bz2 -o jemalloc.tar.bz2
tar xjf ./jemalloc.tar.bz2
cd jemalloc-5.2.1
./configure --with-jemalloc-prefix='je_' --with-malloc-conf='background_thread:true,metadata_thp:auto';
make
sudo make install

After having jemalloc installed you should be able to run the tests OK.