build/builder.sh go generate ./... doesn't call glock, errors out

[RaduBerinde]

make protobuf hits this error:

cd ./cockroach/build && npm install --no-progress
npm WARN package.json @ No repository field.
npm ERR! git clone git@github.com:github:sintef-9012/Proto2TypeScript Cloning into bare repository '/root/.npm/_git-remotes/git-github-com-github-sintef-9012-Proto2TypeScript-0d69e939'...
npm ERR! git clone git@github.com:github:sintef-9012/Proto2TypeScript fatal: remote error: 
npm ERR! git clone git@github.com:github:sintef-9012/Proto2TypeScript    is not a valid repository name
npm ERR! git clone git@github.com:github:sintef-9012/Proto2TypeScript   Email support@github.com for help
npm ERR! notarget No compatible version found: proto2ts@'github:sintef-9012/Proto2TypeScript'
npm ERR! notarget Valid install targets:
npm ERR! notarget ["1.0.0","2.0.0"]
npm ERR! notarget 
npm ERR! notarget This is most likely not a problem with npm itself.
npm ERR! notarget In most cases you or one of your dependencies are requesting
npm ERR! notarget a package version that doesn't exist.

npm ERR! System Linux 3.16.0-4-amd64
npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install" "--no-progress"
npm ERR! cwd /go/src/github.com/cockroachdb/cockroach/build
npm ERR! node -v v0.10.29
npm ERR! npm -v 1.4.21
npm ERR! code ETARGET
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /go/src/github.com/cockroachdb/cockroach/build/npm-debug.log
npm ERR! not ok code 0
cockroach/build/protobuf.mk:91: recipe for target 'cockroach/build/npm.installed' failed
make[1]: *** [cockroach/build/npm.installed] Error 1
make[1]: Leaving directory '/go/src/github.com/cockroachdb'
Makefile:208: recipe for target 'protobuf' failed
make: *** [protobuf] Error 2

It works if I make this change to proto2ts:

-    "proto2ts": "github:sintef-9012/Proto2TypeScript",
+    "proto2ts": "sintef-9012/Proto2TypeScript",

This is on a Debian 8.1.

By the way, I had to install npm and nodejs-legacy (the latter wasn't easy to identify..). These should be added to the prerequisites in CONTRIBUTING.md

@tamird

[RaduBerinde]

My bad - CONTRIBUTING.md mentions this and has instructions for running go generate. I will try that (I was used to just running make protobuf).

[RaduBerinde]

Not much luck running build/builder.sh go generate ./... either:

make -C .. -f cockroach/build/protobuf.mk
make[1]: Entering directory '/go/src/github.com/cockroachdb'
make[1]: *** No rule to make target '/go/bin/protoc', needed by 'cockroach/acceptance/cluster/testconfig.pb.go'.  Stop.
make[1]: Leaving directory '/go/src/github.com/cockroachdb'
Makefile:208: recipe for target 'protobuf' failed
make: *** [protobuf] Error 2
main.go:19: running "make": exit status 2
cli/context.go:69: running "stringer": exec: "stringer": executable file not found in $PATH
roachpb/method.go:22: running "stringer": exec: "stringer": executable file not found in $PATH
security/securitytest/securitytest.go:27: running "go-bindata": exec: "go-bindata": executable file not found in $PATH
sql/session.go:99: running "stringer": exec: "stringer": executable file not found in $PATH
Type checking sql.y
Compiling sql.go
awk -f all_keywords.awk < sql.y > keywords.go
gofmt -s -w keywords.go
awk -f reserved_keywords.awk < sql.y > reserved_keywords.go
gofmt -s -w reserved_keywords.go
sql/parser/parse.go:52: running "stringer": exec: "stringer": executable file not found in $PATH
sql/pgwire/types.go:39: running "stringer": exec: "stringer": executable file not found in $PATH
sql/privilege/privilege.go:25: running "stringer": exec: "stringer": executable file not found in $PATH
sql/sqlbase/structured.go:51: running "stringer": exec: "stringer": executable file not found in $PATH
make -C /go/src/github.com/cockroachdb/cockroach/.. -f cockroach/build/protobuf.mk
make[1]: Entering directory '/go/src/github.com/cockroachdb'
make[1]: *** No rule to make target '/go/bin/protoc', needed by 'cockroach/acceptance/cluster/testconfig.pb.go'.  Stop.
make[1]: Leaving directory '/go/src/github.com/cockroachdb'
Makefile:52: recipe for target 'protobuf' failed
make: *** [protobuf] Error 2
ui/ui.go:21: running "make": exit status 2
util/encoding/encoding.go:836: running "stringer": exec: "stringer": executable file not found in $PATH

I do have protoc and stringer in my go bin dir:

# echo $GOPATH
/go
# ls -la /go/bin/protoc /go/bin/stringer
-rwxr-xr-x 1 root root 56337480 Jun 20 15:00 /go/bin/protoc
-rwxr-xr-x 1 root root  7328152 Jun 14 17:29 /go/bin/stringer

[tamird]

The first error you hit is because your node version is absolutely ancient. We should add to our docs that node v6.1+ or whatever is current is required.

The second error you posted looks like two separate issues:

1. you're missing protoc and stringer inside the build container. what does build/builder.sh echo $PATH print?
2. you say you have protoc and stringer in $GOPATH/bin, but is that in your $PATH? seems to me that $GOPATH is irrelevant here.

[RaduBerinde]

PATH is ok:

# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/root/bin:/root/bin:/go/bin
# build/builder.sh echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/root/bin:/root/bin:/go/bin

I am not sure what gets populated inside the container and what doesn't. Looks like the contents of /go/bin aren't there:

# build/builder.sh ls -la /go/bin
total 8
drwxr-xr-x 2 root root 4096 Jun 28 23:26 .
drwxrwxrwx 7 root root 4096 Jun 29 01:49 ..
# ls -la /go/bin
total 311264
drwxr-xr-x 3 root root     4096 Jun 28 19:26 .
drwxr-xr-x 5 root root     4096 Jan  5 11:51 ..
-rwxr-xr-x 1 root root  3056624 Apr 25 13:20 asmfmt
-rwxr-xr-x 1 root root  3215368 Feb 17 13:14 benchstat
...

[tamird]

From the repo root, try rm .bootstrap && build/builder.sh go generate ./...

[RaduBerinde]

Same as above. Looks like build/builder.sh is trying to map /go/bin/ to the contents of /go/bin/linux_amd64: --volume="${gopath0}/bin/linux_amd64:/go/bin"

I don't have linux_amd64 inside my /go/bin..

[tamird]

Shouldn't matter. For some reason go generate ./... isn't running glock sync inside the container for you, which is the reason that you're missing those binaries (inside the container).

[RaduBerinde]

As for the npm version, it looks like even the most recent Debian release has this version. I wonder why https://packages.debian.org/search?keywords=npm

[tamird]

https://github.com/cockroachdb/cockroach/blob/master/build/Dockerfile#L6

[RaduBerinde]

I was able to upgrade npm using itself haha (npm install npm -g)

[tamird]

Nice, but your node version is still ancient.

[RaduBerinde]

It was npm that was complaining, make protobuf works now locally. I will update node too just in case.

[maxlang]

I recommend using n to manage node versions.

https://github.com/tj/n

[RaduBerinde]

Going back to the build/builder.sh problem, through what mechanism is it supposed to call glock? The Dockerfile has ENV SKIP_BOOTSTRAP=1 so we won't process the .bootstrap target in the Makefile

# build/builder.sh env | grep SKIP_BOOTSTRAP
SKIP_BOOTSTRAP=1

[tamird]

Hm, indeed, that's interesting. In CI we actually run glock sync -n < GLOCKFILE - I'm not sure at what point I ran that locally (I have these binaries in the container).

Sounds like you can just go generate ./... normally once you have reasonable node/npm versions, though.

[RaduBerinde]

I see. I didn't realize the changes in the container persist from one run to the next. I ran build/builder.sh env SKIP_BOOTSTRAP=0 go generate ./... and it ran glock and started building. After this I can run it without the env part.

Now it runs into GitHub rate limit reached but that's another story :)

I think we should either fix builder.sh to mount the go/bin dir correctly, or update the CONTRIBUTING instructions to run glock once before go generate.

[tamird]

builder.sh does mount the go/bin dir correctly - see the comment at the bottom of the file.

I agree we should update CONTRIBUTING, I think it should say build/builder.sh SKIP_BOOTSTRAP=0 go generate ./..., though.

[RaduBerinde]

I see. Thanks, I understand the bin stuff now (maybe the linux_amd64 subdir should include the name docker or similar so it doesn't cause confusion on the host, though changing it might impact folks).

Sending out a PR to update CONTRIBUTING. We do need the env in the cmdline:

# build/builder.sh SKIP_BOOTSTRAP=0 go generate ./...
docker: Error response from daemon: Container command 'SKIP_BOOTSTRAP=0' not found or does not exist..
# build/builder.sh env SKIP_BOOTSTRAP=0 go generate ./...
make -C .. -f cockroach/build/protobuf.mk
...

[petermattis]

Using a different {bin,pkg}/linux_amd64 directory for build/builder.sh is exactly what I'm thinking of doing for #7477.