`make buildoss` build failure on 22.1.5

davepacheco commented 2 years ago

Describe the problem

I'm trying to build an OSS build of v22.1.5. I'll post fuller details below. In my environment, I first saw this failure:

docgen http \
--gendoc ./bin/protoc-gen-doc \
--out docs/generated/http \
--protoc-flags "-Ipkg -I./vendor/github.com/gogo/protobuf -I./vendor/go.etcd.io -I./vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis -I./vendor/github.com/cockroachdb/errors -I./vendor/github.com/prometheus/client_model ./pkg/server/serverpb/status.proto ./pkg/server/serverpb/admin.proto ./pkg/server/status/statuspb/status.proto"
touch bin/.docgen_http
docgen logformats docs/generated/logformats.md
touch bin/.docgen_logformats
Hash: db7417ee41f9729a2871
Version: webpack 4.46.0
Time: 81155ms
Built at: 08/17/2022 10:24:23 PM
 1 asset
Entrypoint vendor = vendor.oss.dll.js
  [47] (webpack)/buildin/global.js 472 bytes {0} [built]
 [207] (webpack)/buildin/module.js 497 bytes {0} [built]
[1068] dll vendor 12 bytes {0} [built]
[1069] ../cluster-ui/dist/js/main.js 7.69 MiB {0} [built]
[1078] ./src/js/protos.js 11.2 MiB {0} [built]
[1232] /home/dap/garbage-compactor/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/moment/locale sync ^\.\/.*$ 3.21 KiB {0} [built]
    + 1972 hidden modules

ERROR in dll vendor
Module not found: Error: Can't resolve '@cockroachlabs/crdb-protobuf-client-ccl' in '/home/dap/garbage-compactor/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/ui/workspaces/db-console'
 @ dll vendor vendor[2]
gmake: *** [Makefile:1380: pkg/ui/workspaces/db-console/dist/vendor.oss.dll.js] Error 2
gmake: Leaving directory '/home/dap/garbage-compactor/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach'

It seemed a little suspicious to me that it's trying to pull in a file with ccl in it for an OSS build but I haven't tracked down what's going on here. My environment is a little weird (I'm building on illumos with a few local patches) so I figured I'd try with the stock builder and I got a different problem:

Killed
error Command failed with exit code 137.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
ERROR: "build:bundle" exited with 137.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
make: *** [Makefile:363: pkg/ui/cluster-ui/dist/main.js] Error 1

To Reproduce

My system:

$ uname -a
Darwin zathras.local 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64 x86_64
$ go version
go version go1.19 darwin/amd64

I know this is newer than Cockroach's Go, but I'm using the Docker builder so I didn't think that mattered.

My path and clone setup:

$ go env GOPATH
/Users/dap/go
$ pwd
/Users/dap/go/src/github.com/cockroachdb/cockroach
$ git status
HEAD detached at v22.1.5
nothing to commit, working tree clean

Start the builder and run the build. The output is too long to put in this description so I've attached it.

Expected behavior

I expected the "buildoss" build to complete successfully and produces binaries.

Environment:

Cockroach v22.1.5 (from Git)
MacOS 11.6 (Big Sur)
Docker Desktop 2.2.0.3

Maybe related: #83528, #69101, #75137?

Jira issue: CRDB-18742

blathers-crl[bot] commented 2 years ago

Hello, I am Blathers. I am here to help you get the issue triaged.

Hoot - a bug! Though bugs are the bane of my existence, rest assured the wretched thing will get the best of care here.

I have CC'd a few people who may be able to assist you:

@dhartunian (author of cockroachdb/cockroach#83528, found keywords: prometheus)
@JeffSwenson (author of cockroachdb/cockroach#69101)
@koorosh (author of cockroachdb/cockroach#75137)

If we have not gotten back to your issue within a few business days, you can try the following:

Join our community slack channel and ask on #cockroachdb.
Try find someone from here if you know they worked closely on the area and CC them.

_{:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

davepacheco commented 2 years ago

Also: I realize the "make" build is deprecated for 22.1.5, but I thought this would still work because "deprecated" is usually a step before "unsupported". I'm afraid the Bazel transition is going to be a lot of work for us.

rickystewart commented 2 years ago

Indeed, make is still supported for now.

I'm a little perplexed about this bit:

Module not found: Error: Can't resolve '@cockroachlabs/crdb-protobuf-client-ccl' in '/home/dap/garbage-compactor/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach/pkg/ui/workspaces/db-console'
 @ dll vendor vendor[2]
gmake: *** [Makefile:1380: pkg/ui/workspaces/db-console/dist/vendor.oss.dll.js] Error 2
gmake: Leaving directory '/home/dap/garbage-compactor/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach'

So it seems like the make build thinks your workspace is at /home/dap/garbage-compactor/cockroach/cache/gopath/src/github.com/cockroachdb/cockroach, but according to your report your GOPATH is /Users/dap/go and pwd is /Users/dap/go/src/github.com/cockroachdb/cockroach. Can you clarify?

davepacheco commented 2 years ago

Sorry for the confusion. There are two different problems here from two different systems.

On my illumos system, GOPATH is /home/dap/garbage-compactor/cockroach/cache/gopath and I get the error about crdb-protobuf-client-ccl. This is the problem I actually care about. But this is a doctored environment (being a different platform, with a few local patches, etc.). So I figured I should start from a more supported configuration and see if I get the same result. This led to problem 2.
To rule out my environment, I switched to a Mac using the Docker builder. On that system, GOPATH is /Users/dap/go and I'm building in $GOPATH/src/github.com/cockroachdb/cockroach. In this environment (which I assume is expected to work), I got the error about "build:bundle" exited with 137.

I wasn't sure if y'all would want to start directly from the first problem or from the known-to-be-working configuration, so I included both.

rickystewart commented 2 years ago

For (2), please provide the full unabridged build log. I don't think there's enough in what you've provided to diagnose the issue.

For (1), I'll take a closer look. I agree it seems weird the oss build is trying to pull in something ccl but am not 100% sure it's a bug.

davepacheco commented 2 years ago

For (2), please provide the full unabridged build log. I don't think there's enough in what you've provided to diagnose the issue.

I believe that's what's attached to this issue. If there's something else I can provide, let me know!

rickystewart commented 2 years ago

So for (2) I wonder if your container OOM'ed. I assume you're using Docker desktop on macOS. Can you double-check how much memory you have allotted to containers? 8GB should be sufficient.

davepacheco commented 2 years ago

Good call! I think it was set to 2 GiB before. I set it to 8 GiB and got a new failure:

build/node-run.sh -C pkg/ui/workspaces/cluster-ui yarn build
yarn run v1.22.19
$ npm-run-all -p build:typescript build:bundle
$ tsc
$ NODE_ENV=production webpack --display-error-details --mode=production
/go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack-cli/bin/cli.js:93
                throw err;
                ^

Error: Cannot find module './DependenciesBlock'
Require stack:
- /go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack/lib/Module.js
- /go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack/lib/NormalModule.js
- /go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack/lib/NormalModuleFactory.js
- /go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack/lib/Compiler.js
- /go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack/lib/webpack.js
- /go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack-cli/bin/utils/validate-options.js
- /go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack-cli/bin/utils/convert-argv.js
- /go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack-cli/bin/cli.js
- /go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack/bin/webpack.js
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
    at Function.Module._load (node:internal/modules/cjs/loader:778:27)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (/go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/v8-compile-cache/v8-compile-cache.js:159:20)
    at Object.<anonymous> (/go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack/lib/Module.js:9:27)
    at Module._compile (/go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/v8-compile-cache/v8-compile-cache.js:192:30)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (/go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/v8-compile-cache/v8-compile-cache.js:159:20)
    at Object.<anonymous> (/go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack/lib/NormalModule.js:19:16)
    at Module._compile (/go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/v8-compile-cache/v8-compile-cache.js:192:30)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (/go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/v8-compile-cache/v8-compile-cache.js:159:20)
    at Object.<anonymous> (/go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack/lib/NormalModuleFactory.js:17:22)
    at Module._compile (/go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/v8-compile-cache/v8-compile-cache.js:192:30)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (/go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/v8-compile-cache/v8-compile-cache.js:159:20)
    at Object.<anonymous> (/go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack/lib/Compiler.js:23:29)
    at Module._compile (/go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/v8-compile-cache/v8-compile-cache.js:192:30)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack/lib/Module.js',
    '/go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack/lib/NormalModule.js',
    '/go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack/lib/NormalModuleFactory.js',
    '/go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack/lib/Compiler.js',
    '/go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack/lib/webpack.js',
    '/go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack-cli/bin/utils/validate-options.js',
    '/go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack-cli/bin/utils/convert-argv.js',
    '/go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack-cli/bin/cli.js',
    '/go/src/github.com/cockroachdb/cockroach/pkg/ui/node_modules/webpack/bin/webpack.js'
  ]
}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
ERROR: "build:bundle" exited with 1.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
make: *** [Makefile:363: pkg/ui/cluster-ui/dist/main.js] Error 1
make: *** Waiting for unfinished jobs....

Here is the full log this time.

sjbarag commented 2 years ago

Heyyyy @davepacheco — sorry about that! I'm not entirely sure how that slipped by us, but I'm able to confirm the original error you reported (Module not found: Error: Can't resolve '@cockroachlabs/crdb-protobuf-client-ccl' in (some path)) on my normal Linux dev machine. Looks like it snuck past me during the 22.1 release while I was getting the Bazel build working!

https://github.com/cockroachdb/cockroach/pull/86425 should take care of things for you. If you're feeling particularly generous and wouldn't mind confirming that that PR fixes your build, I'd super appreciate it 🙂

davepacheco commented 2 years ago

Thanks @sjbarag! I cherry-picked that commit onto v22.1.5 and I was able to build the binary. I haven't tested it too hard yet but so far so good. Thanks for the quick turnaround! Do you expect this to be in a new release of v22.1 soon?

sjbarag commented 2 years ago

Thanks @sjbarag! I cherry-picked that commit onto v22.1.5 and I was able to build the binary. I haven't tested it too hard yet but so far so good. Thanks for the quick turnaround! Do you expect this to be in a new release of v22.1 soon?

Excellent, thanks for letting me know! Just merged that PR, so https://github.com/cockroachdb/cockroach/commit/310ef7155cc5fe2a9d68d3c826d645da18c5cad8 should be a valid upstream target. Interestingly, there's not much to release here — the final binary shouldn't be any different with or without that change modulo the fact that it, you know, didn't actually build before 😓 . I'll ask around and see what the process is for this kind of thing.

davepacheco commented 2 years ago

Totally. I had thought there were a lot of commits on the release-22.1 branch that have not yet been released. But looking at the dates on those commits, maybe I've got that wrong. Can you clarify? I can point our build system at a particular commit. I just want it to be as close to an actual release as possible. If there's a bunch of unreleased stuff ahead of this change and there's not a release coming soon, then I'll probably cherry-pick this commit onto the latest release. Thanks again!

sjbarag commented 2 years ago

. But looking at the dates on those commits, maybe I've got that wrong. Can you clarify?

Sure! We're about to release v22.1.6 (based on https://github.com/cockroachlabs/cockroach/commit/760a8253ae6478d69da0330133e3efec8e950e4e), and it looks like I just barely missed the release window this time. Patch releases come out at least once per month (more often if something needs an urgent release), so I'd expect v22.1.7 to be release right around the September 12th timeframe. Looks like there's a pretty small set of commits in between 22.1.6 and the fix here, for what it's worth.

As for why so many commits show up in that compare view — yeah that's pretty strange. Github shows the initial commit date, not the push date. A lot of those commits were written back in July, and the cherry-pick/backport process preserves that date. It's quite unfortunate!

davepacheco commented 2 years ago

Cool. Thanks for that info.

sjbarag commented 2 years ago

Hey @davepacheco ! I'm going to close this out, since 22.1.7 was released on 2022-09-15 and the v22.1.7 tag includes these changes. If something's still broken, please let me know!

Thanks again for reporting this 😃

cockroachdb / cockroach

`make buildoss` build failure on 22.1.5 #86403