web-platform-tests / wpt.fyi

web-platform-tests dashboard
https://wpt.fyi/
Other
183 stars 88 forks source link

Local build is broken #3840

Open atjn opened 3 months ago

atjn commented 3 months ago

Hi :) I am trying to follow the guide on setting up local development of the wpt.fyi dashboard. I am running into a lot of problems and I have no idea how to proceed.

I have executed the exact commands detailed in: https://github.com/web-platform-tests/wpt.fyi/tree/f319b11328efa5312fb7f472ca9222df58e2fb9f?tab=readme-ov-file#development The only exception is that I need to run sudo in front of the commands, otherwise Docker won't run. I cannot see how that would create the following errors, I run many other Docker containers this way with no problems.

Error 1

Git decides to crash because the directory /home/user/wpt.fyi does not have the correct owner. I fixed this temporarily by adding the following to /Makefile:

@@ -56,6 +56,7 @@ go_build: git mockgen packr2 github_action_go_setup
        make webapp_node_modules_prod
        go generate ./...
        # Check all packages without producing any output.
+       git config --global --add safe.directory /home/user/wpt.fyi
        go build -v ./...
        # Build the webapp.
        go build -v ./webapp/web

Error 2

The Google cloud emulator crashes because it cannot find env.yaml in the data_dir. I fixed this by adding the following to util/docker-dev/web_server.sh:

@@ -21,4 +21,4 @@ if [ "${DOCKER_STATUS}" != "0" ]; then
 fi

 info "Starting web server. Port forwarded to host: ${WPTD_HOST_WEB_PORT}"
-wptd_exec_it "\$(gcloud beta emulators datastore env-init) && util/server-watch.sh"
+wptd_exec_it "\$(gcloud config set project wptdashboard) && \$(gcloud beta emulators datastore start) && \$(gcloud beta emulators datastore env-init) && util/server-watch.sh"

Error 3

At this point it seems like the setup script runs correctly, but trying to access loclhost:8080 or localhost:8081 results in a connection reset error. I do not know how to fix that.

My operating system: Linux, Ubuntu 22.04 LTS CPU: Modern Intel AMD64

I hope you can help me figure out what is wrong. I am happy to share any other details.

Here is the full log of what I did to see error 3 after I had applied the fixes for error 1 and 2.

Full log ``` atjn:~/Documents/GitHub/wpt.fyi$ sudo ./util/docker-dev/run.sh [sudo] password for atjn: [ 2024-05-25 23:03:18 INFO ] Creating docker instance for dev server. Instance name: wptd-dev-instance [ 2024-05-25 23:03:18 WARN ] Found existing docker instance wptd-dev-instance. Remove? Confirm (y/N): y [ 2024-05-25 23:03:21 INFO ] Stopping wptd-dev-instance... wptd-dev-instance [ 2024-05-25 23:03:31 INFO ] wptd-dev-instance stopped. [ 2024-05-25 23:03:31 INFO ] Removing wptd-dev-instance... wptd-dev-instance [ 2024-05-25 23:03:32 INFO ] wptd-dev-instance removed. [ 2024-05-25 23:03:32 INFO ] Starting docker instance wptd-dev-instance... 8ebfed1339063c589a1f3aa8ec73044d08def38739d615e97bd425da2bd12fae [ 2024-05-25 23:03:33 INFO ] Setting up local user groupadd: GID '0' already exists useradd: UID 0 is not unique atjn:~/Documents/GitHub/wpt.fyi$ sudo ./util/docker-dev/web_server.sh if [[ "$(which inotifywait)" == "" ]]; then \ sudo apt-get install -qqy --no-install-suggests inotify-tools; \ fi debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package libinotifytools0:amd64. (Reading database ... 84304 files and directories currently installed.) Preparing to unpack .../libinotifytools0_3.14-8.1_amd64.deb ... Unpacking libinotifytools0:amd64 (3.14-8.1) ... Selecting previously unselected package inotify-tools. Preparing to unpack .../inotify-tools_3.14-8.1_amd64.deb ... Unpacking inotify-tools (3.14-8.1) ... Setting up libinotifytools0:amd64 (3.14-8.1) ... Setting up inotify-tools (3.14-8.1) ... Processing triggers for libc-bin (2.31-13+deb11u7) ... [ 2024-05-25 23:04:30 INFO ] Building web server... if [[ "$(which git)" == "" ]]; then sudo apt-get install -qqy --no-install-suggests git; fi if [ "$(which mockgen)" == "" ]; then \ go install github.com/golang/mock/mockgen; \ fi go: downloading github.com/golang/mock v1.6.0 go: downloading golang.org/x/mod v0.8.0 go: downloading golang.org/x/tools v0.6.0 go: downloading golang.org/x/sys v0.20.0 if [ "$(which packr2)" == "" ]; then \ go install github.com/gobuffalo/packr/v2/packr2; \ fi go: downloading github.com/gobuffalo/packr/v2 v2.8.3 go: downloading github.com/gobuffalo/logger v1.0.7 go: downloading github.com/spf13/cobra v1.6.1 go: downloading github.com/gobuffalo/packd v1.0.2 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/markbates/oncer v1.0.0 go: downloading github.com/markbates/safe v1.0.1 go: downloading github.com/karrick/godirwalk v1.17.0 go: downloading github.com/markbates/errx v1.1.0 go: downloading github.com/rogpeppe/go-internal v1.9.0 go: downloading golang.org/x/sync v0.7.0 go: downloading golang.org/x/term v0.20.0 go: downloading github.com/spf13/pflag v1.0.5 # https://github.com/web-platform-tests/wpt.fyi/issues/3089 if [ -d "/github/workspace" ]; then \ echo "Avoiding buildvcs error for Go 1.18+ by marking github workspace safe."; \ git config --global --add safe.directory /github/workspace ; \ else \ echo "Did not detect github workspace. Skipping." ; \ fi Did not detect github workspace. Skipping. make webapp_node_modules_prod make[1]: Entering directory '/home/user/wpt.fyi' if [[ "$(which curl)" == "" ]]; then sudo apt-get install -qqy --no-install-suggests curl; fi if [[ "$(which gpg)" == "" ]]; then \ sudo apt-get install -qqy --no-install-suggests gnupg; \ fi if [[ "$(which node)" == "" ]]; then \ curl -sL https://deb.nodesource.com/setup_18.x | sudo -E bash -; \ sudo apt-get install -qqy nodejs; \ fi cd webapp; npm install npm WARN skipping integrity check for git dependency ssh://git@github.com/web-padawan/launchpad.git npm WARN deprecated har-validator@5.1.5: this library is no longer supported npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated npm WARN deprecated source-map-url@0.4.1: See https://github.com/lydell/source-map-url#deprecated npm WARN deprecated stable@0.1.8: Modern JS already guarantees Array#sort() is a stable sort, so this library is deprecated. See the compatibility table on MDN: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort#browser_compatibility npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated npm WARN deprecated source-map-resolve@0.5.3: See https://github.com/lydell/source-map-resolve#deprecated npm WARN deprecated nomnom@1.8.1: Package no longer supported. Contact support@npmjs.com for more info. npm WARN deprecated multer@1.4.4: Multer 1.x is affected by CVE-2022-24434. This is fixed in v1.4.4-lts.1 which drops support for versions of Node.js before 6. Please upgrade to at least Node.js 6 and version 1.4.4-lts.1 of Multer. If you need support for older versions of Node.js, we are open to accepting patches that would fix the CVE on the main 1.x release line, whilst maintaining compatibility with Node.js 0.10. npm WARN deprecated samsam@1.1.2: This package has been deprecated in favour of @sinonjs/samsam npm WARN deprecated formatio@1.2.0: This package is unmaintained. Use @sinonjs/formatio instead npm WARN deprecated formatio@1.1.1: This package is unmaintained. Use @sinonjs/formatio instead npm WARN deprecated xmldom@0.1.31: Deprecated due to CVE-2021-21366 resolved in 0.5.0 npm WARN deprecated samsam@1.3.0: This package has been deprecated in favour of @sinonjs/samsam npm WARN deprecated json3@3.3.2: Please use the native JSON object instead of JSON 3 npm WARN deprecated debug@4.1.1: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797) npm WARN deprecated mkdirp@0.5.1: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.) npm WARN deprecated uuid@3.4.0: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details. npm WARN deprecated sw-precache@5.2.1: Please migrate to Workbox: https://developers.google.com/web/tools/workbox/guides/migrations/migrate-from-sw npm WARN deprecated debug@4.1.1: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797) npm WARN deprecated debug@4.1.1: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797) npm WARN deprecated sw-toolbox@3.6.0: Please migrate to Workbox: https://developers.google.com/web/tools/workbox/guides/migrations/migrate-from-sw npm WARN deprecated request@2.88.0: request has been deprecated, see https://github.com/request/request/issues/3142 npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142 npm WARN deprecated text-encoding@0.6.4: no longer maintained npm WARN deprecated core-js@2.6.12: core-js@<3.23.3 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Some versions have web compatibility issues. Please, upgrade your dependencies to the actual version of core-js. > postinstall > cpy 'node_modules/@browser-logos/{chrome,chrome-beta,chrome-canary,chrome-dev,chromium,deno,edge,edge-beta,edge-canary,edge-dev,firefox,firefox-beta,firefox-nightly,geckoview,node.js,safari,servo,uc}/*_64x64.png' static && cpy 'node_modules/@browser-logos/firefox-developer-edition/*_64x64.png' static --rename=firefox-dev_64x64.png && cpy 'node_modules/@browser-logos/safari/*_64x64.png' static --rename=safari-beta_64x64.png && cpy 'node_modules/@browser-logos/safari-technology-preview/*_64x64.png' static --rename=safari-dev_64x64.png && cpy 'node_modules/@browser-logos/safari-technology-preview/*_64x64.png' static --rename=safari-preview_64x64.png && cpy 'node_modules/@browser-logos/webkit-nightly/*_64x64.png' static --rename=wktr_64x64.png added 1328 packages, and audited 1409 packages in 18s 108 packages are looking for funding run `npm fund` for details 48 vulnerabilities (5 low, 10 moderate, 19 high, 14 critical) To address issues that do not require attention, run: npm audit fix Some issues need review, and may require choosing a different dependency. Run `npm audit` for details. npm notice npm notice New minor version of npm available! 10.5.0 -> 10.8.0 npm notice Changelog: npm notice Run `npm install -g npm@10.8.0` to update! npm notice cd webapp; npm prune --production npm WARN config production Use `--omit=dev` instead. removed 1 package, and audited 81 packages in 2s found 0 vulnerabilities make[1]: Leaving directory '/home/user/wpt.fyi' go generate ./... go: downloading github.com/google/go-github/v47 v47.1.0 go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible go: downloading golang.org/x/oauth2 v0.20.0 go: downloading github.com/gorilla/mux v1.8.1 go: downloading github.com/deckarep/golang-set v1.8.0 go: downloading cloud.google.com/go/datastore v1.17.0 go: downloading google.golang.org/api v0.180.0 go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading github.com/stretchr/testify v1.9.0 go: downloading cloud.google.com/go/storage v1.41.0 go: downloading github.com/taskcluster/taskcluster-lib-urls v13.0.1+incompatible go: downloading github.com/taskcluster/taskcluster/v44 v44.23.4 go: downloading github.com/google/uuid v1.6.0 go: downloading cloud.google.com/go v0.113.0 go: downloading github.com/phayes/freeport v0.0.0-20220201140144-74d24b5ae9f5 go: downloading github.com/samthor/nicehttp v1.0.0 go: downloading cloud.google.com/go/cloudtasks v1.12.8 go: downloading cloud.google.com/go/logging v1.10.0 go: downloading cloud.google.com/go/secretmanager v1.13.0 go: downloading github.com/gomodule/redigo v1.9.2 go: downloading github.com/gorilla/handlers v1.5.2 go: downloading github.com/gorilla/securecookie v1.1.2 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240513163218-0867130af1f8 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading github.com/tebeka/selenium v0.9.9 go: downloading google.golang.org/grpc v1.64.0 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/davecgh/go-spew v1.1.1 go: downloading github.com/pmezard/go-difflib v1.0.0 go: downloading github.com/google/go-querystring v1.1.0 go: downloading golang.org/x/crypto v0.23.0 go: downloading github.com/googleapis/gax-go/v2 v2.12.4 go: downloading google.golang.org/protobuf v1.34.1 go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading cloud.google.com/go/iam v1.1.8 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/felixge/httpsnoop v1.0.4 go: downloading github.com/blang/semver v3.5.1+incompatible go: downloading golang.org/x/net v0.25.0 go: downloading go.opencensus.io v0.24.0 go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240513163218-0867130af1f8 go: downloading cloud.google.com/go/longrunning v0.5.7 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/go-logr/logr v1.4.1 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading golang.org/x/text v0.15.0 go: downloading google.golang.org/api v0.180.0 go: downloading cloud.google.com/go/auth v0.4.1 go: downloading cloud.google.com/go/auth/oauth2adapt v0.2.2 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading golang.org/x/time v0.5.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading github.com/golang/protobuf v1.5.4 go: downloading github.com/taskcluster/taskcluster/v44 v44.23.4 go: downloading github.com/taskcluster/httpbackoff/v3 v3.1.0 go: downloading github.com/cenkalti/backoff/v3 v3.2.2 go: downloading github.com/tent/hawk-go v0.0.0-20161026210932-d341ea318957 go: downloading github.com/orcaman/writerseeker v0.0.0-20200621085525-1d3f536ff85e go: downloading github.com/taskcluster/slugid-go v1.1.0 go: downloading github.com/johncgriffin/overflow v0.0.0-20211019200055-46fa312c352c go: downloading github.com/fatih/camelcase v1.0.0 go: downloading github.com/pborman/uuid v1.2.1 # Check all packages without producing any output. git config --global --add safe.directory /home/user/wpt.fyi go build -v ./... net/http/internal/testcert internal/sysinfo github.com/dgryski/go-farm github.com/web-platform-tests/wpt.fyi/api/query/cache/lru github.com/tebeka/selenium/log github.com/golang/mock/gomock github.com/pmezard/go-difflib/difflib github.com/davecgh/go-spew/spew net/http/httptest github.com/web-platform-tests/wpt.fyi/api/query testing github.com/web-platform-tests/wpt.fyi/shared/metrics github.com/web-platform-tests/wpt.fyi/api/screenshot github.com/phayes/freeport github.com/web-platform-tests/wpt.fyi/util github.com/web-platform-tests/wpt.fyi/api/azure/mock_azure github.com/web-platform-tests/wpt.fyi/api/checks/mock_checks github.com/web-platform-tests/wpt.fyi/api/manifest/mock_manifest github.com/web-platform-tests/wpt.fyi/api/query/cache/backfill/mock_backfill github.com/stretchr/testify/assert github.com/web-platform-tests/wpt.fyi/webapp/packrd github.com/web-platform-tests/wpt.fyi/api/receiver/mock_receiver github.com/web-platform-tests/wpt.fyi/api/taskcluster/mock_taskcluster github.com/web-platform-tests/wpt.fyi/api github.com/web-platform-tests/wpt.fyi/api/query/cache/index github.com/web-platform-tests/wpt.fyi/api/query/cache/query gopkg.in/yaml.v2 github.com/blang/semver archive/zip github.com/web-platform-tests/wpt.fyi/api/query/cache/monitor github.com/web-platform-tests/wpt.fyi/api/query/cache/poll github.com/web-platform-tests/wpt.fyi/api/query/cache/backfill github.com/web-platform-tests/wpt.fyi/api/query/test github.com/web-platform-tests/wpt.fyi/shared/sharedtest github.com/tebeka/selenium/internal/zip github.com/tebeka/selenium/chrome github.com/tebeka/selenium/firefox github.com/web-platform-tests/wpt.fyi/api/query/cache/service github.com/tebeka/selenium github.com/samthor/nicehttp github.com/web-platform-tests/wpt.fyi/webdriver github.com/web-platform-tests/wpt.fyi/webapp github.com/web-platform-tests/wpt.fyi/webapp/web # Build the webapp. go build -v ./webapp/web [ 2024-05-25 23:05:54 INFO ] Starting web server. Port forwarded to host: 8080 Updated property [core/project]. Executing: /usr/lib/google-cloud-sdk/platform/cloud-datastore-emulator/cloud_datastore_emulator create --project_id=wptdashboard /root/.config/gcloud/emulators/datastore [datastore] May 25, 2024 9:05:56 PM com.google.cloud.datastore.emulator.CloudDatastore$CreateAction$1 apply [datastore] INFO: Provided project_id to Cloud Datastore emulator create command, which is no longer necessary. [datastore] Created new Cloud Datastore project in '/root/.config/gcloud/emulators/datastore'. Executing: /usr/lib/google-cloud-sdk/platform/cloud-datastore-emulator/cloud_datastore_emulator start --host=localhost --port=8081 --store_on_disk=True --allow_remote_shutdown --consistency=0.9 /root/.config/gcloud/emulators/datastore [datastore] May 25, 2024 9:05:56 PM com.google.cloud.datastore.emulator.CloudDatastore$FakeDatastoreAction$9 apply [datastore] INFO: Provided --allow_remote_shutdown to start command which is no longer necessary. [datastore] May 25, 2024 9:05:56 PM com.google.cloud.datastore.emulator.CloudDatastore$StartAction apply [datastore] INFO: Running CLOUD_DATASTORE emulator. [datastore] May 25, 2024 9:05:56 PM com.google.cloud.datastore.emulator.impl.LocalDatastoreFileStub [datastore] INFO: Local Datastore initialized: [datastore] Type: High Replication [datastore] Storage: /root/.config/gcloud/emulators/datastore/WEB-INF/appengine-generated/local_db.bin [datastore] May 25, 2024 9:05:57 PM com.google.cloud.datastore.emulator.impl.LocalDatastoreFileStub load [datastore] INFO: The backing store, /root/.config/gcloud/emulators/datastore/WEB-INF/appengine-generated/local_db.bin, does not exist. It will be created. [datastore] API endpoint: http://localhost:8081 [datastore] If you are using a library that supports the DATASTORE_EMULATOR_HOST environment variable, run: [datastore] [datastore] export DATASTORE_EMULATOR_HOST=localhost:8081 [datastore] [datastore] Dev App Server is now running. [datastore] [datastore] The previous line was printed for backwards compatibility only. [datastore] If your tests rely on it to confirm emulator startup, [datastore] please migrate to the emulator health check endpoint (/). Thank you! [datastore] The health check endpoint for this emulator instance is http://localhost:8081/ ```