Plugin site builds failing on infra.ci

zbynek commented 10 months ago

Service(s)

infra.ci.jenkins.io

Summary

The PR builds are failing with

NestedError: Cannot copy from `.cache/api-runner-browser-plugins.js` to `/opt/  build/cache/cwd/.cache/api-runner-browser-plugins.js`: Cannot create directory   `/opt/build/cache/cwd/.cache`: EACCES: permission denied, mkdir '/opt/build'

- Caused By: NestedError: Cannot create directory `/opt/build/cache/cwd/.cache    `: EACCES: permission denied, mkdir '/opt/build'

e.g. https://infra.ci.jenkins.io/job/website-jobs/job/plugin-site/job/PR-1556/4/display/redirect

Reproduction steps

Check the PR builds for the plugin site -- I guess the error is the same for all of them but cannot verify (thanks @NotMyFault for obtaining that error message)

github-actions[bot] commented 10 months ago

Take a look at these similar issues to see if there isn't already a response to your problem:

71% #3696

dduportal commented 10 months ago

This is a weird error: not sure why the build is trying to temper with /opt:

Gatsby documentation shows https://www.gatsbyjs.com/docs/reference/gatsby-project-structure/ that the .cache directory should be at the project root: not sure why is it trying to copy its content somewhere else 🤔
https://www.gatsbyjs.com/docs/how-to/local-development/environment-variables/#reserved-environment-variables is not really clear on what/why could be updated in the environment

It looks like we have a divergence between Jenkinsfile (on ci.jenkins.io) and Jenkinsfile_k8s (infra.ci.jenkins.io). Maybe trying to merge them both could help to have the same behavior (or at least clostest possible).

But using a docker image requires VM agent which is annoying: I guess we should find a way to understand the technical stack here (NPM + gastby + babel + webpack + other sutff)?

Ping @NotMyFault @halkeye if this error rings a bell ?

dduportal commented 10 months ago

Even weirder:

The master branch build worked on infra.ci (built successfully 2 hours ago): https://github.com/jenkins-infra/plugin-site/blob/ca86de593d0716c5270afc63d9c85c23e989954b/Jenkinsfile_k8s#L93-L107
But not the PR part (https://github.com/jenkins-infra/plugin-site/blob/ca86de593d0716c5270afc63d9c85c23e989954b/Jenkinsfile_k8s#L76-L91)

dduportal commented 10 months ago

Some related issues on the internet:

=> looks like there is "something" in this technical stack which want to write into /opt while it should not (or at least should be configurable)

halkeye commented 10 months ago

Nothing i'm aware of that writes to /opt/build. This feels like an env, or lack of env problem.

Nothing in my greps really stand out, except maybe TMPDIR?

``` fgrep -r process.env . > process.env egrep -o '(process.env.\w+)' process.env | sort | uniq process$env$FORCE_TE process$env$GATSBY_O process$env$GATSBY_W process$env$GATSBY_W2 process$env.COLORS process$env.RTL_SKIP_AUTO_CLEANUP process.env._ process.env access process.env.ACCESS_TOKEN process.env.ACCESS_TOKEN_SECRET process.env.ALGOLIA_API_KEY process.env.ALGOLIA_APP_ID process.env.ALGOLIA_INDEX_NAME process.env.ANDROID_HOME process.env.ANDROID_NDK process.env.ANDROID_NDK_HOME process.env.API_KEY process.env.APPDATA process.env.ARM_VERSION process.env.ASSERT process.env.AUTOPREFIXER_GRID process.env.AWS_EXECUTION_ENV process.env.BABEL_8_BREAKING process.env.BABEL_ENV process.env.BABEL_SHOW_CONFIG_FOR process.env.BABEL_TYPES_8_BREAKING process.env.BEST_API_KEY process.env.BLUEBIRD_DEBUG process.env.BLUEBIRD_LONG_STACK_TRACES process.env.BROWSER process.env.BROWSER_ARGS process.env.BROWSERSLIST process.env.BROWSERSLIST_CONFIG process.env.BROWSERSLIST_DANGEROUS_EXTEND process.env.BROWSERSLIST_DISABLE_CACHE process.env.BROWSERSLIST_ENV process.env.BROWSERSLIST_IGNORE_OLD_DATA process.env.BROWSERSLIST_STATS process.env.BUILD_STAGE process.env but process.env by process.env.CC process.env.CFLAGS process.env.CHOKIDAR_INTERVAL process.env.CHOKIDAR_PRINT_FSEVENTS_REQUIRE_ERROR process.env.CHOKIDAR_USEPOLLING process.env.CI process.env.CI_NAME process.env.CLANG_FORMAT_START process.env.CLAVE_API process.env.CLI_WIDTH process.env.COLORTERM process.env.COLUMNS process.env.COMMIT_REF process.env.comspec process.env.COMSPEC process.env.CONSUMER_KEY process.env.CONSUMER_SECRET process.env.COREPACK_ROOT process.env.CXX process.env.CYPRESS_SUPPORT process.env,D process.env.DB_HOST process.env.DB_PASS process.env.DB_USER process.env.DEBUG process.env.DEBUG_CLI process.env.DEBUG_FD process.env.DEBUG_MIME process.env.DEBUG_NOPT process.env.DEBUG_PRINT_LIMIT process.env.DESTDIR process.env.DEV process.env.DIRNAME process.env.DISABLE_SEARCH_ENGINE process.env.DISABLE_TURBO_CALLS process.env.DISABLE_V8_COMPILE_CACHE process.env.DISPLAY process.env.DOTENV_CONFIG_DEBUG process.env.DOTENV_CONFIG_ENCODING process.env.DOTENV_CONFIG_OVERRIDE process.env.DOTENV_CONFIG_PATH process.env.DOTENV_KEY process.env[e process.env.EDITOR process.env.ELECTRON_RUN_AS_NODE process.env.ENABLE_GATSBY_EXTERNAL_JOBS process.env.ENABLE_GATSBY_REFRESH_ENDPOINT process.env,encoding process.env[env process.env.ENVINFO_DEBUG process.env.ENVIRONMENT process.env[envVar process.env.ESLINT_NO_DEV_ERRORS process.env.ESLINT_USE_FLAT_CONFIG process.env.FAIL process.env.__FAKE_PLATFORM__ process.env.FAST_REFRESH process.env.FAST_TEST process.env field process.env[flag process.env.foo process.env for process.env.FORCE_COLOR process.env.FORCE_TEST_DATABASE_ID process.env.FORMAT_START process.env from process.env.FUNCTION_TARGET process.env.GATSBY_ACTIVE_ENV process.env.GATSBY_ALGOLIA_APP_ID process.env.GATSBY_ALGOLIA_SEARCH_KEY process.env.GATSBY_ALGOLIA_WRITE_KEY process.env.GATSBY_API_URL process.env.GATSBY_CLOUD process.env.GATSBY_CLOUD_IMAGE_CDN process.env.GATSBY_CONCURRENT_DOWNLOAD process.env.GATSBY_CONNECTION_TIMEOUT process.env.GATSBY_CPU_COUNT process.env.GATSBY_DETECT_NODE_MUTATIONS process.env.GATSBY_DIAGNOSTIC_STUCK_STATUS_TIMEOUT process.env.GATSBY_DISABLE_CACHE_PERSISTENCE process.env.GATSBY_DISABLE_CUSTOM_404 process.env.GATSBY_ENABLE_LAZY_IMAGES_IN_CI process.env.GATSBY_ENABLE_QUERY_ON_DEMAND_IN_CI process.env.GATSBY_ENV process.env.gatsby_executing_command process.env.GATSBY_EXPERIMENTAL_CLOUD_CLI process.env.GATSBY_EXPERIMENTAL_DEV_SSR process.env.GATSBY_EXPERIMENTAL_DISABLE_SCHEMA_REBUILD process.env.GATSBY_EXPERIMENTAL_ENABLE_ADMIN process.env.GATSBY_EXPERIMENTAL_FAST_DEV process.env.GATSBY_EXPERIMENTAL_LMDB_INDEXES process.env.GATSBY_EXPERIMENTAL_PARALLEL_SOURCING process.env.GATSBY_EXPERIMENTAL_PRESERVE_FILE_DOWNLOAD_CACHE process.env.GATSBY_EXPERIMENTAL_QUERY_CONCURRENCY process.env.GATSBY_EXPERIMENTAL_QUERY_ON_DEMAND process.env.GATSBY_EXPERIMENTAL_REMOTE_FILE_PLACEHOLDER process.env.GATSBY_FEEDBACK_DISABLED process.env.GATSBY_HOST process.env.GATSBY_INCOMPLETE_RETRY_LIMIT process.env.GATSBY_INITIAL_WEBHOOK_BODY process.env.GATSBY_IS_PREVIEW process.env.GATSBY_JPEG_ENCODER process.env.GATSBY_LOGGER process.env.gatsby_log_level process.env.GATSBY_MATOMO_SITE_ID process.env.GATSBY_MATOMO_SITE_URL process.env.GATSBY_NODE_GLOBALS process.env.GATSBY_OPEN_TRACING_CONFIG_FILE process.env.GATSBY_PARALLEL_QUERY_CHUNK_SIZE process.env.GATSBY_PARTIAL_HYDRATION process.env.GATSBY_PRECOMPILE_DEVELOP_FUNCTIONS process.env.GATSBY_QUERY_ON_DEMAND process.env.GATSBY_QUERY_ON_DEMAND_LOADING_INDICATOR process.env.GATSBY_REFRESH_TOKEN process.env.GATSBY_REPORTER_ISVERBOSE process.env.GATSBY_SHOULD_TRACK_IMAGE_CDN_URLS process.env.GATSBY_SKIP_WRITING_SCHEMA_TO_FILE process.env.GATSBY_SLICE process.env.GATSBY_SLICES process.env.GATSBY_SOCKET_IO_DEFAULT_TRANSPORT process.env.GATSBY_STALL_RETRY_LIMIT process.env.GATSBY_STALL_TIMEOUT process.env.GATSBY_TELEMETRY_API process.env.GATSBY_TELEMETRY_DEBUG process.env.GATSBY_TELEMETRY_DISABLED process.env.GATSBY_TELEMETRY_METADATA_PATH process.env.GATSBY_TELEMETRY_TAGS process.env.GATSBY_TELEMETRY_VERBOSE process.env.GATSBY_WATCHDOG_STUCK_STATUS_TIMEOUT process.env.GATSBY_WEBPACK_LOGGING process.env.GATSBY_WEBPACK_PUBLICPATH process.env.GATSBY_WORKER_ID process.env.GATSBY_WORKER_IN_FLIGHT_DUMP_LOCATION process.env.GATSBY_WORKER_MODULE_PATH process.env.GATSBY_WORKER_POOL_WORKER process.env.GET_CONTENT process.env.GET_CONTENT_SINGLE process.env,GIT_CONFIG_NOSYSTEM process.env.GIT_DIR process.env.GITHUB_SHA process.env.GIT_LITERAL_PATHSPECS process.env.GIT_SSH process.env,GIT_SSH_COMMAND process.env.GIT_SSH_COMMAND process.env.GIT_WORK_TREE process.env.GOOGLE_RUNTIME process.env.GRACEFUL_FS_PLATFORM process.env.GYP_MSVS_OVERRIDE_PATH process.env.GYP_MSVS_VERSION process.env.hasOwnProperty process.env.HEADER_FILE process.env.HELLO process.env.HOME process.env.HOMEDRIVE process.env.HOMEPATH process.env.http_proxy process.env.HTTP_PROXY process.env.https_proxy process.env.HTTPS_PROXY process.env.HUSKY process.env,i process.env)if process.env.IGNORE_TEST_WIN32 process.env.IMAGE_CDN_ENCRYPTION_IV process.env.IMAGE_CDN_ENCRYPTION_SECRET_KEY process.env.IMAGE_CDN_HOSTNAME process.env.IMAGE_CDN_ROUTE_PREFIX process.env in process.env.INTERNAL_GATSBY_TELEMETRY_SESSION_ID process.env.IS_PUBLISH process.env.__IS_WSL_TEST__ process.env.JENKINS_IO_API_URL process.env.JEST_JASMINE process.env.JEST_WORKER_ID process.env.JITI_ALIAS process.env.JITI_CACHE process.env.JITI_DEBUG process.env.JITI_ESM_RESOLVE process.env.JITI_NATIVE_MODULES process.env.JITI_REQUIRE_CACHE process.env.JITI_RESPECT_TMPDIR_ENV process.env.JITI_SOURCE_MAPS process.env.JITI_TRANSFORM_MODULES process.env.JOBS process-env.js process.env[k process.env[key process.env.KONSOLE_VERSION process.env.LAMBDA_TASK_ROOT process.env.LANG process.env.LANGUAGE process.env.LC_ALL process.env.LC_CTYPE process.env.LC_MESSAGES process.env.LIBC process.env.LISTR_DISABLE_COLOR process.env.LMDB_RESTORE process.env.LNAME process.env.LOCALAPPDATA process.env.LOG process.env.LOGNAME process.env.LOG_STREAM process.env.LOG_TOKENS process.env.MACOSX_DEPLOYMENT_TARGET process.env.MAKE process.env.MSGPACKR_NATIVE_ACCELERATION_DISABLED process.env,n process.env[n process.env[N process.env[name process.env.NAMESPACE process.env.NETLIFY process.env.NM_DEBUG_LEVEL process.env.NOCK_BACK_MODE process.env.NOCK_OFF process.env.NO_COLOR process.env.NODE process.env.NODE_BINDINGS_ARROW process.env.NODE_BINDINGS_COMPILED_DIR process.env.NODE_DEBUG process.env.NODE_DISABLE_COLORS process.env.NODE_ENV process.env.NODE_EXTRA_CA_CERTS process.env.NODE_GYP_FORCE_PYTHON process.env.NODEJS_ORG_MIRROR process.env.NODE_MANIFEST_FILE_LIMIT process.env.NODE_MANIFEST_MAX_DAYS_OLD process.env.NODE_OPTIONS process.env.NODE_PORT process.env.NO_DEPRECATION process.env.NODE_PRE_GYP_ABI_CROSSWALK process.env.NODE_TAPE_NO_ONLY_TEST process.env.NODE_TAPE_OBJECT_PRINT_DEPTH process.env.NODE_TLS_REJECT_UNAUTHORIZED process.env.no_proxy process.env.NO_PROXY process.env.NOPT_DEBUG process.env.npm_config_ process.env.npm_config_arch process.env.npm_config_argv process.env.npm_config_build_from_source process.env.npm_config_force_process_config process.env.npm_config_libc process.env.npm_config_loglevel process.env.npm_config_platform process.env.npm_config_sharp_dist_base_url process.env.npm_config_sharp_install_force process.env.npm_config_sharp_libvips_binary_host process.env.npm_config_sharp_libvips_local_prebuilds process.env.npm_config_user_agent process.env.npm_config_x process.env.npm_lifecycle_event process.env.npm_lifecycle_script process.env.npm_package_engines_node process.env.npm_package_name process.env.NYC_CONFIG process.env.NYC_CWD process.env[o process.env;Object process.env.OPEN_MATCH_HOST_ONLY process.env[option process.env.OS process.env.OSTYPE process.env[p process.env.PARCEL_BUILD_ENV process.env.PARCEL_DUMP_GRAPHVIZ process.env.PARCEL_MAX_CONCURRENT_CALLS process.env.PARCEL_SELF_BUILD process.env.PARCEL_SWC_WASM process.env.PARCEL_WORKER_BACKEND process.env.PARCEL_WORKERS process.env.PATH process.env.PATHEXT process.env[pathKey process.env[pkg process.env.PNP_ALWAYS_WARN_ON_FALLBACK process.env.PNP_DEBUG_LEVEL process.env.PORT process.env.PREBUILDS_ONLY process.env.PREFIX process.env.PREFIX_PATHS process.env.PRELOAD_GET_ITERATOR process.env.PRESERVE_FILE_DOWNLOAD_CACHE process.env.ProgramFiles process.env.ProgramW6432 process.env[propName process.env.PROXY process.env[proxyEnv process.env.PYTHON process.env.PYTHONPATH process.env.REACT_EDITOR process.env.REACT_PROFILE process.env.READABLE_STREAM process.env replacement process.env.REPOSITORY_URL process.env;return process.env.RTL_SKIP_AUTO_CLEANUP process.env.RUST_TARGET process.env.S3_BUCKET process.env.SENTRY_DSN process.env.SENTRY_ENVIRONMENT process.env.SENTRY_NAME process.env.SENTRY_RELEASE process.env.SENTRY_TRACES_SAMPLE_RATE process.env.SHARP_DIST_BASE_URL process.env.SHARP_INSTALL_FORCE process.env.SHELL process.env so process.env.socks_proxy process.env,stdin process.env,strict process.env.SUDO_GID process.env.SUDO_UID process.env.SystemDrive process.env.SYSTEMDRIVE process.env.SystemRoot process.env.SYSTEMROOT process.env,t process.env[t process.env.TELEMETRY_BUFFER_INTERVAL process.env.TEMP process.env temporarily process.env.TERM process.env.TERM_PROGRAM process.env.TERSER_DEBUG_DIR process.env.TEST_FORCE_CASE_FS process.env.TEST_GRACEFUL_FS_GLOBAL_PATCH process.env.__TESTING_MKDIRP_NODE_VERSION__ process.env.__TESTING_MKDIRP_PLATFORM__ process.env.TESTING_TAR_FAKE_PLATFORM process.env.TEST_LIB_PATH process.env.TEST_PSEUDOMAP process.env.TEST_VARIANT process.env.TIMING process.env.TMP process.env.TMPDIR process.env.TRACE_DEPRECATION process.env.TRAVIS process.env.TSESTREE_NO_INVALIDATION process.env.TSESTREE_SINGLE_RUN process.env.TS_NODE_BASEURL process.env.TS_NODE_PROJECT process.env.TZ process.env,U process.env[Ua process.env.URL process.env.USER process.env.USERNAME process.env.USERPROFILE process.env.V8_COMPILE_CACHE_CACHE_DIR process.env value process.env[varName process.env.VCINSTALLDIR process.env.VERBOSE process.env.VERBOSE_NODE_MANIFEST process.env.VERCEL_BITBUCKET_COMMIT_SHA process.env.VERCEL_GIT_COMMIT_SHA process.env.VERCEL_GITHUB_COMMIT_SHA process.env.VERCEL_GITLAB_COMMIT_SHA process.env.VISUAL process.env.VSINSTALLDIR process.env.WATCHMAN_SOCK process.env.WATCHPACK_POLLING process.env.WATCHPACK_WATCHER_LIMIT process.env.WATCH_REPORT_DEPENDENCIES process.env.WDS_SOCKET_HOST process.env.WDS_SOCKET_PATH process.env.WDS_SOCKET_PORT process.env.windir process.env with process.env.WS_NO_BUFFER_UTIL process.env.WS_NO_UTF_8_VALIDATE process.env.WT_SESSION process.env.WVM_DEBUG process.env.XDG_CONFIG_HOME process.env.XDG_DATA_HOME process.env.YARGS_MIN_NODE_VERSION process.env.YARN_IGNORE_CWD process.env.YARN_IGNORE_NODE process.env,YARN_IGNORE_PATH process.env.YARN_IGNORE_PATH process.env.YARN_INJECT_NPM_2FA_TOKEN process.env.YARN_INJECT_NPM_PASSWORD process.env.YARN_INJECT_NPM_USER process.env.YARN_IS_TEST_ENV process.env.YOUR_APP_ENV process.env.YURNALIST_SILENT process.env.ZEIT_BITBUCKET_COMMIT_SHA process.env.ZEIT_GITHUB_COMMIT_SHA process.env.ZEIT_GITLAB_COMMIT_SHA process.env.ZOSLIB_INCLUDES process.env.ZSH_NAME ```

halkeye commented 10 months ago

I did look for .cache too, and for the most part its path.join(process.cwd, '.cache') which could produce this if /opt/build is pwd, but i don't think it can be.

path.join(store.getState().program.directory,public) is a maybe though.

directoryPath(path.join(.cache,gatsby-browser-entry.js))

i'd have to go look at the non minified github though

zbynek commented 10 months ago

@halkeye another weird thing is that the wrong path is build/cache/cwd/.cache/api-runner-browser-plugins.js so it looks like escaping problem with cwd where something prints "cwd" as string literal instead of process.cwd... not sure if that helps in any way though 🤷

dduportal commented 10 months ago

Thanks folks, I'm gonna replay some builds to see what I can get from the environmenet during the build.

If we can't get something in 2 days, and if it ok for you, I'll unify the build tasks to make it work like on ci.jenkins.io

zbynek commented 10 months ago

@dduportal thanks, merging the Jenkinsfiles was also suggested in https://github.com/jenkins-infra/plugin-site/issues/1377

dduportal commented 10 months ago

@dduportal thanks, merging the Jenkinsfiles was also suggested in [jenkins-infra/plugin-site#1377](https://github.com/jenkins-infra/plugin-site/issues/1377

Thanks for the pointer! I did not see the issue earlier, however it is annoying having to spin up a VM instead of a pod to build a static website (that seems overkill, hence the initial reluctance). But we have to unblock this so let's consider the option