Closed dr0i closed 2 years ago
@sroertgen Do you have an idea what we should do?
Thanks @dr0i for reminding me of #130.
I think this is related and I guess the pathPrefix-behaviour changed a little bit updating to Gatsby v4. As I guessed in #130, the pathPrefix
was not correctly picked up.
Please check if branch https://github.com/skohub-io/skohub-vocabs/tree/182-pathPrefix works for you.
Unfortunately that doesn't help.
Also, "pathPrefix" must be a valid relative uri
may be misleading, i.e. it is always followed by (node:18079) UnhandledPromiseRejectionWarning: Error: ENOENT: no such file or directory, lstat '/home/lod/git/skohub-vocabs/src/../public/'
.
This last message results from src/webHookServer.js: fs.moveSync(${__dirname}/../public/, ${__dirname}/../dist/${webhook.repository}/${ref}/)
.
I suspect some weird async filesystem race conditions? I can ACK that the public
folder is temporarily created with the gatsby files resuting from triggering a build. So, maybe it is moved already while another vocab is build in parallel (our test builds include more than one vocab to be build, while the first succeeds the second fails), and when the last vocab build is finished, the webHookServer.js
tries to move public
(while it's not here anymore). This then seems to bring the (which?) programs into confusion in that way that they are not terminated but that subsequent newly triggered build are just logged as Build triggered
but are stuck after that (normally then appears more gatsby log messages).
These observations are also more of substance as our test instance functions very well despite being in the same state of git commits and analogous .env
etc. Only difference is a different OS, different amount of vocabs, maybe lesser CPU load and in general much lesser IO operations.
So question is - can you somehow ensure that the public
directory is not moved when another process is writing to it? Or, even better, don't move it at all but directly write into the proper directory /dist/${webhook.repository}/
?
Don't know if https://github.com/skohub-io/skohub-vocabs/pull/163#issuecomment-1049621769 has anything to do with it.
For completion, some background of BASEURL
:
If a build is triggered via the webHookServer.js
that BASEURL
is passed as a web parameter. Example from the process list:
lod 383827 383826 4 17:55 ? 00:00:02 node src/webHookServer.js
lod 383841 383827 0 17:55 ? 00:00:00 /bin/sh -c BASEURL=/dr0i/skos/heads/master GATSBY_RESPOSITORY_URL=https://github.com/dr0i/skos CI=true npm run build
After checking out a new temporary master branch that one works. Switching back to master and reset --hard origin/master
and redeploying that one works also :astonished: .
Added a clean
when installing via nvm
in ca5e606.
This should do it.
Closing this issue (and hopefully keep it close).
As @sroertgen asked about it - here is one of the log snippets with the error. Now that I am looking at this (huge) log again - I think I spot the problem (it's the second Build triggered
while the first Build triggered
hasn't finished yet):
Build triggered
Processing
Created file: hcrt.ttl
gatsbyLog:·
> skohub-vocabs@0.1.0 build /home/lod/git/skohub-vocabs
> gatsby clean && gatsby build --prefix-paths
gatsbyLog: info Deleting .cache, public, /home/lod/git/skohub-vocabs/node_modules/.cache/babel-loader, /home/lod/git/skohub-vocabs/node_modules/.cache/terser-webpack-plugin
gatsbyLog: info Successfully deleted directories
gatsbyLog: success compile gatsby files - 19.644s
gatsbyLog: success load gatsby config - 0.094s
gatsbyLog: success load plugins - 0.983s
gatsbyLog: success onPreInit - 0.008s
gatsbyLog: success initialize cache - 0.758s
gatsbyLog: success copy gatsby files - 0.210s
gatsbyLog: success Compiling Gatsby Functions - 0.348s
gatsbyLog: info Found these turtle files:
gatsbyLog: info /home/lod/git/skohub-vocabs/data/dini-ag-kim/hcrt/hcrt.ttl
gatsbyLog: success onPreBootstrap - 0.476s
gatsbyLog: success createSchemaCustomization - 0.005s
gatsbyError: warning Calling `createTypes` in the `sourceNodes` API is deprecated. Please use: `createSchemaCustomization`.
gatsbyLog: success Checking for changed pages - 0.000s
gatsbyLog: success source and transform nodes - 0.239s
gatsbyLog: info Writing GraphQL type definitions to /home/lod/git/skohub-vocabs/.cache/schema.gql
gatsbyLog: success building schema - 3.399s
gatsbyLog: Built index for language "de" {
id: 0,
items: 26,
cache: false,
matcher: 0,
worker: undefined,
threshold: 0,
depth: 0,
resolution: 9,
contextual: 0
}
gatsbyLog: Built index for language "en" {
id: 1,
items: 26,
cache: false,
matcher: 0,
worker: undefined,
threshold: 0,
depth: 0,
resolution: 9,
contextual: 0
}
gatsbyLog: Built index for language "uk" {
id: 2,
items: 26,
cache: false,
matcher: 0,
worker: undefined,
threshold: 0,
depth: 0,
resolution: 9,
contextual: 0
}
gatsbyLog: success createPages - 0.392s
gatsbyLog: success createPagesStatefully - 0.180s
gatsbyLog: info Total nodes: 152, SitePage nodes: 86 (use --verbose for breakdown)
gatsbyLog: success Checking for changed pages - 0.000s
gatsbyLog: success onPreExtractQueries - 0.000s
gatsbyLog: success extract queries from components - 3.012s
gatsbyLog: success write out redirect data - 0.003s
gatsbyLog: success onPostBootstrap - 0.001s
gatsbyLog: info bootstrap finished - 33.182s
gatsbyLog: success write out requires - 0.010s
Build triggered
gatsbyLog: success Building production JavaScript and CSS bundles - 35.733s
gatsbyLog: success Building HTML renderer - 25.357s
gatsbyLog: success Execute page configs - 0.066s
gatsbyLog: success Caching Webpack compilations - 0.001s
gatsbyLog: success run queries in workers - 0.227s - 89/89 391.56/s
gatsbyLog: success Running gatsby-plugin-sharp.IMAGE_PROCESSING jobs - 0.212s - 2/2 9.41/s
gatsbyLog: success Merge worker state - 0.002s
gatsbyLog: success Rewriting compilation hashes - 0.001s
gatsbyLog: success Writing page-data.json files to public directory - 0.118s - 86/86 729.64/s
gatsbyLog: success Building static HTML for pages - 3.553s - 86/86 24.21/s
gatsbyLog: success onPostBuild - 0.000s
gatsbyLog:·
Pages
┌ src/components/Concept.js
│ ├ /w3id.org/kim/hcrt/application.de.html
│ └ ...77 more pages available
├ src/components/ConceptScheme.js
│ ├ /w3id.org/kim/hcrt/scheme.de.html
│ └ ...2 more pages available
├ src/components/index.js
│ ├ /index.de.html
│ └ ...2 more pages available
└ src/pages/404.js
├ /404/
└ /404.html
╭────────────────────────────────────────────────────────────────────╮
│ │
│ (SSG) Generated at build time │
│ D (DSG) Deferred static generation - page generated at runtime │
│ ∞ (SSR) Server-side renders at runtime (uses getServerData) │
│ λ (Function) Gatsby function │
│ │
╰────────────────────────────────────────────────────────────────────╯
gatsbyLog: info Done building in 100.415417044 sec
Build Finish
Processing
Created file: hochschulfaechersystematik.ttl
gatsbyLog:·
> skohub-vocabs@0.1.0 build /home/lod/git/skohub-vocabs
> gatsby clean && gatsby build --prefix-paths
gatsbyLog: info Deleting .cache, public, /home/lod/git/skohub-vocabs/node_modules/.cache/babel-loader, /home/lod/git/skohub-vocabs/node_modules/.cache/terser-webpack-plugin
gatsbyLog: info Successfully deleted directories
gatsbyLog: success compile gatsby files - 3.097s
gatsbyError: error The site's gatsby-config.js failed validation:
"pathPrefix" must be a valid relative uri
gatsbyLog: not finished load gatsby config - 0.104s
gatsbyError: npm·
gatsbyError: ERR! code ELIFECYCLE
npm ERR! errno 1
gatsbyError: npm ERR!
gatsbyError: skohub-vocabs@0.1.0 build: `gatsby clean && gatsby build --prefix-paths`
npm ERR! Exit status 1
npm ERR!·
npm ERR! Failed at the skohub-vocabs@0.1.0 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
gatsbyError:·
gatsbyError: npm ERR! A complete log of this run can be found in:
npm ERR! /home/lod/.npm/_logs/2022-04-22T11_03_56_413Z-debug.log
(node:18079) UnhandledPromiseRejectionWarning: Error: ENOENT: no such file or directory, lstat '/home/lod/git/skohub-vocabs/src/../public/'
at Object.lstatSync (fs.js:1115:3)
at Object.lstatSync (/home/lod/git/skohub-vocabs/node_modules/graceful-fs/polyfills.js:311:34)
at statFunc (/home/lod/git/skohub-vocabs/node_modules/fs-extra/lib/util/stat.js:24:20)
at getStatsSync (/home/lod/git/skohub-vocabs/node_modules/fs-extra/lib/util/stat.js:25:19)
at Object.checkPathsSync (/home/lod/git/skohub-vocabs/node_modules/fs-extra/lib/util/stat.js:67:33)
at Object.moveSync (/home/lod/git/skohub-vocabs/node_modules/fs-extra/lib/move/move-sync.js:14:52)
at ChildProcess.<anonymous> (/home/lod/git/skohub-vocabs/src/webHookServer.js:174:12)
at ChildProcess.emit (events.js:400:28)
at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:18079) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:18079) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Build triggered
Build triggered
The programm doesn't get killed, but all subsequent Build triggered
(beginning with the last two) don't do anything.
FYI, just tested this parallel building of vocabs with many many build triggering , but couldn't get it to error. So maybe we are really fine :)
Triggering a build triggers gatsby to build, but the build fails, see e.g. https://skohub.io/build/?id=8a0c33fe-1d3b-4e2e-b6dd-3365f2dbcd0e#11:
We don't have any
BASEURL
configured in.env
. It's used ingatsby_config.js
:May be related to https://github.com/skohub-io/skohub-vocabs/issues/130 or/and https://github.com/skohub-io/skohub-vocabs/issues/175 .