search

skohub-io / skohub-vocabs

A lightweight tool to publish SKOS Vocabularies
https://skohub.io/
Apache License 2.0
34 stars 25 forks source link

gatsby: "pathPrefix" must be a valid relative uri #182

Closed dr0i closed 2 years ago

dr0i commented 2 years ago

Triggering a build triggers gatsby to build, but the build fails, see e.g. https://skohub.io/build/?id=8a0c33fe-1d3b-4e2e-b6dd-3365f2dbcd0e#11:

error The site's gatsby-config.js failed validation: "pathPrefix" must be a valid relative uri

We don't have any BASEURL configured in .env. It's used in gatsby_config.js:

pathPrefix: ${process.env.BASEURL || ''},

May be related to https://github.com/skohub-io/skohub-vocabs/issues/130 or/and https://github.com/skohub-io/skohub-vocabs/issues/175 .

acka47 commented 2 years ago

@sroertgen Do you have an idea what we should do?

sroertgen commented 2 years ago

Thanks @dr0i for reminding me of #130.

I think this is related and I guess the pathPrefix-behaviour changed a little bit updating to Gatsby v4. As I guessed in #130, the pathPrefix was not correctly picked up.

Please check if branch https://github.com/skohub-io/skohub-vocabs/tree/182-pathPrefix works for you.

dr0i commented 2 years ago

Unfortunately that doesn't help. Also, "pathPrefix" must be a valid relative uri may be misleading, i.e. it is always followed by (node:18079) UnhandledPromiseRejectionWarning: Error: ENOENT: no such file or directory, lstat '/home/lod/git/skohub-vocabs/src/../public/'. This last message results from src/webHookServer.js: fs.moveSync(${__dirname}/../public/, ${__dirname}/../dist/${webhook.repository}/${ref}/). I suspect some weird async filesystem race conditions? I can ACK that the public folder is temporarily created with the gatsby files resuting from triggering a build. So, maybe it is moved already while another vocab is build in parallel (our test builds include more than one vocab to be build, while the first succeeds the second fails), and when the last vocab build is finished, the webHookServer.js tries to move public (while it's not here anymore). This then seems to bring the (which?) programs into confusion in that way that they are not terminated but that subsequent newly triggered build are just logged as Build triggered but are stuck after that (normally then appears more gatsby log messages).

These observations are also more of substance as our test instance functions very well despite being in the same state of git commits and analogous .env etc. Only difference is a different OS, different amount of vocabs, maybe lesser CPU load and in general much lesser IO operations.

So question is - can you somehow ensure that the public directory is not moved when another process is writing to it? Or, even better, don't move it at all but directly write into the proper directory /dist/${webhook.repository}/ ?

Don't know if https://github.com/skohub-io/skohub-vocabs/pull/163#issuecomment-1049621769 has anything to do with it.

dr0i commented 2 years ago

For completion, some background of BASEURL: If a build is triggered via the webHookServer.js that BASEURL is passed as a web parameter. Example from the process list:

lod       383827  383826  4 17:55 ?        00:00:02 node src/webHookServer.js
lod       383841  383827  0 17:55 ?        00:00:00 /bin/sh -c BASEURL=/dr0i/skos/heads/master GATSBY_RESPOSITORY_URL=https://github.com/dr0i/skos CI=true npm run build
dr0i commented 2 years ago

After checking out a new temporary master branch that one works. Switching back to master and reset --hard origin/master and redeploying that one works also :astonished: . Added a clean when installing via nvm in ca5e606. This should do it. Closing this issue (and hopefully keep it close).

dr0i commented 2 years ago

As @sroertgen asked about it - here is one of the log snippets with the error. Now that I am looking at this (huge) log again - I think I spot the problem (it's the second Build triggered while the first Build triggered hasn't finished yet):

Build triggered
Processing
Created file: hcrt.ttl
gatsbyLog:·
> skohub-vocabs@0.1.0 build /home/lod/git/skohub-vocabs
> gatsby clean && gatsby build --prefix-paths

gatsbyLog: info Deleting .cache, public, /home/lod/git/skohub-vocabs/node_modules/.cache/babel-loader, /home/lod/git/skohub-vocabs/node_modules/.cache/terser-webpack-plugin

gatsbyLog: info Successfully deleted directories

gatsbyLog: success compile gatsby files - 19.644s

gatsbyLog: success load gatsby config - 0.094s

gatsbyLog: success load plugins - 0.983s

gatsbyLog: success onPreInit - 0.008s

gatsbyLog: success initialize cache - 0.758s

gatsbyLog: success copy gatsby files - 0.210s

gatsbyLog: success Compiling Gatsby Functions - 0.348s

gatsbyLog: info Found these turtle files:

gatsbyLog: info /home/lod/git/skohub-vocabs/data/dini-ag-kim/hcrt/hcrt.ttl

gatsbyLog: success onPreBootstrap - 0.476s

gatsbyLog: success createSchemaCustomization - 0.005s

gatsbyError: warning Calling `createTypes` in the `sourceNodes` API is deprecated. Please use: `createSchemaCustomization`.

gatsbyLog: success Checking for changed pages - 0.000s

gatsbyLog: success source and transform nodes - 0.239s

gatsbyLog: info Writing GraphQL type definitions to /home/lod/git/skohub-vocabs/.cache/schema.gql

gatsbyLog: success building schema - 3.399s

gatsbyLog: Built index for language "de" {
  id: 0,
  items: 26,
  cache: false,
  matcher: 0,
  worker: undefined,
  threshold: 0,
  depth: 0,
  resolution: 9,
  contextual: 0
}

gatsbyLog: Built index for language "en" {
  id: 1,
  items: 26,
  cache: false,
  matcher: 0,
  worker: undefined,
  threshold: 0,
  depth: 0,
  resolution: 9,
  contextual: 0
}

gatsbyLog: Built index for language "uk" {
  id: 2,
  items: 26,
  cache: false,
  matcher: 0,
  worker: undefined,
  threshold: 0,
  depth: 0,
  resolution: 9,
  contextual: 0
}

gatsbyLog: success createPages - 0.392s

gatsbyLog: success createPagesStatefully - 0.180s

gatsbyLog: info Total nodes: 152, SitePage nodes: 86 (use --verbose for breakdown)

gatsbyLog: success Checking for changed pages - 0.000s

gatsbyLog: success onPreExtractQueries - 0.000s

gatsbyLog: success extract queries from components - 3.012s

gatsbyLog: success write out redirect data - 0.003s

gatsbyLog: success onPostBootstrap - 0.001s

gatsbyLog: info bootstrap finished - 33.182s

gatsbyLog: success write out requires - 0.010s

Build triggered
gatsbyLog: success Building production JavaScript and CSS bundles - 35.733s

gatsbyLog: success Building HTML renderer - 25.357s

gatsbyLog: success Execute page configs - 0.066s

gatsbyLog: success Caching Webpack compilations - 0.001s

gatsbyLog: success run queries in workers - 0.227s - 89/89 391.56/s

gatsbyLog: success Running gatsby-plugin-sharp.IMAGE_PROCESSING jobs - 0.212s - 2/2 9.41/s

gatsbyLog: success Merge worker state - 0.002s

gatsbyLog: success Rewriting compilation hashes - 0.001s

gatsbyLog: success Writing page-data.json files to public directory - 0.118s - 86/86 729.64/s

gatsbyLog: success Building static HTML for pages - 3.553s - 86/86 24.21/s

gatsbyLog: success onPostBuild - 0.000s

gatsbyLog:·
Pages

┌ src/components/Concept.js
│ ├   /w3id.org/kim/hcrt/application.de.html
│ └   ...77 more pages available
├ src/components/ConceptScheme.js
│ ├   /w3id.org/kim/hcrt/scheme.de.html
│ └   ...2 more pages available
├ src/components/index.js
│ ├   /index.de.html
│ └   ...2 more pages available
└ src/pages/404.js
  ├   /404/
  └   /404.html

  ╭────────────────────────────────────────────────────────────────────╮
  │                                                                    │
  │     (SSG) Generated at build time                                  │
  │   D (DSG) Deferred static generation - page generated at runtime   │
  │   ∞ (SSR) Server-side renders at runtime (uses getServerData)      │
  │   λ (Function) Gatsby function                                     │
  │                                                                    │
  ╰────────────────────────────────────────────────────────────────────╯

gatsbyLog: info Done building in 100.415417044 sec

 Build Finish
Processing
Created file: hochschulfaechersystematik.ttl
gatsbyLog:·
> skohub-vocabs@0.1.0 build /home/lod/git/skohub-vocabs
> gatsby clean && gatsby build --prefix-paths

gatsbyLog: info Deleting .cache, public, /home/lod/git/skohub-vocabs/node_modules/.cache/babel-loader, /home/lod/git/skohub-vocabs/node_modules/.cache/terser-webpack-plugin

gatsbyLog: info Successfully deleted directories

gatsbyLog: success compile gatsby files - 3.097s

gatsbyError: error The site's gatsby-config.js failed validation:

"pathPrefix" must be a valid relative uri

gatsbyLog: not finished load gatsby config - 0.104s

gatsbyError: npm·
gatsbyError: ERR! code ELIFECYCLE
npm ERR! errno 1

gatsbyError: npm ERR!
gatsbyError:  skohub-vocabs@0.1.0 build: `gatsby clean && gatsby build --prefix-paths`
npm ERR! Exit status 1
npm ERR!·
npm ERR! Failed at the skohub-vocabs@0.1.0 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

gatsbyError:·

gatsbyError: npm ERR! A complete log of this run can be found in:
npm ERR!     /home/lod/.npm/_logs/2022-04-22T11_03_56_413Z-debug.log

(node:18079) UnhandledPromiseRejectionWarning: Error: ENOENT: no such file or directory, lstat '/home/lod/git/skohub-vocabs/src/../public/'
    at Object.lstatSync (fs.js:1115:3)
    at Object.lstatSync (/home/lod/git/skohub-vocabs/node_modules/graceful-fs/polyfills.js:311:34)
    at statFunc (/home/lod/git/skohub-vocabs/node_modules/fs-extra/lib/util/stat.js:24:20)
    at getStatsSync (/home/lod/git/skohub-vocabs/node_modules/fs-extra/lib/util/stat.js:25:19)
    at Object.checkPathsSync (/home/lod/git/skohub-vocabs/node_modules/fs-extra/lib/util/stat.js:67:33)
    at Object.moveSync (/home/lod/git/skohub-vocabs/node_modules/fs-extra/lib/move/move-sync.js:14:52)
    at ChildProcess.<anonymous> (/home/lod/git/skohub-vocabs/src/webHookServer.js:174:12)
    at ChildProcess.emit (events.js:400:28)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:18079) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:18079) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Build triggered
Build triggered

The programm doesn't get killed, but all subsequent Build triggered (beginning with the last two) don't do anything.

dr0i commented 2 years ago

FYI, just tested this parallel building of vocabs with many many build triggering , but couldn't get it to error. So maybe we are really fine :)