Unable to build/run site locally

ronaldtse commented 1 year ago

What has changed? I can't build the site locally anymore.

I have run:

$ make update-init
$ make update-modules

$ make data
cd breviter && yarn install && yarn build && yarn export
yarn install v1.22.19
warning ../package.json: No license field
[1/4] 🔍  Resolving packages...
success Already up-to-date.
$ npm run compile

> breviter@0.2.0 compile
> tsc

✨  Done in 1.24s.
yarn run v1.22.19
warning ../package.json: No license field
$ ([[ -e 'public/db.json' ]] || yarn compute) && yarn init-build
warning ../package.json: No license field
$ yarn cli compute data -o public/db.json
warning ../package.json: No license field
$ ts-node --project tsconfig.bin.json src/cli.ts compute data -o public/db.json
cli handler
[breviter] Computing DB concept dataset at data ...Error: ENOENT: no such file or directory, scandir '/Users/me/src/geolexica/isotc211.geolexica.org/breviter/data/concepts'
    at Object.readdirSync (node:fs:1450:3)
    at listFolder (/Users/me/src/geolexica/isotc211.geolexica.org/breviter/lib/server.ts:9:20)
    at savePrecomputedDB (/Users/me/src/geolexica/isotc211.geolexica.org/breviter/lib/server.ts:36:16)
    at Object.handler (/Users/me/src/geolexica/isotc211.geolexica.org/breviter/src/commands/compute.ts:46:20)
    at /Users/me/src/geolexica/isotc211.geolexica.org/breviter/node_modules/yargs/build/index.cjs:1:8891
    at j (/Users/me/src/geolexica/isotc211.geolexica.org/breviter/node_modules/yargs/build/index.cjs:1:4956)
    at M.handleValidationAndGetResult (/Users/me/src/geolexica/isotc211.geolexica.org/breviter/node_modules/yargs/build/index.cjs:1:8860)
    at M.applyMiddlewareAndGetResult (/Users/me/src/geolexica/isotc211.geolexica.org/breviter/node_modules/yargs/build/index.cjs:1:9502)
    at M.runCommand (/Users/me/src/geolexica/isotc211.geolexica.org/breviter/node_modules/yargs/build/index.cjs:1:7231)
    at Xt.[runYargsParserAndExecuteCommands] (/Users/me/src/geolexica/isotc211.geolexica.org/breviter/node_modules/yargs/build/index.cjs:1:57762) {
  errno: -2,
  syscall: 'scandir',
  code: 'ENOENT',
  path: '/Users/me/src/geolexica/isotc211.geolexica.org/breviter/data/concepts'
}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
make: *** [breviter/out] Error 1

strogonoff commented 1 year ago

I can reproduce the same error.

strogonoff commented 1 year ago

My observations:

Breviter uses a hard-coded path to concepts. That may be one problem an and indicator that breviter may rely on old data model (but not sure, maybe it works with the new one).
That path is empty. In fact, <repo>/breviter/data is completely empty. I am unable to find any code that puts something there, either in Makefile or in package.json scripts or in CI code. I believe @ribose-jeffreylau ran into issues with breviter in CI too, so I’m not sure if it’s doing much currently.

ribose-jeffreylau commented 1 year ago

@strogonoff I could not reproduce the error on branch main.

strogonoff commented 1 year ago

@ribose-jeffreylau

Did you run all three commands?

$ make update-init
$ make update-modules
$ make data

What are your versions?

❯ node --version
v16.18.1
❯ yarn --version
1.22.19

What is your commit? Mine is:

commit 38d02195180665cfe32d9546ad3c2599f3ab33dc

My traceback from make data:

... many lines omitted
Processing isotc211-glossary/geolexica/concept/fed367d1-6df2-5a06-9342-d087265e60fc.yaml
Processing isotc211-glossary/geolexica/concept/ff47ab71-ed1f-55a0-ae5b-f1a5c264f5d0.yaml
Processing isotc211-glossary/geolexica/concept/ffb66d79-8325-5023-888d-46c40a8f16f2.yaml
Processing isotc211-glossary/geolexica/concept/ffbf5e24-7a75-5efb-a7ee-d95320d2b16a.yaml
Done.
cp -f metadata.yaml _source/_data/metadata.yaml
cd breviter && yarn install && yarn build && yarn export
yarn install v1.22.19
warning ../package.json: No license field
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
warning " > @tensorflow-models/universal-sentence-encoder@1.3.3" has unmet peer dependency "@tensorflow/tfjs-converter@^3.6.0".
warning " > @tensorflow-models/universal-sentence-encoder@1.3.3" has unmet peer dependency "@tensorflow/tfjs-core@^3.6.0".
warning "@tensorflow/tfjs > @tensorflow/tfjs-data@3.20.0" has unmet peer dependency "seedrandom@^3.0.5".
[4/4] 🔨  Building fresh packages...
$ npm run compile
npm WARN lifecycle The node binary used for scripts is /var/folders/ln/tn8s27hd7pg4_jx4_49rctfc0000gp/T/yarn--1691049015337-0.7965129327164355/node but npm is using /Volumes/Ops/Setup/SW/NodeJS/node-v16.18.1-darwin-arm64/bin/node itself. Use the `--scripts-prepend-node-path` option to include the path for the node binary npm was executed with.

> breviter@0.2.0 compile /Volumes/Geolexica/isotc211.geolexica.org/breviter
> tsc

✨  Done in 39.73s.
yarn run v1.22.19
warning ../package.json: No license field
$ ([[ -e 'public/db.json' ]] || yarn compute) && yarn init-build
warning ../package.json: No license field
$ yarn cli compute data -o public/db.json
warning ../package.json: No license field
$ ts-node --project tsconfig.bin.json src/cli.ts compute data -o public/db.json
cli handler
[breviter] Computing DB concept dataset at data ...Error: ENOENT: no such file or directory, scandir '/Volumes/Geolexica/isotc211.geolexica.org/breviter/data/concepts'
    at Object.readdirSync (node:fs:1438:3)
    at listFolder (/Volumes/Geolexica/isotc211.geolexica.org/breviter/lib/server.ts:9:20)
    at savePrecomputedDB (/Volumes/Geolexica/isotc211.geolexica.org/breviter/lib/server.ts:36:16)
    at Object.handler (/Volumes/Geolexica/isotc211.geolexica.org/breviter/src/commands/compute.ts:46:20)
    at /Volumes/Geolexica/isotc211.geolexica.org/breviter/node_modules/yargs/build/index.cjs:1:8891
    at j (/Volumes/Geolexica/isotc211.geolexica.org/breviter/node_modules/yargs/build/index.cjs:1:4956)
    at M.handleValidationAndGetResult (/Volumes/Geolexica/isotc211.geolexica.org/breviter/node_modules/yargs/build/index.cjs:1:8860)
    at M.applyMiddlewareAndGetResult (/Volumes/Geolexica/isotc211.geolexica.org/breviter/node_modules/yargs/build/index.cjs:1:9502)
    at M.runCommand (/Volumes/Geolexica/isotc211.geolexica.org/breviter/node_modules/yargs/build/index.cjs:1:7231)
    at Xt.[runYargsParserAndExecuteCommands] (/Volumes/Geolexica/isotc211.geolexica.org/breviter/node_modules/yargs/build/index.cjs:1:57762) {
  errno: -2,
  syscall: 'scandir',
  code: 'ENOENT',
  path: '/Volumes/Geolexica/isotc211.geolexica.org/breviter/data/concepts'
}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
make: *** [breviter/out] Error 1

strogonoff commented 1 year ago

Can you share your output? Do you have a _site folder with content? If you serve it locally, does “semantic search” work (or is it broken as on https://isotc211.geolexica.org/semantic-search/)?

ribose-jeffreylau commented 1 year ago

Semantic search works locally as well as on https://isotc211.geolexica.org/semantic-search/.

This is from https://isotc211.geolexica.org/semantic-search/

ribose-jeffreylau commented 1 year ago

Folder _site is 341MB, not sure how best to share it.

UPDATE: the tar.bz2 is still 177MB. UPDATE: .7z is still 138MB.

ribose-jeffreylau commented 1 year ago

$ node --version ; yarn --version
v16.14.0
3.5.0
$ make clean update-init update-modules data all
rm -rf _site _source/_data/info.yaml _source/_data/metadata.yaml
git submodule update --init
Submodule path 'breviter': checked out '1ae3308eb0ba1ff0b83a028e76b980956de43955'
Submodule path 'isotc211-glossary': checked out '81ad00ef1bcf0124768d1e057a0930d3ff3b5379'
git submodule foreach git pull origin main
Entering 'breviter'
Host key fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU
+--[ED25519 256]--+
|                 |
|     .           |
|      o          |
|     o o o  .    |
|     .B S oo     |
|     =+^ =...    |
|    oo#o@.o.     |
|    E+.&.=o      |
|    ooo.X=.      |
+----[SHA256]-----+
From github.com:geolexica/breviter
 * branch            main       -> FETCH_HEAD
Updating 1ae3308..c1ac6d2
Fast-forward
 .npmignore           |   2 ++
 LICENSE.md           |  29 ++++++++++++++++++
 next.config.js       |   7 +++++
 package.json         |   1 +
 src/ui/lookup.tsx    |  76 ++++++++++++++++++++++++++++++++--------------
 src/ui/testingui.tsx | 208 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------
 styles/globals.css   |  59 ++++++++++++++++++++++++++++++++++++
 7 files changed, 330 insertions(+), 52 deletions(-)
 create mode 100644 .npmignore
 create mode 100644 LICENSE.md
 create mode 100644 next.config.js
Entering 'isotc211-glossary'
Host key fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU
+--[ED25519 256]--+
|                 |
|     .           |
|      o          |
|     o o o  .    |
|     .B S oo     |
|     =+^ =...    |
|    oo#o@.o.     |
|    E+.&.=o      |
|    ooo.X=.      |
+----[SHA256]-----+
remote: Enumerating objects: 2, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 2 (delta 1), reused 1 (delta 1), pack-reused 1
Unpacking objects: 100% (2/2), 218 bytes | 21.00 KiB/s, done.
From github.com:geolexica/isotc211-glossary
 * branch              main       -> FETCH_HEAD
   ea6b70fb..8bd0cbff  main       -> origin/main
Updating 81ad00ef..8bd0cbff
Updating files: 100% (11700/11700), done.
Fast-forward
 .github/workflows/build.yml                                           |    2 +-
...<snip>...
Using geolexica-site 1.7.4
Bundle complete! 1 Gemfile dependency, 184 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
bundle exec jekyll build
Configuration file: /Users/user/share/src/groups/paneron/isotc211.geolexica.org/_config.yml
 Theme Config file: /Users/user/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/jekyll-geolexica-1.8.9/_config.yml
            Source: /Users/user/share/src/groups/paneron/isotc211.geolexica.org/_source
       Destination: /Users/user/share/src/groups/paneron/isotc211.geolexica.org/_site
 Incremental build: disabled. Enable with --incremental
      Generating...
         Geolexica: Loading concepts
         Geolexica: Generating concept pages
         Geolexica: Generating concept images
         Geolexica: Generating meta pages
       Jekyll Feed: Generating feed for posts
Browserslist: caniuse-lite is outdated. Please run:
  npx browserslist@latest --update-db
  Why you should do it regularly: https://github.com/browserslist/browserslist#browsers-data-updating
Successfully compiled 5 files with Babel (828ms).
                    done in 85.281 seconds.
 Auto-regeneration: disabled. Use --watch to enable.

ribose-jeffreylau commented 1 year ago

I think the clean script needs to remove more files. Let me try again with more thorough clean-ing.

ribose-jeffreylau commented 1 year ago

OK, now I can confirm something is missing...

ribose-jeffreylau commented 1 year ago

Try commit 540354a2d14a19a6b14f275f861b53a0c6c6fa11.

$ # make update-init update-modules  # Nothing updated so no need.
$ make clean all

strogonoff commented 1 year ago

OK, will try. By the way, how do you use https://isotc211.geolexica.org/semantic-search/? I never get any results and console writes something about WebGL not available.

ReesePlews commented 1 year ago

for example, in the semantic search type in "attributes of a feature" and it will return entries related to feature attributes. if you enter "cartographic styling of point symbols" it will return entries related to portrayal.

it is working correctly and the results are good. you can type in a description of something and it will return very reasonable results. i think it is helpful for people who are unfamiliar with the terms in the tc211 vocabulary.

hope that helps.

ribose-jeffreylau commented 1 year ago

@strogonoff I think you may also need to use the mouse to click "search" instead of hitting Enter.

ReesePlews commented 1 year ago

interesting, i see that now; it should "sense" a carriage return as "a submission to start the search"; can we add a ticket to address a fix for this? very minor, there is no hurry. good eye for finding that! thanks

strogonoff commented 1 year ago

@strogonoff I think you may also need to use the mouse to click "search" instead of hitting Enter.

I did that every time when testing, to no results. Can you share a query?

ReesePlews commented 1 year ago

the two query strings i added above produced answers for me today.

"attributes of a feature" "cartographic styling of point symbols" "map display" "project map data" etc... should work...

strogonoff commented 1 year ago

~~Interesting. No results for me. Perhaps it only works in some browsers. @ribose-jeffreylau @ReesePlews which browsers do you use for it?~~

Scratch that, the queries from Reese worked, though took a while to load!

ReesePlews commented 1 year ago

i made those tests using ffox (current update); would it be safe to say that the first query takes a while to "initialize"?

strogonoff commented 1 year ago

I think that could be the case, I haven’t looked into our semantic search implementation enough to see, if so it can be updated to display loading message (and in any case it should at least be updated to look consistently with the main page search widget)…

ronaldtse commented 1 year ago

@strogonoff can this be closed now?

ronaldtse commented 3 months ago

All fixed.

geolexica / isotc211.geolexica.org

Unable to build/run site locally #186