npm install is very slow

[StefanSalewski]

Dear Sir,

last year I created two tiny pages with your great Doks Hugo theme in version 0.5. Recently I discovered that now finally versions 1.x are available, so I just tried creating a new project from scratch -- when it should finally succeed, I might transfer my existing text files and upgrade the pages.

Two initial remarks, in case you are interested in improving your website with the instructions:

https://getdoks.org/docs/start-here/getting-started/

npm create hyas@latest -- --template doks

For people like me, not evolved with professional web development, it might not be that obvious that this command is interactive and will ask for the actual project name. Please consider adding a hint -- I was tempted to replace doks with my actual project name.

The next point is much more important:

npm install

started for me with some animations and displaying timings, but then stalled for minutes. I finally pressed some CTRL codes in the terminal after four minutes without visible changes, and tried again. Again same behaviour. And my Internet connection is typically very fast! This time I let it running, while using the web browser to see if I could use other packages managers on my Gentoo Linux box. Then finally, after more than 8 minutes, it finished successfully:

$ npm create hyas@latest -- --template doks
✔ Project name: … nimprogramming.com

Scaffolding project in /home/salewski/nimprogramming.com...

Done. Now run:

  cd nimprogramming.com
  npm install
  npm run dev

salewski@hx90 ~ $ cd nimprogramming.com
salewski@hx90 ~/nimprogramming.com $ npm install

> nimprogramming.com@0.0.0 postinstall
> hugo-installer --version otherDependencies.hugo --extended --destination node_modules/.bin/hugo

Hugo Installer

> Checking for existing binary
> Downloading binary from "https://github.com/gohugoio/hugo/releases/download/v0.123.7/hugo_extended_0.123.7_Linux-64bit.tar.gz"
> Downloading checksum from "https://github.com/gohugoio/hugo/releases/download/v0.123.7/hugo_0.123.7_checksums.txt"
> Verifying binary checksum
> Extracting binary to disk
> Verifying binary health

Hugo is now available in "node_modules/.bin/hugo".

- Version       0.123.7
- Extended      Yes
- OS            linux
- Architecture  x64

hugo v0.123.7-312735366b20d64bd61bff8627f593749f86c964+extended linux/amd64 BuildDate=2024-03-01T16:16:06Z VendorInfo=gohugoio

Success!

added 414 packages in 8m

60 packages are looking for funding
  run `npm fund` for details

So please consider adding a hint that this action can take very long!

Best regards,

Dr. Stefan Salewski

[h-enk]

Thank you for your feedback!

I will update the "Create a new project" instructions (I'm thinking of an animated gif, so you can see how it works).

The npm install should take about 30 seconds maximum (also/especially on Linux). Most likely, it will have to do with your current setup. You could try using the latest Node.js (LTS) — if you're not using it already. Or, use pnpm — highly recommended for it is significantly faster than npm.

Giving it some second thought, it could also have to do with Hugo Installer (I recall an issue with it not working smoothly on a Virtual Machine). The upcoming Doks release (available by the end of this week) will require you to use your globally installed Hugo instead. It would be interesting to see how that works for you.

[StefanSalewski]

Dear Sir,

thank you very much for the fast and detailed reply. I will try to repeat the reported steps this evening just to ensure that it was not only a temporary issue. (Well, I tried two times yesterday.) In my personal view, animated gifs are more than necessary, just a few words that some interactive action may follow should be fine. Animated Gifs might look unprofessional -- if displayed slowly, they might look boring, if played fast, it might be difficult to follow. Well, that is my personal view only, for a few situations animations might be really helpful.

My box is an up-to-date Gentoo-Linux, two years old AMD64 processor with 8 cores and 64 GB RAM with fast 100 MB/s Internet connection in Germany. Latest nodejs-20.12.1 is installed. I might try pnpm. Please consider also adding that hint "use pnpm — highly recommended for it is significantly faster than npm" to your instructions page, because not all of your users are professional web developers and might be not aware of this fact.

will require you to use your globally installed Hugo instead.

Oh, I was assuming that the globally Hugo was used, I had it already installed as www-apps/hugo-0.124.1::gentoo, I think last year with your Doks 0.5 I was required to install it globally with the Gentoo package manager? I will try again when that version is available and report the results.

Best regards,

Dr. Stefan Salewski

[StefanSalewski]

Next test, same behaviour:

rm -rf nimprogramming.com
salewski@hx90 ~ $ npm create hyas@latest -- --template doks
✔ Project name: … nimprogramming.com

Scaffolding project in /home/salewski/nimprogramming.com...

Done. Now run:

  cd nimprogramming.com
  npm install
  npm run dev

salewski@hx90 ~ $ cd nimprogramming.com
salewski@hx90 ~/nimprogramming.com $ npm install
(#########⠂⠂⠂⠂⠂⠂⠂⠂⠂) ⠦ idealTree: timing idealTree Completed in 34646ms

It hangs displaying above message for about seven minutes. Finally, after eight minutes it terminates successfully. Not really that bad, but some people might assume an issue after seven minutes of no update and abort the install.

[StefanSalewski]

The upcoming Doks release (available by the end of this week) will require you to use your globally installed Hugo instead. It would be interesting to see how that works for you.

My feeling is, that that release is not yet available? Does it make sense to wait with updating my two pages

https://nimprogramming.com/ and https://nimprogrammingbook.com/

Well, I am not in a hurry, I have even considered removing that pages from the Internet, as the interest in my book is very low. But in case I should decide to update the pages, I would like to use a sort of "stable" doks release, so that I do not have to update again soon.

[h-enk]

My feeling is, that that release is not yet available?

It is — I only did not find the time to communicate it. I will release update instructions soonish (I'm actually working on them now, so you're timing is great 😄 )

But in case I should decide to update the pages, I would like to use a sort of "stable" doks release, so that I do not have to update again soon.

I get that. Doks 1.6 is a stable, restructured release — I don't expect/have planned for significant changes (after about three and a half years since the first release)

I'll leave a comment here when the instructions are published

[StefanSalewski]

I just found the announcement of Hyas 2.4 at https://gethyas.com/blog/hyas-2-4/. Actually, I was aware that Doks uses Hyas, but not about the fact that you are the main author of Hyas.

So I just tried a fresh installation following the instructions at https://getdoks.org/docs/start-here/getting-started/

And it runs fine on my Gentoo box with Hugo v0.125.4+extended linux/amd64 installed with Gentoo package manager.

The step

$ npm install

added 348 packages, and audited 349 packages in 2m

48 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

takes about one to two minutes, with a short stalling period of about 30 seconds with no graphical update, but that is fine. Much better than the eight minutes stall without update before.

And the advantage of use of global Hugo is, that the created main directory is now only 134 MB instead of more than 300 MB as before.

I will continue my install in the next few days. I hope I will get the deploy to website-hoster all-inkl.com working again -- initially one year ago, I had some issues, and i can not remember the details.

[StefanSalewski]

Unfortunately the slow install still can occur. I just started with updating my other site, and got the following static message display for about six minutes:

$ npm install
(#########⠂⠂⠂⠂⠂⠂⠂⠂⠂) ⠇ idealTree:yargs: timing idealTree:node_modules/glob-all/node_modules/yargs Completed in 114ms

as a result of

✔ Project name: … nimprogrammingbook.com

Scaffolding project in /home/salewski/nimprogrammingbook.com...

Done. Now run:

  cd nimprogrammingbook.com
  npm install
  npm run dev

salewski@hx90 ~ $ cd nimprogrammingbook.com
salewski@hx90 ~/nimprogrammingbook.com $ npm install

added 348 packages in 7m

48 packages are looking for funding
  run `npm fund` for details

Some people might press CTRL-C in that situation. Two days ago the initial install was very fast, so I assume there is some server involved which is sometimes very slow.

[StefanSalewski]

2024-05-14T16_40_45_664Z-debug-0.log

I have attached the build log.

[StefanSalewski]

Additional, I tried to delete the .npm directory:

rm -rf .npm/

$ rm -rf nimprogrammingbook.com
salewski@hx90 ~ $ npm create hyas@latest -- --template doks
Need to install the following packages:
create-hyas@1.5.0
Ok to proceed? (y) yes
✔ Project name: … nimprogrammingbook.com

Scaffolding project in /home/salewski/nimprogrammingbook.com...

Done. Now run:

  cd nimprogrammingbook.com
  npm install
  npm run dev

npm notice 
npm notice New minor version of npm available! 10.5.0 -> 10.7.0
npm notice Changelog: https://github.com/npm/cli/releases/tag/v10.7.0
npm notice Run npm install -g npm@10.7.0 to update!
npm notice 
salewski@hx90 ~ $ cd nimprogrammingbook.com
salewski@hx90 ~/nimprogrammingbook.com $ npm install

added 348 packages in 7m

48 packages are looking for funding
  run `npm fund` for details

But that made no difference, it took seven minutes in total again.

[StefanSalewski]

Surprisingly, two $ npm install calls done with only 30 minutes time difference results in different directory structures:

$ du -h -d1 nimprogrammingbook.com
8.0K    nimprogrammingbook.com/.vscode
72K nimprogrammingbook.com/assets
48K nimprogrammingbook.com/config
40K nimprogrammingbook.com/content
16K nimprogrammingbook.com/layouts
0   nimprogrammingbook.com/static
130M    nimprogrammingbook.com/node_modules
130M    nimprogrammingbook.com
salewski@hx90 ~ $ du -h -d1 nimprogrammingbook.bak
8.0K    nimprogrammingbook.bak/.vscode
72K nimprogrammingbook.bak/assets
48K nimprogrammingbook.bak/config
40K nimprogrammingbook.bak/content
16K nimprogrammingbook.bak/layouts
0   nimprogrammingbook.bak/static
139M    nimprogrammingbook.bak/node_modules
140M    nimprogrammingbook.bak

And for the initial install the size was 134 MB.

[h-enk]

It looks like it's got something to do with your Gentoo setup (you're Node and npm versions look OK though).

BTW I've no issues on my Manjaro machine and WSL

[h-enk]

Maybe you could try using a lower version of Node.js and npm?

https://github.com/npm/cli/issues/4028

[StefanSalewski]

Thanks for that hint. I will try a different node version.

[StefanSalewski]

I just tried the latest nodejs-22.1.0, but install takes again 7 minutes. It is a bit strange, but no real problem for me.

$ npm version
{
  npm: '10.7.0',
  node: '22.1.0',
  acorn: '8.11.3',
  ada: '2.7.8',
  ares: '1.26.0',
  base64: '0.5.2',
  brotli: '1.1.0',
  cjs_module_lexer: '1.2.2',
  cldr: '44.1',
  icu: '74.2',
  llhttp: '9.2.1',
  modules: '127',
  napi: '9',
  nghttp2: '1.61.0',
  openssl: '3.0.13',
  simdjson: '3.9.1',
  simdutf: '5.2.4',
  tz: '2023c',
  undici: '6.13.0',
  unicode: '15.1',
  uv: '1.48.0',
  uvwasi: '0.0.20',
  v8: '12.4.254.14-node.11',
  zlib: '1.3.1'
}
salewski@hx90 ~ $ emerge -pv nodejs

These are the packages that would be merged, in order:

Calculating dependencies... done!
Dependency resolution took 1.75 s (backtrack: 0/20).

[ebuild   R   ~] net-libs/nodejs-22.1.0:0/22::gentoo  USE="icu inspector npm snapshot ssl system-icu system-ssl -corepack -debug -doc (-lto) -pax-kernel -test" CPU_FLAGS_X86="sse2" 0 KiB