search

asyncapi / generator

Use your AsyncAPI definition to generate literally anything. Markdown documentation, Node.js code, HTML documentation, anything!
https://asyncapi.com/docs/tools/generator
Apache License 2.0
759 stars 211 forks source link

The generate does not install template modules correctly if local #391

Closed jonaslagoni closed 3 years ago

jonaslagoni commented 4 years ago

Describe the bug

If you use the generator with a local path for a template i.e. ag "./streetlight.json" "./ts-nats-template" it does install all the modules however it still says missing module when moving to the generation phase.

npm http fetch GET 304 https://registry.npmjs.org/is-url 489ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/lodash 503ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/quicktype-core 501ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/@asyncapi%2fgenerator-filters 1359ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/markdown-it 53ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/openapi-sampler 55ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/entities 40ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/json-pointer 52ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/foreach 45ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/collection-utils 66ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/js-base64 68ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/pluralize 82ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/browser-or-node 72ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/wordwrap 147ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/urijs 149ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/pako 153ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/readable-stream 152ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/yaml 156ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/isomorphic-fetch 158ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/@mark.probst%2funicode-properties 936ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/unicode-trie 80ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/brfs 92ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/resolve 62ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/quote-stream 97ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/static-module 100ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/through2 100ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/minimist 51ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/buffer-equal 57ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/xtend 51ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/inherits 102ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/isarray 107ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/string_decoder 108ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/process-nextick-args 115ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/safe-buffer 117ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/core-util-is 124ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/util-deprecate 123ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/path-parse 44ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/convert-source-map 205ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/concat-stream 209ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/duplexer2 220ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/has 236ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/escodegen 258ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/falafel 265ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/merge-source-map 265ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/magic-string 268ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/object-inspect 290ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/shallow-copy 261ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/static-eval 99ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/buffer-from 63ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/typedarray 67ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/esprima 133ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/estraverse 142ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/esutils 145ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/source-map 144ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/optionator 149ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/word-wrap 80ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/type-check 82ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/deep-is 85ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/levn 85ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/fast-levenshtein 87ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/prelude-ls 96ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/acorn 49ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/object-keys 55ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/function-bind 44ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/vlq 39ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/tiny-inflate 45ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/node-fetch 55ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/whatwg-fetch 58ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/is-stream 40ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/encoding 54ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/iconv-lite 54ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/safer-buffer 42ms (from cache)
+ @asyncapi/ts-nats-template@0.1.0
added 80 packages from 98 contributors and updated 1 package in 6.647s

3 packages are looking for funding
  run `npm fund` for details

Something went wrong:
Error: Cannot find module 'lodash'
Require stack:
- ./ts-nats-template\filters\all.js
- ./ts-nats-template\hooks\filename-change.js
- AppData\Roaming\npm\node_modules\@asyncapi\generator\lib\hooksRegistry.js
- AppData\Roaming\npm\node_modules\@asyncapi\generator\lib\generator.js
- AppData\Roaming\npm\node_modules\@asyncapi\generator\cli.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:980:15)
    at Function.Module._load (internal/modules/cjs/loader.js:862:27)
    at Module.require (internal/modules/cjs/loader.js:1040:19)
    at require (internal/modules/cjs/helpers.js:72:18)
    at Object.<anonymous> (./ts-nats-template\filters\all.js:2:11)
    at Module._compile (internal/modules/cjs/loader.js:1151:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
    at Module.load (internal/modules/cjs/loader.js:1000:32)
    at Function.Module._load (internal/modules/cjs/loader.js:899:14)
    at Module.require (internal/modules/cjs/loader.js:1040:19)
    at require (internal/modules/cjs/helpers.js:72:18)
    at Object.<anonymous> (./ts-nats-template\hooks\filename-change.js:1:17)
    at Module._compile (internal/modules/cjs/loader.js:1151:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
    at Module.load (internal/modules/cjs/loader.js:1000:32)
    at Function.Module._load (internal/modules/cjs/loader.js:899:14)

How to Reproduce

  1. Clone any template locally
  2. Run the command `ag "[some AsyncAPI document]" "[some local path to template]"

To solve manually:

  1. Clone any template locally
  2. go into the template and manually run the command npm i
  3. Run the command `ag "[some AsyncAPI document]" "[some local path to template]" Which works.

Expected behavior

I would expect the generate to correctly install the node_modules and be able to generate correctly.

derberg commented 4 years ago

@jonaslagoni I tried and cannot reproduce this one. Had local HTML template, removed its node-modules and then ran generator both, from local sources and with ag installed and all is fine. I'll see how it goes with docker

derberg commented 4 years ago

ok, got it with fresh env through docker:

npm http fetch GET 200 https://registry.npmjs.org/@asyncapi%2fgenerator-filters 2204ms
npm http fetch GET 200 https://registry.npmjs.org/@asyncapi/generator-filters/-/generator-filters-1.0.0.tgz 1722ms
npm http fetch GET 200 https://registry.npmjs.org/markdown-it 117ms
npm http fetch GET 200 https://registry.npmjs.org/markdown-it/-/markdown-it-10.0.0.tgz 22ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/openapi-sampler 265ms
npm http fetch GET 200 https://registry.npmjs.org/openapi-sampler/-/openapi-sampler-1.0.0-beta.16.tgz 119ms
npm http fetch GET 200 https://registry.npmjs.org/entities 95ms
npm http fetch GET 200 https://registry.npmjs.org/entities/-/entities-2.0.3.tgz 133ms
npm http fetch GET 200 https://registry.npmjs.org/json-pointer 96ms
npm http fetch GET 200 https://registry.npmjs.org/json-pointer/-/json-pointer-0.6.0.tgz 82ms
npm http fetch GET 200 https://registry.npmjs.org/foreach 92ms
npm http fetch GET 200 https://registry.npmjs.org/foreach/-/foreach-2.0.5.tgz 91ms
npm WARN eslint-plugin-sonarjs@0.5.0 requires a peer of eslint@^3.0.0 || ^4.0.0 || ^5.0.0 || ^6.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN tsutils@3.17.1 requires a peer of typescript@>=2.8.0 || >= 3.2.0-dev || >= 3.3.0-dev || >= 3.4.0-dev || >= 3.5.0-dev || >= 3.6.0-dev || >= 3.6.0-beta || >= 3.7.0-dev || >= 3.7.0-beta but none is installed. You must install peer dependencies yourself.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.1.2 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.2: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

+ @asyncapi/html-template@0.9.0
added 7 packages from 7 contributors in 15.595s

76 packages are looking for funding
  run `npm fund` for details

Something went wrong:
Error: Cannot find module '/app/node_modules/@asyncapi/html-template/node_modules/@asyncapi/generator-filters'
Require stack:
- /app/lib/filtersRegistry.js
- /app/lib/generator.js
- /app/cli.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:965:15)
    at Function.Module._load (internal/modules/cjs/loader.js:841:27)
    at Module.require (internal/modules/cjs/loader.js:1025:19)
    at require (internal/modules/cjs/helpers.js:72:18)
    at /app/lib/filtersRegistry.js:74:17
    at async Promise.all (index 0)
    at async registerConfigFilters (/app/lib/filtersRegistry.js:79:3)
    at async module.exports.registerFilters (/app/lib/filtersRegistry.js:15:3)
    at async Generator.generate (/app/lib/generator.js:173:5)
    at async /app/cli.js:151:9
derberg commented 4 years ago

From what I've learned so far error Cannot find module '/app/node_modules/@asyncapi/html-template/node_modules/@asyncapi/generator-filters' is there because the dependency is installed here /app/node_modules/@asyncapi/generator-filters which kind of makes sense as in docker the cli.js runs inside generator/app dir and all deps are installed there...

derberg commented 4 years ago

In docker if I change the location where I run cli.js, or in other words, when I change the location of the template, to not have it inside the generator folder, then all works file. So the issue is only if the template is inside the generator, then the installation is done in the generator node_modules instead of the template node_modules

this fails:

docker run -it \
-v ${PWD}:/app/generator \
-v ${PWD}/../html-template:/app/html-template \
-v ${PWD}/test/docs/dummy.yml:/app/asyncapi.yml \
-v ${PWD}/output:/app/output \
asyncapi/generator -o ./output asyncapi.yml ./html-template --force-write

this is ok:

docker run -it \
-v ${PWD}:/app/generator \
-v ${PWD}/../html-template:/snap/html-template \
-v ${PWD}/test/docs/dummy.yml:/app/asyncapi.yml \
-v ${PWD}/output:/app/output \
asyncapi/generator -o ./output asyncapi.yml ../snap/html-template --force-write

and the difference is that I'm mounting html-template outside app where generator sources are located

derberg commented 4 years ago

@jonaslagoni I investigated it a lot and I can now confirm that it is simply not possible to have it working when the template is inside the generator because this is how npm works:

source: https://www.stackchief.com/tutorials/npm%20install%20%7C%20how%20it%20works

npm install (directory) npm install myfolder When a folder argument is specified, npm install downloads everything from the specified location to the original node_modules directory at the root of the project. This assumes a myfolder sub-directory exists in the same location as package.json. Additionally, the myfolder has it's own package.json npm install WILL NOT generate a new node_modules folder inside the sub-directory. Instead, the dependencies are hoisted to the root directory's node_modules folder.

So basically when you do ./ts-nats-template all deps are installed but in the root of generator's node-modules I do not see a way to solve it differently than making README explain it well. We could also add something like this to the CLI if (err.stack.search('Cannot find module')) console.error(red('Make sure your template is not located inside the generator directory'));

Thoughts?

derberg commented 4 years ago

Will have another look into it after the open meeting as it seems the issue is also with global generator installation.

derberg commented 4 years ago

@jonaslagoni I have no idea how I can help you as I'm not able to reproduce it with ag. I've setup super fresh installation on my local with nvm (I'm using it to easily switch between node versions) and it works as expected with local template that is not installed manually. I also tried it wit docker and all works fine

$ docker run --rm -it \
> -v ${PWD}/../html-template:/templates/html-template \
> -v ${PWD}/test/docs/dummy.yml:/app/asyncapi.yml \
> -v ${PWD}/output:/app/output \
> --entrypoint sh \
> asyncapi/generator
/app # ag asyncapi.yml ../templates/html-template -o output
sh: ag: not found
/app # npm install @asyncapi/generator -g
npm WARN deprecated mkdirp@0.3.5: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
/usr/local/bin/asyncapi-generator -> /usr/local/lib/node_modules/@asyncapi/generator/cli.js
/usr/local/bin/ag -> /usr/local/lib/node_modules/@asyncapi/generator/cli.js
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@~2.1.2 (node_modules/@asyncapi/generator/node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

+ @asyncapi/generator@1.0.0-rc.9
added 151 packages from 107 contributors in 16.779s
/app # ag asyncapi.yml ../templates/html-template -o output
Something went wrong:
Error: "/app/output" is not an empty directory. You might override your work. To skip this rule, please make your code a git repository or use the --force-write flag (not recommended).
    at Generator.verifyTargetDir (/usr/local/lib/node_modules/@asyncapi/generator/lib/generator.js:781:30)
    at async Generator.generate (/usr/local/lib/node_modules/@asyncapi/generator/lib/generator.js:158:29)
    at async /usr/local/lib/node_modules/@asyncapi/generator/cli.js:149:9
/app # ag asyncapi.yml ../templates/html-template -o output --force-write

npm http fetch GET 200 https://registry.npmjs.org/@asyncapi%2fgenerator-filters 1127ms
npm http fetch GET 200 https://registry.npmjs.org/@asyncapi/generator-filters/-/generator-filters-1.0.0.tgz 1040ms
npm http fetch GET 200 https://registry.npmjs.org/openapi-sampler 106ms
npm http fetch GET 200 https://registry.npmjs.org/openapi-sampler/-/openapi-sampler-1.0.0-beta.16.tgz 94ms
npm http fetch GET 304 https://registry.npmjs.org/lodash 228ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/markdown-it 259ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/markdown-it/-/markdown-it-10.0.0.tgz 9ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/argparse 103ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/entities 110ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/linkify-it 112ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/entities/-/entities-2.0.3.tgz 103ms
npm http fetch GET 304 https://registry.npmjs.org/mdurl 234ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/uc.micro 255ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/sprintf-js 89ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/json-pointer 92ms
npm http fetch GET 200 https://registry.npmjs.org/json-pointer/-/json-pointer-0.6.0.tgz 128ms
npm http fetch GET 200 https://registry.npmjs.org/foreach 106ms
npm http fetch GET 200 https://registry.npmjs.org/foreach/-/foreach-2.0.5.tgz 91ms
npm http fetch GET 200 https://registry.npmjs.org/fsevents 3ms (from cache)
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@~2.1.2 (node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

+ @asyncapi/html-template@0.9.0
added 13 packages from 12 contributors in 6.966s

12 packages are looking for funding
  run `npm fund` for details

Done! ✨
Check out your shiny new generated files at /app/output.

maybe this is some windows thingy? so access rights issues? maybe try troubleshooting npm? including reinstallation?

jonaslagoni commented 4 years ago

it works as expected with local template that is not installed manually

So it is a fresh copy of a template with no node_modules folder inside? I just tried with a fresh install of the generator and still have the issue. But yea, might be windows 🤔

jonaslagoni commented 4 years ago

hmm, I get some other error now, let me investigate a bit.

jonaslagoni commented 4 years ago

Nope, still get the same error when trying to generate from a template with no node_modules directory.

derberg commented 4 years ago

and what about if you use generator from sources? like /.cli.js?

jonaslagoni commented 4 years ago

Tried to download the html template and got a really really weird result:

$ ag ./asyncapi.json ./html-template-master/ -o output
npm http fetch GET 304 https://registry.npmjs.org/@asyncapi%2fgenerator-filters 1390ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/lodash 61ms
npm http fetch GET 304 https://registry.npmjs.org/markdown-it 99ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/openapi-sampler 107ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/mdurl 16ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/argparse 20ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/uc.micro 18ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/entities 52ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/linkify-it 56ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/sprintf-js 5ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/json-pointer 5ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/foreach 4ms (from cache)
Something went wrong:
Error: ENOENT: no such file or directory, rename 'AppData\Roaming\npm\node_modules\@asyncapi\generator\node_modules\@asyncapi\dotnet-nats-template\node_modules\foreach' -> 'Downloads\New folder\html-template-master\node_modules\foreach'

Why does the previous used local template affect this generation? Got this error twice when trying to generate different things....

Tried running the command in a sub folder which works fine

$ ag ../Rust_API.json ../html-template-master/ -o output
npm http fetch GET 304 https://registry.npmjs.org/@asyncapi%2fgenerator-filters 510ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/openapi-sampler 62ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/markdown-it 107ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/lodash 125ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/argparse 25ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/mdurl 25ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/uc.micro 26ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/entities 56ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/linkify-it 61ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/sprintf-js 6ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/json-pointer 5ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/foreach 4ms (from cache)
+ @asyncapi/html-template@0.10.0
added 13 packages from 12 contributors in 2.561s

12 packages are looking for funding
  run `npm fund` for details

Done! ✨
Check out your shiny new generated files at Downloads\New folder\components\output.

Then I tried to go back to the parent folder and ran the same command again and got 

$ ag ./Rust_API.json ./html-template-master/ -o output
+ @asyncapi/html-template@0.10.0
updated 1 package in 0.946s

12 packages are looking for funding
  run `npm fund` for details

Done! ✨
Check out your shiny new generated files at Downloads\New folder\output.

So yea, now I am confused 😄

Then I tried deleting the node_modules dir in the template and ran it again:

$ ag ./Rust_API.json ./html-template-master/ -o output
npm http fetch GET 304 https://registry.npmjs.org/@asyncapi%2fgenerator-filters 1272ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/lodash 63ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/markdown-it 103ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/openapi-sampler 105ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/mdurl 23ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/argparse 26ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/uc.micro 25ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/entities 60ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/linkify-it 73ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/sprintf-js 5ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/json-pointer 4ms (from cache)
npm http fetch GET 200 https://registry.npmjs.org/foreach 4ms (from cache)
+ @asyncapi/html-template@0.10.0
added 12 packages from 11 contributors and updated 1 package in 2.825s

12 packages are looking for funding
  run `npm fund` for details

Done! ✨
Check out your shiny new generated files at Downloads\New folder\output.

Works again.. Huh.

Then I tried moving the entire template to another drive and still ran the command on the C drive and got the following error:

$ ag ./Rust_API.json D:\html-template-master -o output --install
Something went wrong:
Error: ENOENT: no such file or directory, open 'D:\html-template-master@latest\package.json'
    at Object.manifest (C:\Program Files\nodejs\node_modules\npm\node_modules\pacote\lib\fetchers\directory.js:39:7)
    at Object.Fetcher#manifest [as manifest] (C:\Program Files\nodejs\node_modules\npm\node_modules\genfun\lib\genfun.js:15:38)
    at manifest (C:\Program Files\nodejs\node_modules\npm\node_modules\pacote\lib\fetch.js:23:18)
    at C:\Program Files\nodejs\node_modules\npm\node_modules\pacote\manifest.js:24:12
    at C:\Program Files\nodejs\node_modules\npm\node_modules\promise-inflight\inflight.js:29:24
From previous event:
    at _inflight (C:\Program Files\nodejs\node_modules\npm\node_modules\promise-inflight\inflight.js:28:25)
    at C:\Program Files\nodejs\node_modules\npm\node_modules\promise-inflight\inflight.js:22:14
    at processImmediate (internal/timers.js:456:21)
From previous event:
    at inflight (C:\Program Files\nodejs\node_modules\npm\node_modules\promise-inflight\inflight.js:14:40)
    at Object.manifest (C:\Program Files\nodejs\node_modules\npm\node_modules\pacote\manifest.js:22:10)
    at fetchPackageMetadata (C:\Program Files\nodejs\node_modules\npm\lib\fetch-package-metadata.js:59:10)
    at callFunc (C:\Program Files\nodejs\node_modules\npm\node_modules\call-limit\call-limit.js:25:12)
    at limited (C:\Program Files\nodejs\node_modules\npm\node_modules\call-limit\call-limit.js:13:7)
    at C:\Program Files\nodejs\node_modules\npm\lib\install\deps.js:253:14
    at C:\Program Files\nodejs\node_modules\npm\node_modules\slide\lib\async-map.js:52:35
    at Array.forEach (<anonymous>)
    at C:\Program Files\nodejs\node_modules\npm\node_modules\slide\lib\async-map.js:52:11
    at Array.forEach (<anonymous>)
    at asyncMap (C:\Program Files\nodejs\node_modules\npm\node_modules\slide\lib\async-map.js:51:8)
    at exports.getAllMetadata (C:\Program Files\nodejs\node_modules\npm\lib\install\deps.js:224:3)
    at Installer.loadArgMetadata (C:\Program Files\nodejs\node_modules\npm\lib\install.js:366:3)
    at C:\Program Files\nodejs\node_modules\npm\lib\install.js:701:16
    at C:\Program Files\nodejs\node_modules\npm\lib\install\read-shrinkwrap.js:33:16
From previous event:
    at readShrinkwrap (C:\Program Files\nodejs\node_modules\npm\lib\install\read-shrinkwrap.js:33:5)
    at C:\Program Files\nodejs\node_modules\npm\lib\install.js:697:7
    at C:\Program Files\nodejs\node_modules\npm\node_modules\iferr\iferr.js:3:64
    at C:\Program Files\nodejs\node_modules\npm\node_modules\read-package-tree\rpt.js:162:20

Without install parameter it succeeds but generate no files.

$ ag ./Rust_API.json D:\html-template-master -o output 

Done! ✨
Check out your shiny new generated files at Downloads\New folder\output.

@derberg I think the next step is a clean windows machine and trying it on that 🤔

jonaslagoni commented 4 years ago

and what about if you use generator from sources? like /.cli.js?

You mean running the source files directly instead of the bin shortcuts?

derberg commented 4 years ago

yeap, get generator sources, npm i and just call the ./cli.js directly instead of ag.

Maybe first try to wipe out npm, get entirely fresh installation, and make sure you have the required one

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity :sleeping: It will be closed in 30 days if no further activity occurs. To unstale this issue, add a comment with detailed explanation. Thank you for your contributions :heart:

derberg commented 3 years ago

@jonaslagoni you might want to have a look at the scenarios I described here https://github.com/asyncapi/generator/issues/425#issuecomment-719633496. I think I was able to reproduce your issue, please check my scenarios and confirm the same behavior

jonaslagoni commented 3 years ago

Here is what I observed. Version node: v13.9.0 Windows 10 Pro version 10.0.17134 Build 17134 npm: 6.14.8

Did one after the other and deleted what was necessary.

  1. local html-template without node_modules and local generator without html-template in node_modules - works fine.
  2. local html-template without node_modules and local generator with html-template in node_modules - Deleted the node_modes after running command 1. and tried running it again. Works fine.
  3. local html-template with node_modules and local generator without html-template in node_modules - Deleted the html-template inside node_modules of the generator after running command 2. Works fine
  4. local html-template without node_modules and local generator without html-template in node_modules - This is the same as command 1? Works fine
  5. local html-template without node_modules and local generator with html-template in node_modules - Same as command 2? Works fine
  6. local html-template with node_modules and local generator without html-template in node_modules - Same as command 3? Works fine

Since none of these triggered any problems I tried the same 3 commands with the globally installed generator with same procedure version: @asyncapi/generator@1.0.0-rc.14

  1. local html-template without node_modules and local generator without html-template in node_modules - Works fine
  2. local html-template without node_modules and local generator with html-template in node_modules - Works fine
  3. local html-template with node_modules and local generator without html-template in node_modules - Works fine

@derberg I can honestly not reproduce the error in any way 😮 Do you have any idea if i'm doing it wrong or it simply has been fixed by one of the updates?

derberg commented 3 years ago

local html-template without node_modules and local generator with html-template in node_modules - Deleted the node_modes after running command 1. and tried running it again. Works fine.

No way man 🤯 I tried it again and still fails for me 😄 I'm on node 14, but I don't think node version has anything to do with it

brettle commented 3 years ago

Fwiw, I ran into what appears to be the same issue when trying to run the generator from a github action. Worked fine on linux-latest and macos-latest, but failed on windows-latest. However, the same code worked fine on my local Windows 10 box. In the end, I worked around the issue by creating a tar package of the local template directory and pointing the generator at that. I speculate that the problem might be due to npm's use of symlinks and the fact that github's runner put's the code on a drive other than c:.

When it runs:

D:\a\WPIWebSockets\WPIWebSockets\.gradle\npm\npm-v6.14.10\npx.cmd @asyncapi/generator --force-write -o D:\a\WPIWebSockets\WPIWebSockets\build/generated/sources/asyncapi/org/team199/wpiws/devices D:\a\WPIWebSockets\WPIWebSockets/asyncapi-template/wpilib-ws.yaml D:\a\WPIWebSockets\WPIWebSockets/asyncapi-template/

It produces the error:

Error: -02T07:50:04.905+0000 [ERROR] [system.err] Error: Cannot find module 'C:\npm\cache\_npx\6864\node_modules\@asyncapi\generator\node_modules\wpilib-schema-template\node_modules\@asyncapi\generator-filters'
Error: -02T07:50:04.905+0000 [ERROR] [system.err] Require stack:
Error: -02T07:50:04.905+0000 [ERROR] [system.err] - C:\npm\cache\_npx\6864\node_modules\@asyncapi\generator\lib\filtersRegistry.js
Error: -02T07:50:04.905+0000 [ERROR] [system.err] - C:\npm\cache\_npx\6864\node_modules\@asyncapi\generator\lib\generator.js
Error: -02T07:50:04.905+0000 [ERROR] [system.err] - C:\npm\cache\_npx\6864\node_modules\@asyncapi\generator\cli.js
Error: -02T07:50:04.905+0000 [ERROR] [system.err]     at Function.Module._resolveFilename (internal/modules/cjs/loader.js:880:15)
Error: -02T07:50:04.906+0000 [ERROR] [system.err]     at Function.Module._load (internal/modules/cjs/loader.js:725:27)
Error: -02T07:50:04.906+0000 [ERROR] [system.err]     at Module.require (internal/modules/cjs/loader.js:952:19)
Error: -02T07:50:04.906+0000 [ERROR] [system.err]     at require (internal/modules/cjs/helpers.js:88:18)
Error: -02T07:50:04.906+0000 [ERROR] [system.err]     at C:\npm\cache\_npx\6864\node_modules\@asyncapi\generator\lib\filtersRegistry.js:74:17
Error: -02T07:50:04.906+0000 [ERROR] [system.err]     at async Promise.all (index 0)
Error: -02T07:50:04.908+0000 [ERROR] [system.err]     at async registerConfigFilters (C:\npm\cache\_npx\6864\node_modules\@asyncapi\generator\lib\filtersRegistry.js:79:3)
Error: -02T07:50:04.908+0000 [ERROR] [system.err]     at async module.exports.registerFilters (C:\npm\cache\_npx\6864\node_modules\@asyncapi\generator\lib\filtersRegistry.js:15:3)
Error: -02T07:50:04.908+0000 [ERROR] [system.err]     at async Generator.generate (C:\npm\cache\_npx\6864\node_modules\@asyncapi\generator\lib\generator.js:178:7)
Error: -02T07:50:04.908+0000 [ERROR] [system.err]     at async C:\npm\cache\_npx\6864\node_modules\@asyncapi\generator\cli.js:130:9

Note that the npm cache is on C: but the template is on D:.

Unfortunately, I don't have any more time to investigate at the moment, but perhaps this will be useful to others.

derberg commented 3 years ago

@jonaslagoni I suggest we close this one. While working with #517 and seeing the same behavior with arborist, imho we should just accept how npm tools work. This means that if you refer local template with path, and this template was not initially installed with npm i (I mean there are no node_modules) then this is normal the generation fails as some modules/deps are missing. Otherwise, we would have to add some custom handler for such use cases and I doubt it makes sense really.

jonaslagoni commented 3 years ago

@derberg yes agreed, let's close it.