Closed dawu415 closed 5 years ago
Is it possible that your environment blocked some npm repos? The log comes from the nodejs-buildpack. Seems it cuts some logs by "Installing any new modules (package.json)". Could you have a try "cf ssh" and manually "npm install" inside the app container?
Hi Zhongyi-Zhang,
We cannot cf ssh in, because the app still has not started. However, if we bosh ssh into the diego cell and curl https://registry.npmjs.org, the registry is blocked.
When we checked the meta-azure-servicebroker.zip, we did notice that there were node_modules
already vendored. Wouldn't the Azure service broker already have these vendored dependencies and not go to npm repo's to pull more things in?
Just to add the meta-azure-servicebroker.zip
was copied from the errand VM deploy_all
that is run after the service broker tile is installed.
Could there be something missing in the node_modules? Using the existing files from meta-azure-servicebroker.zip, we ran backed up the node_modules directory and ran npm install on our local machines and did a diff of the node modules and the back up directory.
There's around 200 modules missing.
Only in node_modules/.bin: _mocha
Only in node_modules/.bin: acorn
Only in node_modules/.bin: cdl
Only in node_modules/.bin: escodegen
Only in node_modules/.bin: esgenerate
Only in node_modules/.bin: eslint
Only in node_modules/.bin: esparse
Only in node_modules/.bin: esvalidate
Only in node_modules/.bin: handlebars
Only in node_modules/.bin: istanbul
Only in node_modules/.bin: js-yaml
Only in node_modules/.bin: jshint
Only in node_modules/.bin: mocha
Only in node_modules/.bin: shjs
Only in node_modules/.bin: uglifyjs
Only in node_modules: acorn
Only in node_modules: acorn-jsx
Only in node_modules: ajv
Only in node_modules: ajv-keywords
Only in node_modules: amdefine
Only in node_modules: ansi-escapes
Only in node_modules: ansicolors
Only in node_modules: argparse
Only in node_modules: assertion-error
Only in node_modules: aws-sign
Only in node_modules: azure
Only in node_modules: azure-arm-authorization
Only in node_modules: azure-arm-compute
Only in node_modules: azure-arm-dns
Only in node_modules: azure-arm-insights
Only in node_modules: azure-arm-keyvault
Only in node_modules: azure-arm-network
Only in node_modules: azure-arm-storage
Only in node_modules: azure-arm-website
Only in node_modules: azure-asm-compute
Only in node_modules: azure-asm-hdinsight
Only in node_modules: azure-asm-mgmt
Only in node_modules: azure-asm-network
Only in node_modules: azure-asm-sb
Only in node_modules: azure-asm-scheduler
Only in node_modules: azure-asm-sql
Only in node_modules: azure-asm-storage
Only in node_modules: azure-asm-store
Only in node_modules: azure-asm-subscription
Only in node_modules: azure-asm-trafficmanager
Only in node_modules: azure-asm-website
Only in node_modules: azure-common
Only in node_modules: azure-gallery
Only in node_modules: azure-keyvault
Only in node_modules: azure-monitoring
Only in node_modules: azure-sb
Only in node_modules: azure-scheduler
Only in node_modules: azure-storage
Only in node_modules: babel-code-frame
Only in node_modules: babel-runtime
Only in node_modules: browser-stdout
Only in node_modules: browserify-mime
Only in node_modules: bson
Only in node_modules: buffer-from
Only in node_modules: buffer-shims
Only in node_modules: buffer-writer
Only in node_modules: caller-id
Only in node_modules: caller-path
Only in node_modules: callsites
Only in node_modules: cardinal
Only in node_modules: chai
Only in node_modules: chai-http
Only in node_modules: circular-json
Only in node_modules: cli
Only in node_modules: cli-cursor
Only in node_modules: cli-width
Only in node_modules: co
Only in node_modules: code-point-at
Only in node_modules: compare-version
Only in node_modules: concat-stream
Only in node_modules: console-browserify
Only in node_modules: cookie-jar
Only in node_modules: cookiejar
Only in node_modules: core-js
Only in node_modules: d
Only in node_modules: date-now
Only in node_modules: dateformat
Only in node_modules: deep-eql
Only in node_modules: deep-is
Only in node_modules: denque
Only in node_modules: dns-lookup-all
Only in node_modules: doctrine
Only in node_modules: documentdb
Only in node_modules: dom-serializer
Only in node_modules: domelementtype
Only in node_modules: domhandler
Only in node_modules: domutils
Only in node_modules: double-ended-queue
Only in node_modules: entities
Only in node_modules: envconf
Only in node_modules: es5-ext
Only in node_modules: es6-iterator
Only in node_modules: es6-map
Only in node_modules: es6-promise
Only in node_modules: es6-set
Only in node_modules: es6-symbol
Only in node_modules: es6-weak-map
Only in node_modules: escodegen
Only in node_modules: escope
Only in node_modules: eslint
Only in node_modules: espree
Only in node_modules: esprima
Only in node_modules: esrecurse
Only in node_modules: estraverse
Only in node_modules: esutils
Only in node_modules: event-emitter
Only in node_modules: eventhubs-js
Only in node_modules: exit
Only in node_modules: exit-hook
Only in node_modules: fast-levenshtein
Only in node_modules: figures
Only in node_modules: file-entry-cache
Only in node_modules: flat-cache
Only in node_modules: formatio
Only in node_modules: fs.realpath
Only in node_modules: fsevents
Only in node_modules: globals
Only in node_modules: graceful-readlink
Only in node_modules: growl
Only in node_modules: handlebars
Only in node_modules: htmlparser2
Only in node_modules: ignore
Only in node_modules: inquirer
Only in node_modules: interpret
Only in node_modules: ip-regex
Only in node_modules: is-ip
Only in node_modules: is-resolvable
Only in node_modules: istanbul
Only in node_modules: js-string-escape
Only in node_modules: js-tokens
Only in node_modules: js-yaml
Only in node_modules: jshint
Only in node_modules: json-stable-stringify
Only in node_modules: json3
Only in node_modules: jsonify
Only in node_modules: levn
Only in node_modules: lodash._baseassign
Only in node_modules: lodash._basecopy
Only in node_modules: lodash._basecreate
Only in node_modules: lodash._getnative
Only in node_modules: lodash._isiterateecall
Only in node_modules: lodash.create
Only in node_modules: lodash.isarguments
Only in node_modules: lodash.isarray
Only in node_modules: lodash.keys
Only in node_modules: lolex
Only in node_modules: long
Only in node_modules: methods
Only in node_modules: mocha
Only in node_modules: mock-require
Only in node_modules: mongodb
Only in node_modules: mongodb-core
Only in node_modules: mpns
Only in node_modules: mute-stream
Only in node_modules: mysql2
Only in node_modules: named-placeholders
Only in node_modules: natural-compare
Only in node_modules: next-tick
Only in node_modules: node-uuid
Only in node_modules: number-is-nan
Only in node_modules: object-assign
Only in node_modules: onetime
Only in node_modules: optimist
Only in node_modules: optionator
Only in node_modules: os-homedir
Only in node_modules: packet-reader
Only in node_modules: path-parse
Only in node_modules: pg
Only in node_modules: pg-connection-string
Only in node_modules: pg-int8
Only in node_modules: pg-pool
Only in node_modules: pg-types
Only in node_modules: pgpass
Only in node_modules: pluralize
Only in node_modules: postgres-array
Only in node_modules: postgres-bytea
Only in node_modules: postgres-date
Only in node_modules: postgres-interval
Only in node_modules: prelude-ls
Only in node_modules: progress
Only in node_modules: promise
Only in node_modules: q
Only in node_modules: readline2
Only in node_modules: rechoir
Only in node_modules: redeyed
Only in node_modules: redis
Only in node_modules: redis-commands
Only in node_modules: redis-parser
Only in node_modules: regenerator-runtime
Only in node_modules: require-uncached
Only in node_modules: require_optional
Only in node_modules: resolve
Only in node_modules: resolve-from
Only in node_modules: restore-cursor
Only in node_modules: run-async
Only in node_modules: rx-lite
Only in node_modules: samsam
Only in node_modules: sax
Only in node_modules: seq-queue
Only in node_modules: shelljs
Only in node_modules: should
Only in node_modules: should-equal
Only in node_modules: should-format
Only in node_modules: should-type
Only in node_modules: should-type-adaptors
Only in node_modules: should-util
Only in node_modules: sigmund
Only in node_modules: sinon
Only in node_modules: slice-ansi
Only in node_modules: sprintf-js
Only in node_modules: sqlstring
Only in node_modules: superagent
Only in node_modules: table
Only in node_modules: tedious
Only in node_modules: text-table
Only in node_modules: type-check
Only in node_modules: type-detect
Only in node_modules: typedarray
Only in node_modules: uglify-js
Only in node_modules: user-home
Only in node_modules: util
Only in node_modules: validator
Only in node_modules: wns
Only in node_modules: wordwrap
Only in node_modules: write
Only in node_modules: xml2js
Only in node_modules: xmlbuilder
Would it be possible to package a tile and service broker app that has all the dependencies?
The packages in above list should be all dev dependencies only. But yeah you remind me that MASB vendors all the prod dependencies to make it work in strict network environment. (You can have a try npm install --production
. No more packages would be installed.)
And I took a look at the history logs from the tile test on PCF 2.2 in our pipeline. Compared to yours, it has some extra info.
Starting app azure-service-broker-1.10.0 in org system / space azure-service-broker-space as system_services...
Staging app and tracing logs...
Downloading nodejs_buildpack...
Downloaded nodejs_buildpack (136.1M)
Cell 1d34108f-128b-4a82-a0dc-539d5510a820 creating container for instance a18ea117-7f0c-488a-853b-8547fcb9ab7c
Cell 1d34108f-128b-4a82-a0dc-539d5510a820 successfully created container for instance a18ea117-7f0c-488a-853b-8547fcb9ab7c
Downloading app package...
Downloaded app package (14.9M)
-----> Nodejs Buildpack version 1.6.32
-----> Installing binaries
engines.node (package.json): >= 6.10.0
engines.npm (package.json): unspecified (use default)
[31;1m**WARNING**[0m Dangerous semver range (>) in engines.node. See: http://docs.cloudfoundry.org/buildpacks/node/node-tips.html
-----> Installing node 10.10.0
Copy [/tmp/buildpacks/89ae4960344a0bd8ebceaeddee570d8f/dependencies/653e5cef05268f6fecf83d474d5a10e0/node-10.10.0-linux-x64-cflinuxfs2-9b4303cf.tgz]
Using default npm version: 6.4.1
-----> Installing yarn 1.9.4
Copy [/tmp/buildpacks/89ae4960344a0bd8ebceaeddee570d8f/dependencies/2d10153dc84bbc49dedac3162db062ff/yarn-v1.9.4-7667eb71.tar.gz]
Installed yarn 1.9.4
-----> Creating runtime environment
NODE_ENV=production
NODE_HOME=/tmp/contents844347969/deps/0/node
NODE_MODULES_CACHE=true
NODE_VERBOSE=false
NPM_CONFIG_LOGLEVEL=error
NPM_CONFIG_PRODUCTION=true
-----> Building dependencies
Prebuild detected (node_modules already exists)
Rebuilding any native modules
> nodemon@1.17.5 postinstall /tmp/app/node_modules/nodemon
> node bin/postinstall || exit 0
[32mLove nodemon? You can now support the project via the open collective:[22m[39m
> [96m[1mhttps://opencollective.com/nodemon/donate[0m
> dtrace-provider@0.8.7 install /tmp/app/node_modules/bunyan/node_modules/dtrace-provider
> node-gyp rebuild || node suppress-error.js
make: Entering directory `/tmp/app/node_modules/bunyan/node_modules/dtrace-provider/build'
TOUCH Release/obj.target/DTraceProviderStub.stamp
make: Leaving directory `/tmp/app/node_modules/bunyan/node_modules/dtrace-provider/build'
> dtrace-provider@0.6.0 install /tmp/app/node_modules/dtrace-provider
> node scripts/install.js
@types/node@7.0.66 /tmp/app/node_modules/@types/node
…...
For example this part:
-----> Building dependencies
Prebuild detected (node_modules already exists)
Rebuilding any native modules
It is expected that the buildpack only does rebuilding. But it can't be found in your logs. Any idea? BTW, the installation logs in OpsMan might be more informative.
Yes, sorry, I only attached the snippets after, it is the same as what you had posted.
cf start azure-service-broker-1.10.0
Starting app azure-service-broker-1.10.0 in org system / space azure-service-broker-space as system_services...
Staging app and tracing logs...
Downloading nodejs_buildpack...
Downloaded nodejs_buildpack
Cell 1e83e61a-dfd7-41b6-96e8-45604f2cc254 creating container for instance 61e0d888-edf7-4b04-925f-e9b121aa7ae2
Cell 1e83e61a-dfd7-41b6-96e8-45604f2cc254 successfully created container for instance 61e0d888-edf7-4b04-925f-e9b121aa7ae2
Downloading app package...
Downloaded app package (14.7M)
-----> Nodejs Buildpack version 1.6.32
-----> Installing binaries
engines.node (package.json): >= 6.10.0
engines.npm (package.json): unspecified (use default)
[31;1m**WARNING**[0m Dangerous semver range (>) in engines.node. See: http://docs.cloudfoundry.org/buildpacks/node/node-tips.html
-----> Installing node 10.10.0
Copy [/tmp/buildpacks/206c8132ecb388618c7c963af6b1449c/dependencies/653e5cef05268f6fecf83d474d5a10e0/node-10.10.0-linux-x64-cflinuxfs2-9b4303cf.tgz]
Using default npm version: 6.4.1
-----> Installing yarn 1.9.4
Copy [/tmp/buildpacks/206c8132ecb388618c7c963af6b1449c/dependencies/2d10153dc84bbc49dedac3162db062ff/yarn-v1.9.4-7667eb71.tar.gz]
Installed yarn 1.9.4
-----> Creating runtime environment
NODE_ENV=production
NODE_HOME=/tmp/contents097379234/deps/0/node
NODE_MODULES_CACHE=true
NODE_VERBOSE=false
NPM_CONFIG_LOGLEVEL=error
NPM_CONFIG_PRODUCTION=true
-----> Building dependencies
Prebuild detected (node_modules already exists)
Rebuilding any native modules
> nodemon@1.17.5 postinstall /tmp/app/node_modules/nodemon
> node bin/postinstall || exit 0
[32mLove nodemon? You can now support the project via the open collective:[22m[39m
> [96m[1mhttps://opencollective.com/nodemon/donate[0m
> dtrace-provider@0.8.7 install /tmp/app/node_modules/bunyan/node_modules/dtrace-provider
> node-gyp rebuild || node suppress-error.js
make: Entering directory `/tmp/app/node_modules/bunyan/node_modules/dtrace-provider/build'
TOUCH Release/obj.target/DTraceProviderStub.stamp
make: Leaving directory `/tmp/app/node_modules/bunyan/node_modules/dtrace-provider/build'
> dtrace-provider@0.6.0 install /tmp/app/node_modules/dtrace-provider
> node scripts/install.js
@types/node@7.0.66 /tmp/app/node_modules/@types/node
Thanks for the suggestion. We reset the node_modules
to the original and ran npm install --production
and now we've only ended up with one difference fsevents
. The broker app can be pushed successfully with the fsevents
.
Seems odd that it is missing.
Yes, fsevents
is optional and supports darwin only. Non-darwin OS gets something like unless using --no-optional
for npm install
:
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.2.2 (node_modules\chokidar\node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.4: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})
So it is not vendored. Maybe the buildpack still had a try to install it, so you hit the timeout issue?
Yeah, this is weird. We've tried what you just suggested npm install --production --no-optional
and indeed it does not add the optional fsevent
. We tried pushing again (in case it's some other file or state) and it is stuck on that line.
Yeah, maybe the buildpack is trying to install some optional component and is reaching out? Is there anyway to avoid the optionals?
Could you have a try npm install --production --no-optional --package-lock
(requires npm version >= 5)? If it works for you, I'll create a PR soon and publish a patch version of the tile.
It didn't work and is still timing out at the same point. It is now reading Installing any new modules (package.json + package-lock.json)
. Using npm version 6.4.1
Weird, it is what the buildpack doc said https://docs.cloudfoundry.org/buildpacks/node/index.html#vendoring.
And it is supported in nodejs buildpack v1.6.21 and later, according to https://github.com/cloudfoundry/nodejs-buildpack/blame/b234a4553c4e680d488b54e05dbf0e967454d575/src/nodejs/npm/npm.go#L66. Since your log shows that it did leverage package-lock.json
, it shouldn't be about buildpack version. Maybe you can also try npm shrinkwrap
?
BTW, it is expected to get stuck at that line for a while, but not timeout. Could you wait a little longer?
I tried fresh node_modules and removed package-lock.json again and ran the npm shrinkwrap. CF pushed and waited. The same point had timed out. I'm going to try the shrinkwrap along with the previous suggestion with the --package-lock, CF push and wait. :)
On Wed, Dec 5, 2018, 5:26 PM Zhongyi Zhang notifications@github.com wrote:
Weird, it is what the buildpack doc said https://docs.cloudfoundry.org/buildpacks/node/index.html#vendoring. And it is supported in nodejs buildpack v1.6.21 and later, according to https://github.com/cloudfoundry/nodejs-buildpack/blame/b234a4553c4e680d488b54e05dbf0e967454d575/src/nodejs/npm/npm.go#L66. Since your log shows that it did leverage package-lock.json, it shouldn't be about buildpack version. Maybe you can also try npm shrinkwrap?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Azure/meta-azure-service-broker/issues/198#issuecomment-444417261, or mute the thread https://github.com/notifications/unsubscribe-auth/AYQjVmgyi35-x7TKf9ARVybfl-A035eBks5u15EpgaJpZM4ZBFKS .
@zhongyi-zhang Tried the above and it failed with the 15-minute timeout as well.
urlencode@1.1.0 /tmp/app/node_modules/urlencode
inflight@1.0.6 /tmp/app/node_modules/inflight
Installing any new modules (package.json + npm-shrinkwrap.json)
Error staging application meta-azure-service-broker-1: timed out after 15 minutes
FAILED
Okay, I looked into the buildpack source code and located the issue here.
And according to you logs, the rebuilding succeeded. Then the buildpack ran "npm install" so that it was going to install the missing dev dependencies what are not needed. Since your environment can't access npm registry, that led the timeout. I am not sure this is a bug or expected behavior. If it is expected, I think they don't really provide an offline mode -- what the doc said wouldn't work. The package-lock.json
can't prevent npm install
from network calls even ignoring --production
-- I can set NODE_ENV=production
for the tile -- then npm install
no longer installs dev dependencies -- but it would still try to install fsevent
-- network call.
Added the pull request above.
After hanging for about 15 minutes on
Installing any new modules (package.json)
, we receive the following error:Stderr Error staging application azure-service-broker-1.10.0: timed out after 15 minute(s)
We have tried this 4 times already but cannot seem to find the problem.
Some other notes - we have tried: 1) Downloading the 1.10.0 release here, fill in the manifest.yml and pushed it to the same foundation. It fails at the same location.
2) Downloading the 1.10.0 release here, fill in the manifest.yml, vendor the packages via npm install and pushed it to the same foundation. It works.
What is odd is that the tile's service broker app that gets pushed already has the vendored dependencies. We verified this by ssh'ing onto the deploy errand VM and checking the zip file.
Any ideas on what might be happening?
Part of the broker app log attached.