search

intel-iot-devkit / meta-iot-cloud

OpenEmbedded layer to add support for multiple cloud IoT services including Microsoft Azure & Google Cloud Platform.
MIT License
69 stars 80 forks source link

Node-RED v1.22 do_fetch takes very long/fails #107

Closed ghost closed 2 years ago

ghost commented 2 years ago

Hi,

Building node-red seems to be broken. I noticed in the log that resolving each npm dependency takes several minutes and in the end the whole process fails because of some other packages that could not be resolved at all. I'm not sure whats causing this because it built previously just fine.

My log: DEBUG: Executing python function extend_recipe_sysroot NOTE: Direct dependencies are ['virtual:native:/home/em/Dev/meta-openembedded/meta-oe/recipes-devtools/nodejs/nodejs_12.21.0.bb:do_populate_sysroot'] NOTE: Installed into sysroot: [] NOTE: Skipping as already exists in sysroot: ['nodejs-native', 'quilt-native', 'openssl-native', 'brotli-native', 'zlib-native', 'icu-native', 'c-ares-native', 'xz-native', 'pkgconfig-native', 'python3-native', 'ninja-native', 'cmake-native', 'libtool-native', 'unzip-native', 'automake-native', 'autoconf-native', 'gnu-config-native', 'gettext-minimal-native', 'libffi-native', 'sqlite3-native', 'libnsl2-native', 'readline-native', 'gdbm-native', 'bzip2-native', 'libtirpc-native', 'util-linux-native', 're2c-native', 'ncurses-native', 'curl-native', 'expat-native', 'texinfo-dummy-native', 'm4-native', 'libpcre2-native', 'libcap-ng-native'] DEBUG: Python function extend_recipe_sysroot finished DEBUG: Executing python function do_fetch DEBUG: Executing python function base_do_fetch DEBUG: Fetcher accessed the network with the command npm view node-red@1.2.2 DEBUG: Running 'export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export SSH_AGENT_PID="1386"; export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh"; export PATH="/home/em/Dev/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/usr/bin/python3-native:/home/em/Dev/poky/scripts:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot/usr/bin/crossscripts:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/usr/sbin:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/usr/bin:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/sbin:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/bin:/home/em/Dev/poky/bitbake/bin:/home/em/Dev/build/tmp/hosttools"; export HOME="/tmp/tmpom_xchel"; NPM_CONFIG_GLOBALCONFIG=/tmp/tmpom_xchel/npmrc NPM_CONFIG_USERCONFIG=/tmp/tmpom_xchel/npmrc npm config set json true' in /tmp/tmpom_xchel DEBUG: Running 'export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export SSH_AGENT_PID="1386"; export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh"; export PATH="/home/em/Dev/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/usr/bin/python3-native:/home/em/Dev/poky/scripts:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot/usr/bin/crossscripts:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/usr/sbin:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/usr/bin:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/sbin:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/bin:/home/em/Dev/poky/bitbake/bin:/home/em/Dev/build/tmp/hosttools"; export HOME="/tmp/tmpom_xchel"; NPM_CONFIG_GLOBALCONFIG=/tmp/tmpom_xchel/npmrc NPM_CONFIG_USERCONFIG=/tmp/tmpom_xchel/npmrc npm config set registry http://registry.npmjs.org' in /tmp/tmpom_xchel DEBUG: Running 'export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export SSH_AGENT_PID="1386"; export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh"; export PATH="/home/em/Dev/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/usr/bin/python3-native:/home/em/Dev/poky/scripts:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot/usr/bin/crossscripts:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/usr/sbin:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/usr/bin:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/sbin:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/bin:/home/em/Dev/poky/bitbake/bin:/home/em/Dev/build/tmp/hosttools"; export HOME="/tmp/tmpom_xchel"; NPM_CONFIG_GLOBALCONFIG=/tmp/tmpom_xchel/npmrc NPM_CONFIG_USERCONFIG=/tmp/tmpom_xchel/npmrc npm view node-red@1.2.2' in /tmp/tmpom_xchel DEBUG: Trying PREMIRRORS DEBUG: For url ['https', 'registry.npmjs.org', '/node-red/-/node-red-1.2.2.tgz', '', '', OrderedDict([('downloadfilename', 'npm2/node-red-1.2.2.tgz'), ('sha512sum', 'fdc19a8e8523a4280fd4f96a8edbeea66f2013ed7d5739bfcf7d4cdb97f9d1722e2e9cb7c22793922f9eda820f3f2bb34c3ec6cf4e0933249b701f2dc74be834')])] comparing ['bzr', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['https', 'registry.npmjs.org', '/node-red/-/node-red-1.2.2.tgz', '', '', OrderedDict([('downloadfilename', 'npm2/node-red-1.2.2.tgz'), ('sha512sum', 'fdc19a8e8523a4280fd4f96a8edbeea66f2013ed7d5739bfcf7d4cdb97f9d1722e2e9cb7c22793922f9eda820f3f2bb34c3ec6cf4e0933249b701f2dc74be834')])] comparing ['cvs', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['https', 'registry.npmjs.org', '/node-red/-/node-red-1.2.2.tgz', '', '', OrderedDict([('downloadfilename', 'npm2/node-red-1.2.2.tgz'), ('sha512sum', 'fdc19a8e8523a4280fd4f96a8edbeea66f2013ed7d5739bfcf7d4cdb97f9d1722e2e9cb7c22793922f9eda820f3f2bb34c3ec6cf4e0933249b701f2dc74be834')])] comparing ['git', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['https', 'registry.npmjs.org', '/node-red/-/node-red-1.2.2.tgz', '', '', OrderedDict([('downloadfilename', 'npm2/node-red-1.2.2.tgz'), ('sha512sum', 'fdc19a8e8523a4280fd4f96a8edbeea66f2013ed7d5739bfcf7d4cdb97f9d1722e2e9cb7c22793922f9eda820f3f2bb34c3ec6cf4e0933249b701f2dc74be834')])] comparing ['gitsm', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['https', 'registry.npmjs.org', '/node-red/-/node-red-1.2.2.tgz', '', '', OrderedDict([('downloadfilename', 'npm2/node-red-1.2.2.tgz'), ('sha512sum', 'fdc19a8e8523a4280fd4f96a8edbeea66f2013ed7d5739bfcf7d4cdb97f9d1722e2e9cb7c22793922f9eda820f3f2bb34c3ec6cf4e0933249b701f2dc74be834')])] comparing ['hg', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['https', 'registry.npmjs.org', '/node-red/-/node-red-1.2.2.tgz', '', '', OrderedDict([('downloadfilename', 'npm2/node-red-1.2.2.tgz'), ('sha512sum', 'fdc19a8e8523a4280fd4f96a8edbeea66f2013ed7d5739bfcf7d4cdb97f9d1722e2e9cb7c22793922f9eda820f3f2bb34c3ec6cf4e0933249b701f2dc74be834')])] comparing ['osc', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['https', 'registry.npmjs.org', '/node-red/-/node-red-1.2.2.tgz', '', '', OrderedDict([('downloadfilename', 'npm2/node-red-1.2.2.tgz'), ('sha512sum', 'fdc19a8e8523a4280fd4f96a8edbeea66f2013ed7d5739bfcf7d4cdb97f9d1722e2e9cb7c22793922f9eda820f3f2bb34c3ec6cf4e0933249b701f2dc74be834')])] comparing ['p4', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['https', 'registry.npmjs.org', '/node-red/-/node-red-1.2.2.tgz', '', '', OrderedDict([('downloadfilename', 'npm2/node-red-1.2.2.tgz'), ('sha512sum', 'fdc19a8e8523a4280fd4f96a8edbeea66f2013ed7d5739bfcf7d4cdb97f9d1722e2e9cb7c22793922f9eda820f3f2bb34c3ec6cf4e0933249b701f2dc74be834')])] comparing ['svn', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: Trying Upstream DEBUG: Trying PREMIRRORS DEBUG: For url ['https', 'registry.npmjs.org', '/node-red/-/node-red-1.2.2.tgz', '', '', OrderedDict([('downloadfilename', 'npm2/node-red-1.2.2.tgz'), ('sha512sum', 'fdc19a8e8523a4280fd4f96a8edbeea66f2013ed7d5739bfcf7d4cdb97f9d1722e2e9cb7c22793922f9eda820f3f2bb34c3ec6cf4e0933249b701f2dc74be834')])] comparing ['bzr', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['https', 'registry.npmjs.org', '/node-red/-/node-red-1.2.2.tgz', '', '', OrderedDict([('downloadfilename', 'npm2/node-red-1.2.2.tgz'), ('sha512sum', 'fdc19a8e8523a4280fd4f96a8edbeea66f2013ed7d5739bfcf7d4cdb97f9d1722e2e9cb7c22793922f9eda820f3f2bb34c3ec6cf4e0933249b701f2dc74be834')])] comparing ['cvs', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['https', 'registry.npmjs.org', '/node-red/-/node-red-1.2.2.tgz', '', '', OrderedDict([('downloadfilename', 'npm2/node-red-1.2.2.tgz'), ('sha512sum', 'fdc19a8e8523a4280fd4f96a8edbeea66f2013ed7d5739bfcf7d4cdb97f9d1722e2e9cb7c22793922f9eda820f3f2bb34c3ec6cf4e0933249b701f2dc74be834')])] comparing ['git', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['https', 'registry.npmjs.org', '/node-red/-/node-red-1.2.2.tgz', '', '', OrderedDict([('downloadfilename', 'npm2/node-red-1.2.2.tgz'), ('sha512sum', 'fdc19a8e8523a4280fd4f96a8edbeea66f2013ed7d5739bfcf7d4cdb97f9d1722e2e9cb7c22793922f9eda820f3f2bb34c3ec6cf4e0933249b701f2dc74be834')])] comparing ['gitsm', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['https', 'registry.npmjs.org', '/node-red/-/node-red-1.2.2.tgz', '', '', OrderedDict([('downloadfilename', 'npm2/node-red-1.2.2.tgz'), ('sha512sum', 'fdc19a8e8523a4280fd4f96a8edbeea66f2013ed7d5739bfcf7d4cdb97f9d1722e2e9cb7c22793922f9eda820f3f2bb34c3ec6cf4e0933249b701f2dc74be834')])] comparing ['hg', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['https', 'registry.npmjs.org', '/node-red/-/node-red-1.2.2.tgz', '', '', OrderedDict([('downloadfilename', 'npm2/node-red-1.2.2.tgz'), ('sha512sum', 'fdc19a8e8523a4280fd4f96a8edbeea66f2013ed7d5739bfcf7d4cdb97f9d1722e2e9cb7c22793922f9eda820f3f2bb34c3ec6cf4e0933249b701f2dc74be834')])] comparing ['osc', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['https', 'registry.npmjs.org', '/node-red/-/node-red-1.2.2.tgz', '', '', OrderedDict([('downloadfilename', 'npm2/node-red-1.2.2.tgz'), ('sha512sum', 'fdc19a8e8523a4280fd4f96a8edbeea66f2013ed7d5739bfcf7d4cdb97f9d1722e2e9cb7c22793922f9eda820f3f2bb34c3ec6cf4e0933249b701f2dc74be834')])] comparing ['p4', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['https', 'registry.npmjs.org', '/node-red/-/node-red-1.2.2.tgz', '', '', OrderedDict([('downloadfilename', 'npm2/node-red-1.2.2.tgz'), ('sha512sum', 'fdc19a8e8523a4280fd4f96a8edbeea66f2013ed7d5739bfcf7d4cdb97f9d1722e2e9cb7c22793922f9eda820f3f2bb34c3ec6cf4e0933249b701f2dc74be834')])] comparing ['svn', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: Trying Upstream DEBUG: Fetching https://registry.npmjs.org/node-red/-/node-red-1.2.2.tgz;downloadfilename=npm2/node-red-1.2.2.tgz;sha512sum=fdc19a8e8523a4280fd4f96a8edbeea66f2013ed7d5739bfcf7d4cdb97f9d1722e2e9cb7c22793922f9eda820f3f2bb34c3ec6cf4e0933249b701f2dc74be834 using command '/usr/bin/env wget -t 2 -T 30 --passive-ftp --no-check-certificate -O /home/em/Dev/build/downloads/npm2/node-red-1.2.2.tgz -P /home/em/Dev/build/downloads 'https://registry.npmjs.org/node-red/-/node-red-1.2.2.tgz'' DEBUG: Fetcher accessed the network with the command /usr/bin/env wget -t 2 -T 30 --passive-ftp --no-check-certificate -O /home/em/Dev/build/downloads/npm2/node-red-1.2.2.tgz -P /home/em/Dev/build/downloads 'https://registry.npmjs.org/node-red/-/node-red-1.2.2.tgz' DEBUG: Running export PSEUDO_DISABLED=1; export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export SSH_AGENT_PID="1386"; export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh"; export PATH="/home/em/Dev/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/usr/bin/python3-native:/home/em/Dev/poky/scripts:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot/usr/bin/crossscripts:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/usr/sbin:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/usr/bin:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/sbin:/home/em/Dev/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/node-red/1.2.2-r0/recipe-sysroot-native/bin:/home/em/Dev/poky/bitbake/bin:/home/em/Dev/build/tmp/hosttools"; export HOME="/home/em"; /usr/bin/env wget -t 2 -T 30 --passive-ftp --no-check-certificate -O /home/em/Dev/build/downloads/npm2/node-red-1.2.2.tgz -P /home/em/Dev/build/downloads 'https://registry.npmjs.org/node-red/-/node-red-1.2.2.tgz' --progress=dot -v --2021-12-07 15:19:03-- https://registry.npmjs.org/node-red/-/node-red-1.2.2.tgz Resolving registry.npmjs.org (registry.npmjs.org)... 2606:4700::6810:1b23, 2606:4700::6810:1423, 2606:4700::6810:1923, ... Connecting to registry.npmjs.org (registry.npmjs.org)|2606:4700::6810:1b23|:443... failed: Connection timed out. Connecting to registry.npmjs.org (registry.npmjs.org)|2606:4700::6810:1423|:443... failed: Connection timed out. Connecting to registry.npmjs.org (registry.npmjs.org)|2606:4700::6810:1923|:443... failed: Connection timed out. Connecting to registry.npmjs.org (registry.npmjs.org)|2606:4700::6810:1623|:443... failed: Connection timed out. Connecting to registry.npmjs.org (registry.npmjs.org)|2606:4700::6810:1223|:443... failed: Connection timed out. Connecting to registry.npmjs.org (registry.npmjs.org)|2606:4700::6810:1023|:443... failed: Connection timed out. Connecting to registry.npmjs.org (registry.npmjs.org)|2606:4700::6810:1323|:443... failed: Connection timed out. Connecting to registry.npmjs.org (registry.npmjs.org)|2606:4700::6810:1a23|:443... failed: Connection timed out. Connecting to registry.npmjs.org (registry.npmjs.org)|2606:4700::6810:1123|:443... failed: Connection timed out. Connecting to registry.npmjs.org (registry.npmjs.org)|2606:4700::6810:1723|:443... failed: Connection timed out. Connecting to registry.npmjs.org (registry.npmjs.org)|2606:4700::6810:1823|:443... failed: Connection timed out.

srware commented 2 years ago

Hi @Pan5ky,

This particular issue looks like it might be related to a large number of requests to the NPM registry, I can't recall having a similar problem. The NPMSW fetcher in bitbake pulls down all dependencies from the NPM registry and packages them which is why do_configure() takes a long time.

Having said all this, the NPM fetcher seems to be quite broken since master updated to NodeJS 16.X and NPM 8. I'm having issues with scripts trying to run for packages where previous NPM versions adhered to the "ignore-scripts" flag and several other problems which are currently making updating the node-red recipe difficult.

I will keep tabs on developments here and see what I can do but right now it's looking more like the node-red recipe will need to be disabled for master until a later nodejs and fixed NPM are pulled in.

ghost commented 2 years ago

Hi @srware, thank you for your reply. It seems that the issue was caused by me running the build system in a virtual machine. I didn't encounter the problem again after setting everything up on real hardware. I tried an older commit with node-red v1.02 and the current v1.2.2 and both commits worked without any problems