search

rxaviers / cldr-data-downloader

Download tool for Unicode CLDR JSON data
MIT License
16 stars 24 forks source link

Downloading during install stage of `cldr-data` fails #33

Open jvivs opened 6 years ago

jvivs commented 6 years ago

Having difficulty successfully installing the cldr-data package and its raising the following error from cldr-data-downloader:

Exit code: 1
Command: sh
Arguments: -c node install.js
/some/path/on/my/filesystem/node_modules/cldr-data
Output:
GET `https://github.com/unicode-cldr/cldr-core/archive/31.0.1.zip`
GET `https://github.com/unicode-cldr/cldr-dates-modern/archive/31.0.1.zip`
GET `https://github.com/unicode-cldr/cldr-cal-buddhist-modern/archive/31.0.1.zip`
GET `https://github.com/unicode-cldr/cldr-cal-chinese-modern/archive/31.0.1.zip`
GET `https://github.com/unicode-cldr/cldr-cal-coptic-modern/archive/31.0.1.zip`
GET `https://github.com/unicode-cldr/cldr-cal-dangi-modern/archive/31.0.1.zip`
GET `https://github.com/unicode-cldr/cldr-cal-ethiopic-modern/archive/31.0.1.zip`
GET `https://github.com/unicode-cldr/cldr-cal-hebrew-modern/archive/31.0.1.zip`
GET `https://github.com/unicode-cldr/cldr-cal-indian-modern/archive/31.0.1.zip`
GET `https://github.com/unicode-cldr/cldr-cal-islamic-modern/archive/31.0.1.zip`
GET `https://github.com/unicode-cldr/cldr-cal-japanese-modern/archive/31.0.1.zip`
GET `https://github.com/unicode-cldr/cldr-cal-persian-modern/archive/31.0.1.zip`
GET `https://github.com/unicode-cldr/cldr-cal-roc-modern/archive/31.0.1.zip`
GET `https://github.com/unicode-cldr/cldr-localenames-modern/archive/31.0.1.zip`
GET `https://github.com/unicode-cldr/cldr-misc-modern/archive/31.0.1.zip`
GET `https://github.com/unicode-cldr/cldr-numbers-modern/archive/31.0.1.zip`
GET `https://github.com/unicode-cldr/cldr-segments-modern/archive/31.0.1.zip`
GET `https://github.com/unicode-cldr/cldr-units-modern/archive/31.0.1.zip`
/some/path/on/my/filesystem/node_modules/cldr-data-downloader/lib/download.js:119
      throw error;
      ^

Error: connect ETIMEDOUT 192.30.255.112:443
    at Object._errnoException (util.js:1022:11)
    at _exceptionWithHostPort (util.js:1044:20)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1182:14)".

Platform information: macOS 10.12.6 cldr-data@31.0.1 cldr-data-downloader@0.3.2 node@8.9.4 npm@5.6.0 yarn@1.5.1

I have run into this issue with the downloader in the past, but it has been chugging along for the past ~6mos or so without much of a hiccup until today. My success rate today is ~5%.

When I try to download what seems to be the offending cldr zip (https://github.com/unicode-cldr/cldr-units-modern/archive/31.0.1.zip), the browser completes the download just fine.

I've done a bit of digging around in both cldr-data-downloader and request to try and get a better picture of what's going on. Investigation shows that request appears to be getting a 200, closing the connection, and emitting the complete event, but the downloader still does not not finish successfully:

(end of output NODE_DEBUG=request npm run install from inside node_modules/cldr-data)

REQUEST response end https://codeload.github.com/unicode-cldr/cldr-numbers-modern/zip/31.0.1 200 { 'content-length': '3629675',
  'access-control-allow-origin': 'https://render.githubusercontent.com',
  'content-security-policy': 'default-src \'none\'; style-src \'unsafe-inline\'; sandbox',
  'strict-transport-security': 'max-age=31536000',
  vary: 'Authorization,Accept-Encoding',
  'x-content-type-options': 'nosniff',
  'x-frame-options': 'deny',
  'x-xss-protection': '1; mode=block',
  etag: '"39f1f5bac2fa7a22fd33636ad4d6f1b179c61cd5"',
  'content-type': 'application/zip',
  'content-disposition': 'attachment; filename=cldr-numbers-modern-31.0.1.zip',
  'x-geo-block-list': '',
  date: 'Wed, 02 May 2018 23:16:18 GMT',
  'x-github-request-id': 'F304:5639:18A56:27D35:5AEA46C2',
  connection: 'close' }
REQUEST end event https://codeload.github.com/unicode-cldr/cldr-numbers-modern/zip/31.0.1
REQUEST has body https://codeload.github.com/unicode-cldr/cldr-numbers-modern/zip/31.0.1 3629675
REQUEST emitting complete https://codeload.github.com/unicode-cldr/cldr-numbers-modern/zip/31.0.1
  [----------------------------------------] 29012347/Infinity 0% 0.0s^C

I also get the above result when attempting to install or upgrade a dependency with yarn, bootstrapping with lerna (using yarn + workspaces under the hood), or installing with npm.

One thing that does seem to stick out to me is that the final bytes counted by request-progress may don't seem to be accurate or indicative of a successful download or not.

Does this jog anyone's memory?

paulmolluzzo commented 6 years ago

I'm also seeing this issue on every first installation, but it's not occurring on subsequent installs.

Something that stands out is the line at the end of @jvivs's request logs:

 [----------------------------------------] 29012347/Infinity 0% 0.0s^C

I have a similar line in my output: 

 [----------------------------------------] 31710606/Infinity 0% 0.0sUnpacking it into `./`

Maybe the ZIP file at https://codeload.github.com/unicode-cldr/cldr-numbers-modern/zip/31.0.1 is corrupted somehow?

v-echo commented 6 years ago

I get a similar error at the same line.

throw error; ^

Error: connect ETIMEDOUT 192.30.253.112:443 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1161:14) npm WARN globalize-webpack-plugin@2.1.0 requires a peer of webpack@^3.0.0 but none is installed. You must install peer dependencies yourself. npm WARN skip-amd-webpack-plugin@0.2.0 requires a peer of webpack@^1.9.0 || ^2.2.0 || ^3.0.0 but none is installed. You must install peer dependencies yourself.

RedRelay commented 6 years ago

I get the same error. Your librairy is embedded into the new ecommerce storefront boilerplate from Salesforce. And a lot of user will be in trouble with this issue :/

davojan commented 5 years ago

Running download script manually worked for me:

./node_modules/cldr-data-downloader/bin/download.sh -i ./node_modules/cldr-data/urls.json -o ./node_modules/cldr-data/
yarn
KenACollins commented 3 years ago

Hi, for the Windows users, this is the syntax...

npm install cldr-data-downloader
node ./node_modules/cldr-data-downloader/bin/download.js -i http://www.unicode.org/Public/cldr/26/json.zip -o ./cldr

This added cldr-data-downloader to package.json and created a cldr folder within the root folder of my application and filled it with files.

However, this did not resolve the problem for me and it still kept bombing with a local certificate error for which the usual fix...

npm config set strict-ssl=false

...did not work.

In the end, I found that a bombing installation did not stop the app from running.

kapsiR commented 2 years ago

We have this issue in our CI environment:

npm ERR! \node_modules\cldr-data-downloader\lib\download.js:119 npm ERR! throw error; npm ERR! ^ npm ERR! npm ERR! Error: connect ETIMEDOUT *.82.121.3:443

Details - Click to expand > npm info run cldr-data@36.0.0 install { code: 1, signal: null } > npm timing reify:rollback:createSparse Completed in 15570ms > npm timing reify:rollback:retireShallow Completed in 0ms > npm timing command:install Completed in 73691ms > npm ERR! code 1 > npm ERR! path \node_modules\cldr-data > npm ERR! command failed > npm ERR! command C:\Windows\system32\cmd.exe /d /s /c node install.js > npm ERR! GET `https://github.com/unicode-cldr/cldr-core/archive/36.0.0.zip` > npm ERR! GET `https://github.com/unicode-cldr/cldr-dates-modern/archive/36.0.0.zip` > npm ERR! GET `https://github.com/unicode-cldr/cldr-cal-buddhist-modern/archive/36.0.0.zip` > npm ERR! GET `https://github.com/unicode-cldr/cldr-cal-chinese-modern/archive/36.0.0.zip` > npm ERR! GET `https://github.com/unicode-cldr/cldr-cal-coptic-modern/archive/36.0.0.zip` > npm ERR! GET `https://github.com/unicode-cldr/cldr-cal-dangi-modern/archive/36.0.0.zip` > npm ERR! GET `https://github.com/unicode-cldr/cldr-cal-ethiopic-modern/archive/36.0.0.zip` > npm ERR! GET `https://github.com/unicode-cldr/cldr-cal-hebrew-modern/archive/36.0.0.zip` > npm ERR! GET `https://github.com/unicode-cldr/cldr-cal-indian-modern/archive/36.0.0.zip` > npm ERR! GET `https://github.com/unicode-cldr/cldr-cal-islamic-modern/archive/36.0.0.zip` > npm ERR! GET `https://github.com/unicode-cldr/cldr-cal-japanese-modern/archive/36.0.0.zip` > npm ERR! GET `https://github.com/unicode-cldr/cldr-cal-persian-modern/archive/36.0.0.zip` > npm ERR! GET `https://github.com/unicode-cldr/cldr-cal-roc-modern/archive/36.0.0.zip` > npm ERR! GET `https://github.com/unicode-cldr/cldr-localenames-modern/archive/36.0.0.zip` > npm ERR! GET `https://github.com/unicode-cldr/cldr-misc-modern/archive/36.0.0.zip` > npm ERR! GET `https://github.com/unicode-cldr/cldr-numbers-modern/archive/36.0.0.zip` > npm ERR! GET `https://github.com/unicode-cldr/cldr-segments-modern/archive/36.0.0.zip` > npm ERR! GET `https://github.com/unicode-cldr/cldr-units-modern/archive/36.0.0.zip` > npm ERR! npm info using npm@8.5.0 > npm ERR! npm info using node@v16.14.2 > npm ERR! npm timing npm:load:whichnode Completed in 1ms > npm ERR! npm timing config:load:defaults Completed in 1ms > npm ERR! npm timing config:load:file:C:\Program Files\nodejs\node_modules\npm\npmrc Completed in 6ms > npm ERR! npm timing config:load:builtin Completed in 6ms > npm ERR! npm timing config:load:cli Completed in 2ms > npm ERR! npm timing config:load:env Completed in 2ms > npm ERR! npm timing config:load:project Completed in 3ms > npm ERR! npm timing config:load:file:C:\Users\**\.npmrc Completed in 0ms > npm ERR! npm timing config:load:user Completed in 0ms > npm ERR! npm timing config:load:file:C:\Users\**\AppData\Roaming\npm\etc\npmrc Completed in 0ms > npm ERR! npm timing config:load:global Completed in 1ms > npm ERR! npm timing config:load:validate Completed in 0ms > npm ERR! npm timing config:load:credentials Completed in 1ms > npm ERR! npm timing config:load:setEnvs Completed in 1ms > npm ERR! npm timing config:load Completed in 18ms > npm ERR! npm timing npm:load:configload Completed in 18ms > npm ERR! npm timing npm:load:setTitle Completed in 0ms > npm ERR! npm timing config:load:flatten Completed in 4ms > npm ERR! npm timing npm:load:display Completed in 7ms > npm ERR! npm timing npm:load:logFile Completed in 6ms > npm ERR! npm timing npm:load:timers Completed in 0ms > npm ERR! npm timing npm:load:configScope Completed in 0ms > npm ERR! npm timing npm:load Completed in 33ms > npm ERR! npm timing command:prefix Completed in 0ms > npm ERR! npm timing npm Completed in 248ms > npm ERR! npm info ok > npm ERR! npm info using npm@8.5.0 > npm ERR! npm info using node@v16.14.2 > npm ERR! npm timing npm:load:whichnode Completed in 1ms > npm ERR! npm timing config:load:defaults Completed in 1ms > npm ERR! npm timing config:load:file:C:\Program Files\nodejs\node_modules\npm\npmrc Completed in 7ms > npm ERR! npm timing config:load:builtin Completed in 7ms > npm ERR! npm timing config:load:cli Completed in 2ms > npm ERR! npm timing config:load:env Completed in 3ms > npm ERR! npm timing config:load:file:\node_modules\cldr-data\.npmrc Completed in 1ms > npm ERR! npm timing config:load:project Completed in 4ms > npm ERR! npm timing config:load:file:C:\Users\**\.npmrc Completed in 0ms > npm ERR! npm timing config:load:user Completed in 0ms > npm ERR! npm timing config:load:file:C:\Users\**\AppData\Roaming\npm\etc\npmrc Completed in 0ms > npm ERR! npm timing config:load:global Completed in 0ms > npm ERR! npm timing config:load:validate Completed in 0ms > npm ERR! npm timing config:load:credentials Completed in 1ms > npm ERR! npm timing config:load:setEnvs Completed in 2ms > npm ERR! npm timing config:load Completed in 21ms > npm ERR! npm timing npm:load:configload Completed in 21ms > npm ERR! npm timing npm:load:setTitle Completed in 0ms > npm ERR! npm timing config:load:flatten Completed in 4ms > npm ERR! npm timing npm:load:display Completed in 8ms > npm ERR! npm timing npm:load:logFile Completed in 8ms > npm ERR! npm timing npm:load:timers Completed in 0ms > npm ERR! npm timing npm:load:configScope Completed in 0ms > npm ERR! npm timing npm:load Completed in 38ms > npm ERR! npm timing npm Completed in 226ms > npm ERR! npm info ok > npm ERR! \node_modules\cldr-data-downloader\lib\download.js:119 > npm ERR! throw error; > npm ERR! ^ > npm ERR! > npm ERR! Error: connect ETIMEDOUT *.82.121.3:443 > npm ERR! at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1157:16) { > npm ERR! errno: -4039, > npm ERR! code: 'ETIMEDOUT', > npm ERR! syscall: 'connect', > npm ERR! address: '*.82.121.3', > npm ERR! port: 443 > npm ERR! } > npm timing npm Completed in 73945ms

Doing something manually here is not really an option for us...
Is there any workaround available? (maybe increasing the timeout?)

rxaviers commented 2 years ago

A fix is welcome, in case someone has bandwidth to provide one. Thank you all.

x-NIROSHAN-x commented 1 year ago

I don't know the reason but switching the network connection fixed the issue for me

cw-son commented 1 year ago

In Mac OS, i had same issue, but when i remove Virus Security filter Network Content. I passed error. Maybe problem in network firewall or Internet Security Virus tool. Also Thanks @NiroshanKJ 

GET `https://github.com/unicode-cldr/cldr-units-modern/archive/32.0.0.zip`
  [========================================] 31967630/31967630 100% 0.0s
Received 31218K total.
zhjuncai commented 1 year ago

No matter how I change the network, I'm always getting the error 

GET `https://github.com/unicode-cldr/cldr-misc-modern/archive/36.0.0.zip`
GET `https://github.com/unicode-cldr/cldr-numbers-modern/archive/36.0.0.zip`
GET `https://github.com/unicode-cldr/cldr-segments-modern/archive/36.0.0.zip`
GET `https://github.com/unicode-cldr/cldr-units-modern/archive/36.0.0.zip`
Whops Error making request.
Error: error request aborted
    at createError (/xxx/node_modules/axios/lib/core/createError.js:16:15)
    at IncomingMessage.handlerStreamAborted (/Users/xxx/git/xxx/node_modules/axios/lib/adapters/http.js:301:18)
    at IncomingMessage.emit (events.js:400:28)
    at TLSSocket.socketCloseListener (_http_client.js:432:11)
    at TLSSocket.emit (events.js:412:35)
    at net.js:686:12
    at TCP.done (_tls_wrap.js:564:7)

Please report this full log at https://github.com/rxaviers/cldr-data-downloader
aurasolis commented 1 year ago

same.. GET https://github.com/unicode-cldr/cldr-numbers-modern/archive/36.0.0.zip GET https://github.com/unicode-cldr/cldr-segments-modern/archive/36.0.0.zip GET https://github.com/unicode-cldr/cldr-units-modern/archive/36.0.0.zip Whops Error making request. Error: error request aborted at createError (/Users/xxx/Developer/reservamos-transporters/node_modules/axios/lib/core/createError.js:16:15) at IncomingMessage.handlerStreamAborted (/Users/xxx/Developer/xxx/node_modules/axios/lib/adapters/http.js:301:18) at IncomingMessage.emit (events.js:375:28) at TLSSocket.socketCloseListener (_http_client.js:432:11) at TLSSocket.emit (events.js:387:35) at net.js:675:12

litterGuy commented 1 year ago

the same error:

[INFO] GET "https://github.com/unicode-cldr/cldr-units-modern/archive/36.0.0.zip" [INFO] D:\works\java\radixIot\ma-dashboards\UI\node_modules\cldr-data-downloader\lib\download.js:119 [INFO] throw error; [INFO] ^ [INFO] [INFO] Error: read ECONNRESET [INFO] at TLSWrap.onStreamRead (node:internal/stream_base_commons:220:20) { [INFO] errno: -4077, [INFO] code: 'ECONNRESET', [INFO] syscall: 'read' [INFO] }

WilliamHolmes commented 1 year ago

Now using node v14.21.1 (npm v6.14.17)

I see the same error today...

GET `https://github.com/unicode-cldr/cldr-core/archive/36.0.0.zip`
GET `https://github.com/unicode-cldr/cldr-dates-modern/archive/36.0.0.zip`
GET `https://github.com/unicode-cldr/cldr-cal-buddhist-modern/archive/36.0.0.zip`
GET `https://github.com/unicode-cldr/cldr-cal-chinese-modern/archive/36.0.0.zip`
GET `https://github.com/unicode-cldr/cldr-cal-coptic-modern/archive/36.0.0.zip`
GET `https://github.com/unicode-cldr/cldr-cal-dangi-modern/archive/36.0.0.zip`
GET `https://github.com/unicode-cldr/cldr-cal-ethiopic-modern/archive/36.0.0.zip`
GET `https://github.com/unicode-cldr/cldr-cal-hebrew-modern/archive/36.0.0.zip`
GET `https://github.com/unicode-cldr/cldr-cal-indian-modern/archive/36.0.0.zip`
GET `https://github.com/unicode-cldr/cldr-cal-islamic-modern/archive/36.0.0.zip`
GET `https://github.com/unicode-cldr/cldr-cal-japanese-modern/archive/36.0.0.zip`
GET `https://github.com/unicode-cldr/cldr-cal-persian-modern/archive/36.0.0.zip`
GET `https://github.com/unicode-cldr/cldr-cal-roc-modern/archive/36.0.0.zip`
GET `https://github.com/unicode-cldr/cldr-localenames-modern/archive/36.0.0.zip`
GET `https://github.com/unicode-cldr/cldr-misc-modern/archive/36.0.0.zip`
GET `https://github.com/unicode-cldr/cldr-numbers-modern/archive/36.0.0.zip`
GET `https://github.com/unicode-cldr/cldr-segments-modern/archive/36.0.0.zip`
GET `https://github.com/unicode-cldr/cldr-units-modern/archive/36.0.0.zip`
Whops Error making request.
Error: read ECONNRESET
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:209:20)

Please report this full log at https://github.com/rxaviers/cldr-data-downloader
matthewwood85 commented 6 months ago

Whenever I try and run npm install locally I get this error. Running it through our pipelines has always worked fine, but today I got the error on there as well. Re-ran the job and it was OK again. Very strange.

mazenadel19 commented 4 months ago

Any update on this or a workaround?

davidc-pc commented 1 week ago

Hey everyone, I found a fix for my repository

This works specifically for pipelines: Change npm install to npm ci

This way it will only build what is specified in package-lock.json. it's also much faster for pipelines and best practice apparently

This stopped the socket errors for me and cut the time considerably