search

cnpm / cnpmjs.org

‼️ ‼️ ‼️ ‼️ DEPRECATED, please use https://github.com/cnpm/cnpmcore ‼️ ‼️ ‼️ ‼️
https://npmmirror.com
Other
3.55k stars 749 forks source link

cnpmjs 同步策略问题 #1157

Closed huang-x-h closed 7 years ago

huang-x-h commented 7 years ago

目前 syncModel 支持 'none', 'all', 'exist'

考虑如果设置 all 的话，公司磁盘没有那么大的空间

那么是否有这样的策略，用户在 npm install 时，cnpmjs 拦截检测模块如果服务器上没有，再去官网或者淘宝源去下载到服务器上，然后在传递到用户上，这样服务器就只放置用户需要的模块，下次用户再来 install 时，服务器有就不去官网下，也能解决用户在内网 npm install 的问题

alsotang commented 7 years ago

exist 就是这样工作的吧？

在 2017年3月31日 下午4:27，huang.xinghui notifications@github.com写道：

目前 syncModel 支持 'none', 'all', 'exist'

考虑如果设置 all 的话，公司磁盘没有那么大的空间

那么是否有这样的策略，用户在 npm install 时，cnpmjs 拦截检测模块如果服务器上没有， 再去官网或者淘宝源去下载到服务器上，然后在传递到用户上，这样服务器就只放置用户需要的模块，下次用户再来 install 时，服务器有就不去官网下，也能解决用户在内网 npm install 的问题

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cnpm/cnpmjs.org/issues/1157, or mute the thread https://github.com/notifications/unsubscribe-auth/ABGB75Cgr6trMfhAL6gYz4qytSrbbCA2ks5rrLj6gaJpZM4MvXi2 .

-- GitHub: https://github.com/alsotang

huang-x-h commented 7 years ago

貌似不是，昨天我一台机器没有连外网，配置 register 为自己搭建的 cnpmjs，去下载模块包，还是该机器去淘宝源去下载，但因为没有连外网，导致下载失败

按道理，不是由 cnpmjs 去代理下载吗？

alsotang commented 7 years ago

按照我的理解，第一次安装一个不存在的包时，cnpm会去拉取，同时，会返回302给客户端，让客户端自己去拉取。

在 2017年4月1日 上午9:25，huang.xinghui notifications@github.com写道：

貌似不是，昨天我一台机器没有连外网，配置 register 为自己搭建的 cnpmjs，去下载模块包，还是该机器去淘宝源去下载， 但因为没有连外网，导致下载失败

按道理，不是由 cnpmjs 去代理下载吗？

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cnpm/cnpmjs.org/issues/1157#issuecomment-290883613, or mute the thread https://github.com/notifications/unsubscribe-auth/ABGB74d_wzVEJkprn8fGiyx-U_AnmSFdks5rraebgaJpZM4MvXi2 .

-- GitHub: https://github.com/alsotang

huang-x-h commented 7 years ago

客户端自己去连淘宝源拉吗？

因为我客户端不能连外网，就想让cnpmjs.org这个服务器自己去拉取，放在服务器上，然后再返回给客户端，就类似一个代理cache功能

在 2017年4月5日，上午11:18，alsotang notifications@github.com 写道：

按照我的理解，第一次安装一个不存在的包时，cnpm会去拉取，同时，会返回302给客户端，让客户端自己去拉取。

在 2017年4月1日 上午9:25，huang.xinghui notifications@github.com写道：

貌似不是，昨天我一台机器没有连外网，配置 register 为自己搭建的 cnpmjs，去下载模块包，还是该机器去淘宝源去下载， 但因为没有连外网，导致下载失败

按道理，不是由 cnpmjs 去代理下载吗？

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cnpm/cnpmjs.org/issues/1157#issuecomment-290883613, or mute the thread https://github.com/notifications/unsubscribe-auth/ABGB74d_wzVEJkprn8fGiyx-U_AnmSFdks5rraebgaJpZM4MvXi2 .

-- GitHub: https://github.com/alsotang — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/cnpm/cnpmjs.org/issues/1157#issuecomment-291736997, or mute the thread https://github.com/notifications/unsubscribe-auth/AAXTpNaMela4ajXqCDrI4HcKOPpT4Uppks5rswfvgaJpZM4MvXi2.

alsotang commented 7 years ago

第一次去安装的时候虽然会失败，但是cnpm已经开始拉取了，那么过两分钟再装就好了。

在 2017年4月5日 下午1:10，huang.xinghui notifications@github.com写道：

客户端自己去连淘宝源拉吗？

因为我客户端不能连外网，就想让cnpmjs.org这个服务器自己去拉取，放在服务器上，然后再返回给客户端，就类似一个代理cache功能

在 2017年4月5日，上午11:18，alsotang notifications@github.com 写道：

按照我的理解，第一次安装一个不存在的包时，cnpm会去拉取，同时，会返回302给客户端，让客户端自己去拉取。

在 2017年4月1日 上午9:25，huang.xinghui notifications@github.com写道：

貌似不是，昨天我一台机器没有连外网，配置 register 为自己搭建的 cnpmjs，去下载模块包，还是该机器去淘宝源去下载， 但因为没有连外网，导致下载失败

按道理，不是由 cnpmjs 去代理下载吗？

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cnpm/cnpmjs.org/issues/1157#issuecomment-290883613,

or mute the thread https://github.com/notifications/unsubscribe- auth/ABGB74d_wzVEJkprn8fGiyx-U_AnmSFdks5rraebgaJpZM4MvXi2 .

-- GitHub: https://github.com/alsotang — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/cnpm/cnpmjs.org/issues/1157#issuecomment-291736997>, or mute the thread https://github.com/notifications/unsubscribe-auth/ AAXTpNaMela4ajXqCDrI4HcKOPpT4Uppks5rswfvgaJpZM4MvXi2.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cnpm/cnpmjs.org/issues/1157#issuecomment-291755962, or mute the thread https://github.com/notifications/unsubscribe-auth/ABGB7w0pcuNNDOK17yIE7SQn4A_fXKA_ks5rsyJDgaJpZM4MvXi2 .

-- GitHub: https://github.com/alsotang

huang-x-h commented 7 years ago

查看sync_info日志

[2017-04-05 16:38:46.392][32677] [sync_module_worker] download "http://registry.npm.taobao.org/caniuse-db/download/caniuse-db-1.0.30000567.tgz" to "/lvdata/nfv/.cnpmjs.org/downloads/caniuse-db-1.0.30000567.35acd71b2ded1b65ceca0a7f0305b1de.tgz" error: Error: EMFILE: too many open files, open '/lvdata/nfv/.cnpmjs.org/downloads/caniuse-db-1.0.30000567.35acd71b2ded1b65ceca0a7f0305b1de.tgz' (writeStream "error"), GET http://registry.npm.taobao.org/caniuse-db/download/caniuse-db-1.0.30000567.tgz -1 (connected: false, keepalive socket: false, agent status: {"createSocketCount":8708,"closeSocketCount":8706,"errorSocketCount":0,"timeoutSocketCount":3071,"requestCount":40510,"freeSockets":{},"sockets":{"registry.npm.taobao.org:80::":2},"requests":{}}) headers: {}

经常出现这个问题，然后去看 download 目录，文件都是空的

image

是否哪里配置有问题？

在 2017年4月6日，上午11:07，alsotang notifications@github.com 写道：

第一次去安装的时候虽然会失败，但是cnpm已经开始拉取了，那么过两分钟再装就好了。

在 2017年4月5日 下午1:10，huang.xinghui notifications@github.com写道：

客户端自己去连淘宝源拉吗？

因为我客户端不能连外网，就想让cnpmjs.org这个服务器自己去拉取，放在服务器上，然后再返回给客户端，就类似一个代理cache功能

在 2017年4月5日，上午11:18，alsotang notifications@github.com 写道：

按照我的理解，第一次安装一个不存在的包时，cnpm会去拉取，同时，会返回302给客户端，让客户端自己去拉取。

在 2017年4月1日 上午9:25，huang.xinghui notifications@github.com写道：

貌似不是，昨天我一台机器没有连外网，配置 register 为自己搭建的 cnpmjs，去下载模块包，还是该机器去淘宝源去下载， 但因为没有连外网，导致下载失败

按道理，不是由 cnpmjs 去代理下载吗？

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cnpm/cnpmjs.org/issues/1157#issuecomment-290883613,

or mute the thread <https://github.com/notifications/unsubscribe- auth/ABGB74d_wzVEJkprn8fGiyx-U_AnmSFdks5rraebgaJpZM4MvXi2> .

-- GitHub: https://github.com/alsotang — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/cnpm/cnpmjs.org/issues/1157#issuecomment-291736997>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ AAXTpNaMela4ajXqCDrI4HcKOPpT4Uppks5rswfvgaJpZM4MvXi2>.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cnpm/cnpmjs.org/issues/1157#issuecomment-291755962, or mute the thread https://github.com/notifications/unsubscribe-auth/ABGB7w0pcuNNDOK17yIE7SQn4A_fXKA_ks5rsyJDgaJpZM4MvXi2 .

-- GitHub: https://github.com/alsotang — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/cnpm/cnpmjs.org/issues/1157#issuecomment-292056647, or mute the thread https://github.com/notifications/unsubscribe-auth/AAXTpOpAZESlu1iFL-rqzg7-KA0KBirvks5rtFb-gaJpZM4MvXi2.

huang-x-h commented 7 years ago

还有这个 exist 策略的实现逻辑是什么？

比如我写了一个模块 @myscope/my-module，发布到自己搭建的 cnpmjs 上，该模块依赖第三方模块，例如 commander ，那么用户去安装该模块时，搭建的 cnpmjs 会去淘宝源去下载 commander 模块到服务器上，然后下次用户安装，可以直接去服务器下载目录下找返回，而不用再去淘宝源下载吗？

fengmk2 commented 7 years ago

Error: EMFILE: too many open files, open '/lvdata/nfv/.cnpmjs.org/downloads

系统文件句柄不够了，请将 ulimit 设置成最大。或者将 downloads 目录多余的文件删除了

fengmk2 commented 7 years ago

按照我的理解，第一次安装一个不存在的包时，cnpm会去拉取，同时，会返回302给客户端，让客户端自己去拉取。

@huang-x-h 不存在是客户端会先去下载的，而不是 cnpmjs 去代理

fengmk2 commented 7 years ago

@JacksonTian 赶紧提供基于 oss 的整体解决方案吧，单机磁盘存储运维都是很麻烦的。还是云存储靠谱。

huang-x-h commented 7 years ago

@fengmk2 设置了 ulimit 开放到 65536，删除 download 下面所有内容，然后启动

客户端去安装自己发布模块还是失败

目前后台日志，不停的刷（感觉是死循环了，不停去请求下载），download 目录下下载的模块包大小还是 0

截取部分日志

[2017-04-06 14:31:14.271][10569] send email with type: undefined, subject: undefined, html: undefined
[2017-04-06 14:31:53.336][10569] [sync_module_worker] download "http://registry.npm.taobao.org/readable-stream/download/readable-stream-2.2.5.tgz" to "/lvdata/nfv/.cnpmjs.org/downloads/readable-stream-2.2.5.24d1fbf86392b05ef7f76cd009c169c6.tgz" error: ResponseError: unable to verify the first certificate (req "error"), GET https://cdn.npm.taobao.org/readable-stream/-/readable-stream-2.2.5.tgz -1 (connected: true, keepalive socket: false, agent status: {"createSocketCount":6408,"closeSocketCount":6347,"errorSocketCount":0,"timeoutSocketCount":10,"requestCount":275,"freeSockets":{},"sockets":{"cdn.npm.taobao.org:443::::::::":61},"requests":{}})
headers: {}
[2017-04-06 14:31:53.337][10569] [sync_module_worker] delay: 1808965700 ms, publish_time: 2017-03-16 16:02:27.637, start sync readable-stream@2.2.6
[2017-04-06 14:31:53.375][10569] [sync_module_worker] downloading "http://registry.npm.taobao.org/readable-stream/download/readable-stream-2.2.6.tgz" to "/lvdata/nfv/.cnpmjs.org/downloads/readable-stream-2.2.6.9dbb6fa7abf5c816df2cb7b2a3b39d4a.tgz"
[2017-04-06 14:31:53.729][10569] [sync_module_worker] download "http://registry.npm.taobao.org/readable-stream/download/readable-stream-2.2.6.tgz" to "/lvdata/nfv/.cnpmjs.org/downloads/readable-stream-2.2.6.9dbb6fa7abf5c816df2cb7b2a3b39d4a.tgz" error: ResponseError: unable to verify the first certificate (req "error"), GET https://cdn.npm.taobao.org/readable-stream/-/readable-stream-2.2.6.tgz -1 (connected: true, keepalive socket: false, agent status: {"createSocketCount":6409,"closeSocketCount":6348,"errorSocketCount":0,"timeoutSocketCount":10,"requestCount":275,"freeSockets":{},"sockets":{"cdn.npm.taobao.org:443::::::::":61},"requests":{}})
headers: {}
[2017-04-06 14:31:54.162][10569] [sync_module_worker] delay: 153687992970 ms, publish_time: 2012-05-23 19:25:21.192, start sync xtend@1.0.0
[2017-04-06 14:31:54.162][10569] [sync_module_worker] downloading "http://registry.npm.taobao.org/xtend/download/xtend-1.0.0.tgz" to "/lvdata/nfv/.cnpmjs.org/downloads/xtend-1.0.0.766f6640cd9c5d59658fee6d4f6e1c43.tgz"
[2017-04-06 14:31:54.443][10569] [sync_module_worker] download "http://registry.npm.taobao.org/xtend/download/xtend-1.0.0.tgz" to "/lvdata/nfv/.cnpmjs.org/downloads/xtend-1.0.0.766f6640cd9c5d59658fee6d4f6e1c43.tgz" error: ResponseError: unable to verify the first certificate (req "error"), GET https://cdn.npm.taobao.org/xtend/-/xtend-1.0.0.tgz -1 (connected: true, keepalive socket: false, agent status: {"createSocketCount":6411,"closeSocketCount":6349,"errorSocketCount":0,"timeoutSocketCount":10,"requestCount":276,"freeSockets":{"registry.npm.taobao.org:443::::::::":1},"sockets":{"cdn.npm.taobao.org:443::::::::":61},"requests":{}})
headers: {}
[2017-04-06 14:31:54.443][10569] [sync_module_worker] delay: 153687854723 ms, publish_time: 2012-05-23 19:27:39.720, start sync xtend@1.0.1
[2017-04-06 14:31:54.444][10569] [sync_module_worker] downloading "http://registry.npm.taobao.org/xtend/download/xtend-1.0.1.tgz" to "/lvdata/nfv/.cnpmjs.org/downloads/xtend-1.0.1.2455408728beb18097b68cd6d2ccfea7.tgz"
[2017-04-06 14:31:54.742][10569] [sync_module_worker] download "http://registry.npm.taobao.org/duplexify/download/duplexify-1.5.0.tgz" to "/lvdata/nfv/.cnpmjs.org/downloads/duplexify-1.5.0.91ff7bbe0ab492aceefd60fbc07672ea.tgz" error: ResponseError: unable to verify the first certificate (req "error"), GET https://cdn.npm.taobao.org/duplexify/-/duplexify-1.5.0.tgz -1 (connected: true, keepalive socket: false, agent status: {"createSocketCount":6412,"closeSocketCount":6350,"errorSocketCount":0,"timeoutSocketCount":10,"requestCount":276,"freeSockets":{"registry.npm.taobao.org:443::::::::":1},"sockets":{"cdn.npm.taobao.org:443::::::::":61},"requests":{}})
headers: {}
fengmk2 commented 7 years ago

ResponseError: unable to verify the first certificate

呃，你的服务器没法支持 https://cdn.npm.taobao.org/xtend/-/xtend-1.0.0.tgz 使用的 ssl 证书？

fengmk2 commented 7 years ago

你需要信任一下这个证书

image

JacksonTian commented 7 years ago

收到。

alsotang commented 7 years ago

朴大也开始参与 cnpm 的建设了？

2017-04-08 22:42 GMT+08:00 Jackson Tian notifications@github.com:

收到。

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cnpm/cnpmjs.org/issues/1157#issuecomment-292722328, or mute the thread https://github.com/notifications/unsubscribe-auth/ABGB74qBe_TA7bAgbAlZJ4C_gQG0N-uUks5rt5zNgaJpZM4MvXi2 .

-- GitHub: https://github.com/alsotang