想请教一下为什么与官方仓库包数目相差两万多

[bugyak]

http://registry.npmjs.org/ doc_count: 331639, https://registry.cnpmjs.org/ doc_count: 294289, https://registry.npm.taobao.org/ doc_count: 296713,

[alsotang]

咦。npm.taobao.org 的包竟然比 cnpmjs.org 多？

2016-05-30 10:13 GMT+08:00 吴彦祖 notifications@github.com:

http://registry.npmjs.org/ doc_count: 331639, https://registry.cnpmjs.org/ doc_count: 294289, https://registry.npm.taobao.org/ doc_count: 296713,

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cnpm/cnpmjs.org/issues/939, or mute the thread https://github.com/notifications/unsubscribe/ABGB72XB8dU0bkLZ8RO2IZbiiC7K6Hgpks5qGkfVgaJpZM4IpcPF .

[dead-horse]

之前有过删包的情况可能没有同步过来

[fengmk2]

请问是有什么包安装丢失版本吗？

[bugyak]

没有哦。

2016-06-02 9:01 GMT+08:00 fengmk2 notifications@github.com:

请问是有什么包安装丢失版本吗？

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/cnpm/cnpmjs.org/issues/939#issuecomment-223168919, or mute the thread https://github.com/notifications/unsubscribe/AHWqPkWyh62SrVgdJkEmds6cSrRQaDR_ks5qHiuDgaJpZM4IpcPF .

[xiaqun]

你好，我的也出现比淘宝镜像源少很多，配置文件里面设置的是syncModel: 'all', 不知道为什么出现这种情况，哪位大神有解决办法

[fengmk2]

你看看 sync_info*.log 的日志输出？

[xiaqun]

日志没什么异常 [2016-10-25 23:59:25.864][30927] [sync_module_worker] downloading "http://registry.npm.taobao.org/backendless-coderunner/download/backendless-coderunner-1.9.0.tgz" to "/root/.cnpmjs.org/downloads/backendless-coderunner-1.9.0.2405605f2278006ef4a7b2207aa8d41b.tgz" [2016-10-25 23:59:25.919][30927] [sync_module_worker] uploading {"key":"/backendless-coderunner/-/backendless-coderunner-1.9.0.tgz","size":32234,"shasum":"e96a1bb6ebebb4950f22fa9b8ad0846aee84ffa6"} to nfs [2016-10-25 23:59:25.920][30927] [sync_module_worker] uploaded, saving {"key":"/backendless-coderunner/-/backendless-coderunner-1.9.0.tgz"} to database [2016-10-25 23:59:25.949][30927] [sync_module_worker] sync backendless-coderunner@1.9.0 done! [2016-10-25 23:59:31.731][30927] [sync_module_worker] start sync vue-admin@0.1.8 [2016-10-25 23:59:31.780][30927] [sync_module_worker] downloading "http://registry.npm.taobao.org/vue-admin/download/vue-admin-0.1.8.tgz" to "/root/.cnpmjs.org/downloads/vue-admin-0.1.8.c23e4b653d56f20f73b2588f22a9eae8.tgz" [2016-10-25 23:59:31.847][30927] [sync_module_worker] uploading {"key":"/vue-admin/-/vue-admin-0.1.8.tgz","size":267175,"shasum":"f273144a746bc8b0fc16ecbf6bf3e773eb67163e"} to nfs [2016-10-25 23:59:31.854][30927] [sync_module_worker] uploaded, saving {"key":"/vue-admin/-/vue-admin-0.1.8.tgz"} to database [2016-10-25 23:59:31.989][30927] [sync_module_worker] start sync itsa-redux-clientstorage@15.2.25 [2016-10-25 23:59:31.992][30927] [sync_module_worker] downloading "http://registry.npm.taobao.org/itsa-redux-clientstorage/download/itsa-redux-clientstorage-15.2.25.tgz" to "/root/.cnpmjs.org/downloads/itsa-redux-clientstorage-15.2.25.d1f767214ffcdab5ed17df3349f9c2ce.tgz" [2016-10-25 23:59:32.015][30927] [sync_module_worker] uploading {"key":"/itsa-redux-clientstorage/-/itsa-redux-clientstorage-15.2.25.tgz","size":3126,"shasum":"4a55c9901cec3885cd43aa5ee018fd16b831121a"} to nfs [2016-10-25 23:59:32.040][30927] [sync_module_worker] uploaded, saving {"key":"/itsa-redux-clientstorage/-/itsa-redux-clientstorage-15.2.25.tgz"} to database [2016-10-25 23:59:32.050][30927] [sync_module_worker] sync vue-admin@0.1.8 done! [2016-10-25 23:59:32.086][30927] [sync_module_worker] sync itsa-redux-clientstorage@15.2.25 done! [2016-10-25 23:59:36.487][30927] [sync_module_worker] uploading {"key":"/@style/http-queue/-/@style/http-queue-1.1.1.tgz","size":12814,"shasum":"1497cec14cd2961713397ca5542101ba791d5582"} to nfs [2016-10-25 23:59:36.488][30927] [sync_module_worker] uploaded, saving {"key":"/@style/http-queue/-/@style/http-queue-1.1.1.tgz"} to database [2016-10-25 23:59:36.540][30927] [sync_module_worker] sync @style/http-queue@1.1.1 done! [2016-10-25 23:59:36.540][30927] [sync_module_worker] start sync @style/http-queue@1.2.0 [2016-10-25 23:59:36.545][30927] [sync_module_worker] downloading "http://registry.npm.taobao.org/@style/http-queue/download/@style/http-queue-1.2.0.tgz" to "/root/.cnpmjs.org/downloads/http-queue-1.2.0.2e5044f25a16f29df60b74e4330c03c8.tgz" [2016-10-25 23:59:36.622][30927] [sync_module_worker] uploading {"key":"/@style/http-queue/-/@style/http-queue-1.2.0.tgz","size":9874,"shasum":"c782e8728252c333a36bcd5ed54ca3e344c470fa"} to nfs [2016-10-25 23:59:36.623][30927] [sync_module_worker] uploaded, saving {"key":"/@style/http-queue/-/@style/http-queue-1.2.0.tgz"} to database [2016-10-25 23:59:36.663][30927] [sync_module_worker] sync @style/http-queue@1.2.0 done! [2016-10-25 23:59:36.663][30927] [sync_module_worker] start sync @style/http-queue@1.3.0 [2016-10-25 23:59:36.667][30927] [sync_module_worker] downloading "http://registry.npm.taobao.org/@style/http-queue/download/@style/http-queue-1.3.0.tgz" to "/root/.cnpmjs.org/downloads/http-queue-1.3.0.fe0f665a09a5954e68461aa0d9f32eb1.tgz" [2016-10-25 23:59:36.701][30927] [sync_module_worker] uploading {"key":"/@style/http-queue/-/@style/http-queue-1.3.0.tgz","size":9943,"shasum":"e3f8635f97f209ff91edb5bc974edd114445d91d"} to nfs [2016-10-25 23:59:36.701][30927] [sync_module_worker] uploaded, saving {"key":"/@style/http-queue/-/@style/http-queue-1.3.0.tgz"} to database [2016-10-25 23:59:36.738][30927] [sync_module_worker] sync @style/http-queue@1.3.0 done! [2016-10-25 23:59:36.870][30927] All packages sync done, successes 212, fails 3, updates 93 {"successes":["ecolutis-junit-viewer","gormur","component-basebite-vuejs","dig-csv","postcss-random","angulartics-google-analytics","bootstrap-daterangepicker-v2","scriptabuild","borschik","eslint-config-mirage","d3-svg-annotation","@shoutem/cli_dev","vue-lory","artillery-core","sails-hook-sequelize-multi","wsp","x-hapi","udp-request","eurostar-traveller-select","us-abbreviations","mobx-react-form","@types/typedarray-pool","@emvu/react-scripts","ng2-tree-plugin","fenix-ui-catalog","deepword","@shoutem/cli","abe-cli","fenix-ui-analysis","accord","gremlins","@benefex/inactivity-countdown-timer","vue-bulma-collapse","reactjs-components","ibird-client","generator-feather","ng2-mtna-lib","generator-uamithril-web-starter","artillery-plugin-statsd","ibird","hubot-speaking-of-which","preston-core","redux-solidity","midwest-module-membership","vinyl-fork","simple-url","organism-react-event-timeline","@tlrg/aws-autoscale-deployer","uwmctools","wit-module","rhinostyle","test_track_js_client","pagination-pager","botimize-botkit-middleware","next","ionic-sdk","league-wrapper","resurrect","html-validator","chocolatin","nativescript-dev-webpack","clv-kerberos-translator","hellopackage_fulanodetal","geo-locator","react-flexbox-component-grid","nlp-corpus","microcosm","hd-wallet","pacu","heidihx-api","gamestream","minus-grid","sparta-bootstrap-loader","myfox-wrapper-api","artillery","mount-fs","js_tutorial","consolemd","tokenmanager","openchemlib","consola.css","ember-cli-ecosystem-installer","objectschema","additor-elements","ionic-angular","cold-brew","react-measure","kiss-router","union-fs","chrome-remote-interface","lesspass-webcrypto","@dlghq/babel-preset-dialog","hi-ro","contextable","@sanity/server","tipranks-ab-testing","react-intl-es6","ledger-publisher","fishsticks","supertype","html-validator-cli","response-callbacks","liferay-karma-alloy-config","persistor","@gilt-tech/swig-run","clv-standard-translator","@sanity/cli","outnet-2015","itsa-reflux-clientstorage","itsa-react-router","@types/com.darktalker.cordova.screenshot","itsa-apps","c99","itsa-client-controller","no-kafka","bingzhe","sosijs","errorreporter","dr-js","jwt-user-login","generic-request-options","schlump","videojs-externals","feathers-service-verify-reset","pync","dynamo-merge","npg_ranger","dashode2","@types/murmurhash-js","salad-ui.form","salad-ui.icon","itsa-cli","salad-ui.util","react_native_animated","wix-node-build","nqm-databot-host","angular-qr","ok-js-sdk","generator-onbrand","salad-ui.chart","drakonian","zettapi_server","xclient","qpjs","protractor-perf","postmark","fast-react-server","sparta-font-awesome-webpack","disyuntor","swole","chunker-blocks","videocontext","ajax2","pdf.js-extract","node-qsb","sparta-isomorphic-style-loader","billogram-foundation","purecloud_api_sdk_javascript","@djforth/ap_karma_wp","steal-npm","@cnbritain/wc-card","dredd","all-the-package-names","normalize-pkg","jshook","lesspass","GMP","piratelang","skiff","mergejson","salad-ui.transitions","node-docx","react-universal-saga","rbst-lite","react-lazy-load-x","vbuild","unirouter","iconoo","dalek-reporter-mongodb","@cnbritain/wc-section-show-season","generator-hoop-django","mergeyaml","clusterpost-list","npmtextrz","@types/screenfull","seneca","pubsweet-component-draft","rediserve","publiclab-editor","crehub_node_sdk","@oddnetworks/oddworks","dispersive-react","@bongione/passport-seneca-app-token","rxstore-jest","timequerylog","nativescript-phone","kavie","ng2-chapp","beebon-reporter","readl-async","u-test","aiyo-logger","spruce-goose","fenix-ui-DataEditor","log-extractor","bedrock-messages-client-http","bouton","backendless-coderunner","ledger-client","vue-admin","itsa-redux-clientstorage","@style/http-queue"],"fails":["react-native-apps-flyer","bitmovin-player","@gilt-tech/swig-zk"],"updates":["heidihx-api","minus-grid","sparta-bootstrap-loader","mount-fs","js_tutorial","union-fs","chrome-remote-interface","lesspass-webcrypto","@dlghq/babel-preset-dialog","react-intl-es6","ledger-publisher","html-validator-cli","response-callbacks","itsa-reflux-clientstorage","itsa-react-router","@types/com.darktalker.cordova.screenshot","itsa-apps","c99","itsa-client-controller","no-kafka","bingzhe","errorreporter","dr-js","jwt-user-login","generic-request-options","pync","npg_ranger","dashode2","@types/murmurhash-js","salad-ui.form","salad-ui.icon","salad-ui.util","ok-js-sdk","salad-ui.chart","drakonian","zettapi_server","xclient","qpjs","protractor-perf","postmark","fast-react-server","sparta-font-awesome-webpack","swole","videocontext","ajax2","pdf.js-extract","node-qsb","sparta-isomorphic-style-loader","purecloud_api_sdk_javascript","steal-npm","dredd","normalize-pkg","lesspass","GMP","skiff","salad-ui.transitions","node-docx","react-universal-saga","react-lazy-load-x","vbuild","unirouter","iconoo","dalek-reporter-mongodb","@cnbritain/wc-section-show-season","clusterpost-list","npmtextrz","@types/screenfull","seneca","pubsweet-component-draft","rediserve","publiclab-editor","crehub_node_sdk","@oddnetworks/oddworks","dispersive-react","@bongione/passport-seneca-app-token","rxstore-jest","timequerylog","nativescript-phone","kavie","ng2-chapp","beebon-reporter","readl-async","u-test","aiyo-logger","spruce-goose","fenix-ui-DataEditor","log-extractor","bedrock-messages-client-http","bouton","backendless-coderunner","vue-admin","itsa-redux-clientstorage","@style/http-queue"]} [2016-10-25 23:59:36.870][30927] send email with type: warn, subject: Sync Finished But Some Packages Failed, html: Sync packages from official registry finished, but some packages sync failed. Start sync time is Tue Oct 25 2016 23:59:36 GMT+0800 (CST). 3 packges sync failed: ["react-native-apps-flyer","bitmovin-player","@gilt-tech/swig-zk"] ... 212 packages sync successes :["ecolutis-junit-viewer","gormur","component-basebite-vuejs","dig-csv","postcss-random","angulartics-google-analytics","bootstrap-daterangepicker-v2","scriptabuild","borschik","eslint-config-mirage"] ...

[fengmk2]

All packages sync done, successes 212, fails 3, updates 93 有同步失败的情况出现，能否告知具体差了多少个包？看看数量级。

[xiaqun]

从现在的数量上看，差不多是少了10万，有些依赖包安装的时候一直连接超时，一直安装不了 2016-10-26 16:22:37.456 nodejs.ResponseTimeoutErrorException: ResponseTimeoutError: Response timeout for 20000ms, PUT https://registry.npm.taobao.org/@gilt-tech/swig-zk/sync -1 (connected: true, keepalive socket: false, agent status: {"createSocketCount":4860,"closeSocketCount":4707,"errorSocketCount":0,"timeoutSocketCount":0,"requestCount":4552,"freeSockets":{"registry.npm.taobao.org:443::::::::":1},"sockets":{"registry.npm.taobao.org:443::::::::":152},"requests":{}}) headers: {} headers: {} at null._onTimeout (/usr/local/cnpmjs.org/node_modules/urllib/lib/urllib.js:701:15) at Timer.listOnTimeout (timers.js:92:15) pid: 10993 Host: a3-prod-cnpm-128-44.sh

[fengmk2]

看日志是 PUT https://registry.npm.taobao.org/@gilt-tech/swig-zk/sync 调用超时了，你可以 grep 一下 All packages sync done 关键词，看看有多少是 fails 的

[xiaqun]

找到很多失败的，这个要怎么处理

[xiaqun]

Sync @gilt-tech/swig-zk complete! But some packages sync failed, you can refresh to sync again. 在web页面安装依赖包的时候一直提示上面的内容

[xiaqun]

2016-10-26 22:46:55.106 nodejs.ResponseErrorException: ResponseError: socket hang up (req "error"), PUT https://registry.npm.taobao.org/@benefex/inactivity-countdown-timer/sync -1 (connected: true, keepalive socket: true, agent status: {"createSocketCount":59672,"closeSocketCount":59549,"errorSocketCount":0,"timeoutSocketCount":1,"requestCount":60253,"freeSockets":{"registry.npm.taobao.org:443::::::::":2},"sockets":{"registry.npm.taobao.org:443::::::::":121},"requests":{}}) headers: {} headers: {} at createHangUpError (_http_client.js:211:15) at TLSSocket.socketOnEnd (_http_client.js:303:23) at emitNone (events.js:72:20) at TLSSocket.emit (events.js:166:7) at endReadableNT (_stream_readable.js:921:12) at nextTickCallbackWith2Args (node.js:442:9) at process._tickCallback (node.js:356:17) pid: 10993 Host: a3-prod-cnpm-128-44.sh URL: Data: '' 2016-10-26 22:46:55.106 2016-10-26 22:49:47.949 nodejs.ResponseTimeoutErrorException: ResponseTimeoutError: Response timeout for 20000ms, PUT https://registry.npm.taobao.org/resmq/sync -1 (connected: true, keepalive socket: false, agent status: {"createSocketCount":60090,"closeSocketCount":59960,"errorSocketCount":0,"timeoutSocketCount":1,"requestCount":60627,"freeSockets":{},"sockets":{"registry.npm.taobao.org:443::::::::":130},"requests":{}}) headers: {} headers: {} at null._onTimeout (/usr/local/cnpmjs.org/node_modules/urllib/lib/urllib.js:701:15) at Timer.listOnTimeout (timers.js:92:15)

[fengmk2]

你们的机房连淘宝这个域名速度如何？看错误是网络问题

Sent from my iPhone

On 26 Oct 2016, at 11:00 PM, 樱花雨 notifications@github.com wrote:

2016-10-26 22:46:55.106 nodejs.ResponseErrorException: ResponseError: socket hang up (req "error"), PUT https://registry.npm.taobao.org/@benefex/inactivity-countdown-timer/sync -1 (connected: true, keepalive socket: true, agent status: {"createSocketCount":59672,"closeSocketCount":59549,"errorSocketCount":0,"timeoutSocketCount":1,"requestCount":60253,"freeSockets":{"registry.npm.taobao.org:443::::::::":2},"sockets":{"registry.npm.taobao.org:443::::::::":121},"requests":{}}) headers: {} headers: {} at createHangUpError (_http_client.js:211:15) at TLSSocket.socketOnEnd (_http_client.js:303:23) at emitNone (events.js:72:20) at TLSSocket.emit (events.js:166:7) at endReadableNT (_stream_readable.js:921:12) at nextTickCallbackWith2Args (node.js:442:9) at process._tickCallback (node.js:356:17) pid: 10993 Host: a3-prod-cnpm-128-44.sh URL: Data: '' 2016-10-26 22:46:55.106 2016-10-26 22:49:47.949 nodejs.ResponseTimeoutErrorException: ResponseTimeoutError: Response timeout for 20000ms, PUT https://registry.npm.taobao.org/resmq/sync -1 (connected: true, keepalive socket: false, agent status: {"createSocketCount":60090,"closeSocketCount":59960,"errorSocketCount":0,"timeoutSocketCount":1,"requestCount":60627,"freeSockets":{},"sockets":{"registry.npm.taobao.org:443::::::::":130},"requests":{}}) headers: {} headers: {} at null._onTimeout (/usr/local/cnpmjs.org/node_modules/urllib/lib/urllib.js:701:15) at Timer.listOnTimeout (timers.js:92:15)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[xiaqun]

公司流量做限制了，丢包了，我把配置改成了syncModel 为all，全部更新结束，但是包还是少了很多 132,060 total packages 485,206 total package versions 0 total delete packages 33 packages need to be sync 100% progress

[fengmk2]

https://github.com/cnpm/cnpmjs.org/blob/master/sync/sync_all.js#L37 all 模式只在第一次的时候会做全量同步，后面都是增量同步。

[xiaqun]

我不记得第一次选择什么了，但是这次没有全量同步过来，只有132,060个包，比淘宝少三分之二 2016-10-27 12:08:44.748 nodejs.SequelizeDatabaseErrorException: SequelizeDatabaseError: ER_DATA_TOO_LONG: Data too long for column 'log' at row 1 at Query.formatError (/usr/local/cnpmjs.org/node_modules/sequelize/lib/dialects/mysql/query.js:175:14) at Query._callback (/usr/local/cnpmjs.org/node_modules/sequelize/lib/dialects/mysql/query.js:49:21) at Query.Sequence.end (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/sequences/Sequence.js:85:24) at Query.ErrorPacket (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/sequences/Query.js:94:8) at Protocol._parsePacket (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/Protocol.js:280:23) at Parser.write (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/Parser.js:74:12) at Protocol.write (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/Protocol.js:39:16) at Socket. (/usr/local/cnpmjs.org/node_modules/mysql/lib/Connection.js:109:28) at emitOne (events.js:77:13) at Socket.emit (events.js:169:7) at readableAddChunk (_stream_readable.js:153:18) at Socket.Readable.push (_stream_readable.js:111:10) at TCP.onread (net.js:536:20)

[fengmk2]

ER_DATA_TOO_LONG: Data too long for column 'log' at row 1 log 字段定义有问题？

[xiaqun]

我是直接从github上拉的代码，只改过index.js里面的配置，其他没有改过

[fengmk2]

默认最大只存 1MB 的 log https://github.com/cnpm/cnpmjs.org/blob/fac9df1de5c7e88d95d8fc59f4d357caf1ff2276/services/module_log.js#L46

能否执行一下 show create table module_log 看看 module_log 表的表结构定义。

[xiaqun]

| module_log | CREATE TABLE module_log ( id int(11) NOT NULL AUTO_INCREMENT, username varchar(100) NOT NULL, name varchar(100) NOT NULL, log text, gmt_create datetime NOT NULL, gmt_modified datetime NOT NULL, PRIMARY KEY (id), KEY module_log_name (name) ) ENGINE=InnoDB AUTO_INCREMENT=64 DEFAULT CHARSET=utf8 COMMENT='module sync log' |

[fengmk2]

@xiaqun 你创建表的时候 sql 写错了？我们提供的 sql 是 longtext 类型的 https://github.com/cnpm/cnpmjs.org/blob/master/docs/db.sql#L95

你改一下看看是否 ok？text 类型默认长度才 64KB http://stackoverflow.com/questions/13932750/tinytext-text-mediumtext-and-longtext-maximum-storage-sizes

      Type | Maximum length
-----------+-------------------------------------
  TINYTEXT |           255 (2 8−1) bytes
      TEXT |        65,535 (216−1) bytes = 64 KiB
MEDIUMTEXT |    16,777,215 (224−1) bytes = 16 MiB
  LONGTEXT | 4,294,967,295 (232−1) bytes =  4 GiB

[xiaqun]

安装是根据这里面的步骤操作的http://blog.fens.me/nodejs-cnpm-npm/ 之后就没有改过配置，但是看里面的sql语句是longtext类型，不知道为什么变成了TEXT 我没有nodejs的基础，里面很多东西不懂

[fengmk2]

那你能否改一下这个 mysql table 的 log 字段类型为 longtext 呢？这个跟 node 无关的。

[xiaqun]

我已经改好了，但是现在还是少包

132,169 total packages 489,306 total package versions 0 total delete packages 0 downloads today 0 downloads in this week 0 downloads in this month 0 downloads in the last day 0 downloads in the last week 0 downloads in the last month Sync Status

This registry will sync all packages from official registry.

Last sync time is Thu Oct 27 2016 14:46:25 GMT+0800 (中国标准时间).

54 packages need to be sync 100% progress 54 packages and dependencies sync successed 0 packages and dependencies sync failed last success: cjdsComponents

[fengmk2]

你得将 total 记录删除了，才能重新进行全量同步。一定进行全量同步，那么你要做好心理准备，要同步超过 1TB 的文件数据。

你可以查询这条记录，你将这条记录删除了，就会执行全量同步了。

select * from total;

[xiaqun]

-------+---------------+----------------+----------------------+-------------+---------------+------------------+---------------+---------------+-------------------+---------------------+ | name | module_delete | last_sync_time | last_exist_sync_time | sync_status | need_sync_num | success_sync_num | fail_sync_num | left_sync_num | last_sync_module | gmt_modified | +-------+---------------+----------------+----------------------+-------------+---------------+------------------+---------------+---------------+-------------------+---------------------+ | total | 0 | 1477551514280 | 0 | 0 | 38 | 38 | 0 | 0 | create-react-repo | 2016-10-25 16:30:15 | +-------+---------------+----------------+----------------------+-------------+---------------+------------------+---------------+---------------+-------------------+---------------------+

是吧need_sync_num 里面的数字改成0吗 我知道数据里很大，我准备了20T的空间

[fengmk2]

你直接删除这行记录就可以了，然后重新运行 sync all 就会从新开始同步了。有 20TB 空间那就没问题的。

[xiaqun]

sync all 怎么执行，我重启服务可以吗

[xiaqun]

2016-10-27 15:16:08.846 nodejs.TypeErrorException: TypeError: Cannot read property 'last_sync_time' of null at checkSyncStatus (/usr/local/cnpmjs.org/sync/index.js:198:25) at next (native) at onFulfilled (/usr/local/cnpmjs.org/node_modules/co/index.js:65:19) pid: 27999 Host: a3-prod-cnpm-128-44.sh URL: Data: '' 2016-10-27 15:16:08.846 2016-10-27 15:16:08.846 nodejs.TypeErrorException: TypeError: Cannot read property 'last_sync_time' of null at checkSyncStatus (/usr/local/cnpmjs.org/sync/index.js:198:25) at next (native) at onFulfilled (/usr/local/cnpmjs.org/node_modules/co/index.js:65:19) pid: 27999 Host: a3-prod-cnpm-128-44.sh URL: Data: '' 2016-10-27 15:16:08.846 2016-10-27 15:26:08.765 nodejs.TypeErrorException: TypeError: Cannot read property 'last_sync_time' of null at checkSyncStatus (/usr/local/cnpmjs.org/sync/index.js:198:25) at next (native) at onFulfilled (/usr/local/cnpmjs.org/node_modules/co/index.js:65:19) pid: 27999 Host: a3-prod-cnpm-128-44.sh URL: Data: '' 2016-10-27 15:26:08.765 2016-10-27 15:28:27.215 nodejs.TypeErrorException: TypeError: Cannot read property 'last_sync_time' of null at checkSyncStatus (/usr/local/cnpmjs.org/sync/index.js:198:25) at next (native) at onFulfilled (/usr/local/cnpmjs.org/node_modules/co/index.js:65:19) pid: 28287 Host: a3-prod-cnpm-128-44.sh URL: Data: '' 2016-10-27 15:28:27.215 2016-10-27 15:28:27.215 nodejs.TypeErrorException: TypeError: Cannot read property 'last_sync_time' of null at checkSyncStatus (/usr/local/cnpmjs.org/sync/index.js:198:25) at next (native) at onFulfilled (/usr/local/cnpmjs.org/node_modules/co/index.js:65:19) pid: 28287 Host: a3-prod-cnpm-128-44.sh URL: Data: '' 2016-10-27 15:28:27.215

现在不能同步了

[fengmk2]

我看看，代码好像是有问题的。

[xiaqun]

commit 74e9bdde725791cf63b45f8804815da7eb8f6ca7 Author: fengmk2 m@fengmk2.com Date: Mon Aug 22 21:51:23 2016 +0800

Release 2.16.1

我用的是这个节点版本

[fengmk2]

@xiaqun 执行一下这个 sql

INSERT INTO total(name, gmt_modified) VALUES('total', now())
  ON DUPLICATE KEY UPDATE gmt_modified=now();

[xiaqun]

现在已经在运行了，谢谢~~~

133,449 total packages 494,254 total package versions 0 total delete packages 0 downloads today 0 downloads in this week 0 downloads in this month 0 downloads in the last day 0 downloads in the last week 0 downloads in the last month Sync Status

This registry will sync all packages from official registry.

Last sync time is Thu Jan 01 1970 08:00:00 GMT+0800 (中国标准时间).

The sync worker is working in the backend now. 369169 packages need to be sync 368463 packages and dependencies waiting for sync 0% progress 702 packages and dependencies sync successed 4 packages and dependencies sync failed last success: @angular/benchpress

[fengmk2]

同步速度得看你们服务到 npm.taobao.org 的网速了。

[xiaqun]

下载百分比一直为0 ，这个有点问题，其他目前没有异常 0% progress

同步估计要一两天，我让公司网络部门不加限制了，现在可以同步

[fengmk2]

@xiaqun 你可以 tail -f $yourlog-dir/sync_info*.log 的日志

全量同步完之后回来告知一下，我看看到底需要多久。

[fengmk2]

web 界面也能看到同步进度的

[xiaqun]

现在web页面有进度了，我之前已经同步了622GB了 剩下的同步完成，我告诉你下

[xiaqun]

今天大概晚上9点同步完成，但是，包的数量比淘宝少了，日志里面没有报错， syncModel: 'all'

[xiaqun]

很感谢帮忙，后期维护的时候遇到问题，可能还需要请教大神~~~ @fengmk2

[xiaqun]

同步ava的时候报错 [2016-10-31 11:00:20] [ava:0] sync error, version: 0.4.0, SequelizeDatabaseError: SequelizeDatabaseError: ER_TRUNCATED_WRONG_VALUE_FOR_FIELD: Incorrect string value: '\xF0\x9F\x9A\x80' for column 'description' at row 1 at Query.formatError (/usr/local/cnpmjs.org/node_modules/sequelize/lib/dialects/mysql/query.js:175:14) at Query._callback (/usr/local/cnpmjs.org/node_modules/sequelize/lib/dialects/mysql/query.js:49:21) at Query.Sequence.end (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/sequences/Sequence.js:85:24) at Query.ErrorPacket (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/sequences/Query.js:94:8) at Protocol._parsePacket (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/Protocol.js:280:23) at Parser.write (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/Parser.js:74:12) at Protocol.write (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/Protocol.js:39:16) at Socket. (/usr/local/cnpmjs.org/node_modules/mysql/lib/Connection.js:109:28) at emitOne (events.js:77:13) at Socket.emit (events.js:169:7) at readableAddChunk (_stream_readable.js:153:18) at Socket.Readable.push (_stream_readable.js:111:10) at TCP.onread (net.js:536:20) [2016-10-31 11:00:20] [ava:1] syncing, version: 0.4.1, dist: {"shasum":"c10da8db1942d7b302fcee839b4a583d0d05fa27","size":12853,"noattachment":false,"tarball":"http://registry.npm.taobao.org/ava/download/ava-0.4.1.tgz"}, no deps: false, publish on cnpm: false, dependencies: 26, devDependencies: 0, syncDevDependencies: false [2016-10-31 11:00:20] [ava:1] sync error, version: 0.4.1, SequelizeDatabaseError: SequelizeDatabaseError: ER_TRUNCATED_WRONG_VALUE_FOR_FIELD: Incorrect string value: '\xF0\x9F\x9A\x80' for column 'description' at row 1 at Query.formatError (/usr/local/cnpmjs.org/node_modules/sequelize/lib/dialects/mysql/query.js:175:14) at Query._callback (/usr/local/cnpmjs.org/node_modules/sequelize/lib/dialects/mysql/query.js:49:21) at Query.Sequence.end (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/sequences/Sequence.js:85:24) at Query.ErrorPacket (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/sequences/Query.js:94:8) at Protocol._parsePacket (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/Protocol.js:280:23) at Parser.write (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/Parser.js:74:12) at Protocol.write (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/Protocol.js:39:16) at Socket. (/usr/local/cnpmjs.org/node_modules/mysql/lib/Connection.js:109:28) at emitOne (events.js:77:13) at Socket.emit (events.js:169:7) at readableAddChunk (_stream_readable.js:153:18) at Socket.Readable.push (_stream_readable.js:111:10) at TCP.onread (net.js:536:20) [2016-10-31 11:00:20] [ava:2] syncing, version: 0.4.2, dist: {"shasum":"38c4bcd616b22bc3e6a08a517c67dbfad2fd05e3","size":13051,"noattachment":false,"tarball":"http://registry.npm.taobao.org/ava/download/ava-0.4.2.tgz"}, no deps: false, publish on cnpm: false, dependencies: 26, devDependencies: 0, syncDevDependencies: false [2016-10-31 11:00:20] add dependencies: co-with-promise [2016-10-31 11:00:20] [ava:2] sync error, version: 0.4.2, SequelizeDatabaseError: SequelizeDatabaseError: ER_TRUNCATED_WRONG_VALUE_FOR_FIELD: Incorrect string value: '\xF0\x9F\x9A\x80' for column 'description' at row 1 at Query.formatError (/usr/local/cnpmjs.org/node_modules/sequelize/lib/dialects/mysql/query.js:175:14) at Query._callback (/usr/local/cnpmjs.org/node_modules/sequelize/lib/dialects/mysql/query.js:49:21) at Query.Sequence.end (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/sequences/Sequence.js:85:24) at Query.ErrorPacket (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/sequences/Query.js:94:8) at Protocol._parsePacket (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/Protocol.js:280:23) at Parser.write (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/Parser.js:74:12) at Protocol.write (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/Protocol.js:39:16) at Socket. (/usr/local/cnpmjs.org/node_modules/mysql/lib/Connection.js:109:28) at emitOne (events.js:77:13) at Socket.emit (events.js:169:7) at readableAddChunk (_stream_readable.js:153:18) at Socket.Readable.push (_stream_readable.js:111:10) at TCP.onread (net.js:536:20) [2016-10-31 11:00:20] [ava:3] syncing, version: 0.5.0, dist: {"shasum":"1018aaaee00733215c2d8eb2d63fe7c5661692c5","size":15313,"noattachment":false,"tarball":"http://registry.npm.taobao.org/ava/download/ava-0.5.0.tgz"}, no deps: false, publish on cnpm: false, dependencies: 28, devDependencies: 0, syncDevDependencies: false [2016-10-31 11:00:20] add dependencies: destroy-circular [2016-10-31 11:00:20] add dependencies: is-generator-fn [2016-10-31 11:00:20] add dependencies: loud-rejection [2016-10-31 11:00:20] add dependencies: max-timeout [2016-10-31 11:00:20] [ava:3] sync error, version: 0.5.0, SequelizeDatabaseError: SequelizeDatabaseError: ER_TRUNCATED_WRONG_VALUE_FOR_FIELD: Incorrect string value: '\xF0\x9F\x9A\x80' for column 'description' at row 1 at Query.formatError (/usr/local/cnpmjs.org/node_modules/sequelize/lib/dialects/mysql/query.js:175:14) at Query._callback (/usr/local/cnpmjs.org/node_modules/sequelize/lib/dialects/mysql/query.js:49:21) at Query.Sequence.end (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/sequences/Sequence.js:85:24) at Query.ErrorPacket (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/sequences/Query.js:94:8) at Protocol._parsePacket (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/Protocol.js:280:23) at Parser.write (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/Parser.js:74:12) at Protocol.write (/usr/local/cnpmjs.org/node_modules/mysql/lib/protocol/Protocol.js:39:16) at Socket. (/usr/local/cnpmjs.org/node_modules/mysql/lib/Connection.js:109:28) at emitOne (events.js:77:13) at Socket.emit (events.js:169:7) at readableAddChunk (_stream_readable.js:153:18) at Socket.Readable.push (_stream_readable.js:111:10) at TCP.onread (net.js:536:20)

[ibigbug]

http://utf8-chartable.de/unicode-utf8-table.pl?start=128640&utf8=string-literal

mysql 存 emoji 需要指定字符集 mb4

[fengmk2]

将所有表的 description 字段字符集改成 utf8mb4

[fengmk2]

https://github.com/cnpm/cnpmjs.org/blob/master/docs/db.sql 这里我也去修复一下好了。

[fengmk2]

要在 Mysql 中保存 4 字节长度的 UTF-8 字符，需要使用 utf8mb4 字符集，但只有 5.5.3 版本以后的才支持(查看版本： select version();)。我觉得，为了获取更好的兼容性，应该总是使用 utf8mb4 而非 utf8. 对于 CHAR 类型数据，utf8mb4 会多消耗一些空间，根据 Mysql 官方建议，使用 VARCHAR 替代 CHAR。

需要确保你的 mysql-server 版本是否正确。

[fengmk2]

@xiaqun 这个异常能看到是对那个表操作爆的异常吗？

[2016-10-31 11:00:20] [ava:3] sync error, version: 0.5.0, SequelizeDatabaseError: SequelizeDatabaseError: ER_TRUNCATED_WRONG_VALUE_FOR_FIELD: Incorrect string value: '\xF0\x9F\x9A\x80' for column 'description' at row 1

[xiaqun]

binary-mirror-config Binary mirror config for prebuild and node-pre-gyp 1.1.0 Known Vulnerabilities 我这边的版本是1.1.0，但是淘宝是那边是1.2.0的，不能同步过来，为什么会出现这种情况

[fengmk2]

@xiaqun https://npm.taobao.org/sync/binary-mirror-config 你替换成你的网站 url，然后将里面的日志贴一下，看看提示是什么。

[xiaqun]

我这个是内网的，你看不了