bug: seeing indexing issues with specific package related to conflict

[bcoe]

We seem to be having issues indexing one specific module, react-cent, with the conflict provided below.

Any thoughts regarding what might cause this behavior?

{"pid":26167,"hostname":"elasticsearch-indexer-4-west.internal.npmjs.com","name":"npms-analyzer","level":40,"time":1510078946117,"msg":"Conflict while storing react-cent analysis","err":{"message":"Conflict while storing react-cent analysis","stack":"Error: Conflict while storing react-cent analysis\n    at npmsNano.insertAsync.tap.catch (/mnt/deploys/npms-analyzer-consume/lib/analyze/index.js:82:23)\n    at tryCatcher (/mnt/deploys/npms-analyzer-consume/node_modules/bluebird/js/release/util.js:16:23)\n    at /mnt/deploys/npms-analyzer-consume/node_modules/bluebird/js/release/catch_filter.js:34:37\n    at tryCatcher (/mnt/deploys/npms-analyzer-consume/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/mnt/deploys/npms-analyzer-consume/node_modules/bluebird/js/release/promise.js:510:31)\n    at Promise._settlePromise (/mnt/deploys/npms-analyzer-consume/node_modules/bluebird/js/release/promise.js:567:18)\n    at Promise._settlePromise0 (/mnt/deploys/npms-analyzer-consume/node_modules/bluebird/js/release/promise.js:612:10)\n    at Promise._settlePromises (/mnt/deploys/npms-analyzer-consume/node_modules/bluebird/js/release/promise.js:687:18)\n    at Async._drainQueue (/mnt/deploys/npms-analyzer-consume/node_modules/bluebird/js/release/async.js:138:16)\n    at Async._drainQueues (/mnt/deploys/npms-analyzer-consume/node_modules/bluebird/js/release/async.js:148:10)\n    at Immediate.Async.drainQueues (/mnt/deploys/npms-analyzer-consume/node_modules/bluebird/js/release/async.js:17:14)\n    at runCallback (timers.js:637:20)\n    at tryOnImmediate (timers.js:610:5)\n    at processImmediate [as _immediateCallback] (timers.js:582:5)"},"module":"analyze","v":1}

[satazor]

The error happens when the analysis document for react-cent is being stored concurrently, thus resulting in a couchdb conflict error.

[bcoe]

Gotcha, would it then result in conflicts on future indexer runs? Solution is to just delete the document from CouchDB?

[satazor]

I wonder what is causing react-cent to be analyzed concurrently. Is this something that happen once or multiple times? Can you replicate consistently?

[bcoe]

Aha, this is interesting:

{"pid":26167,"hostname":"elasticsearch-indexer-4-west.internal.npmjs.com","name":"npms-analyzer","level":50,"time":1510091237223,"msg":"Analysis of react-cent failed","err":{"message":"Response code 401 (Unauthorized)","stack":"HTTPError: Response code 401 (Unauthorized)\n    at stream.catch.then.e (/mnt/deploys/npms-analyzer-consume/node_modules/got/index.js:119:13)\nFrom previous event:\n    at EventEmitter.ee.on.res (/mnt/deploys/npms-analyzer-consume/node_modules/got/index.js:104:6)\n    at emitOne (events.js:96:13)\n    at EventEmitter.emit (events.js:188:7)\n    at Immediate.setImmediate (/mnt/deploys/npms-analyzer-consume/node_modules/got/index.js:59:8)\n    at runCallback (timers.js:637:20)\n    at tryOnImmediate (timers.js:610:5)\n    at processImmediate [as _immediateCallback] (timers.js:582:5)","host":"api.github.com","hostname":"api.github.com","method":"GET","path":"/repos/fortis/react-cent/stats/contributors","statusCode":401,"statusMessage":"Unauthorized"},"module":"analyze","v":1}

[satazor]

That's strange. That's not a API key rate limit exceed because the error code is 403.

[bcoe]

seeing the issue, the problem is the deprecated field in package.json:

https://github.com/fortis/react-cent/blob/master/package.json#L28

this then causes the search result to not show up in npm which defaults to adding the search flag not:deprecated we should probably omit this non-standard package.json field when indexing?

[satazor]

I was not counting on the deprecated being false. Anyway, we can change this line https://github.com/npms-io/npms-analyzer/blob/master/lib/scoring/score.js#L209 to be deprecated: collected.metadata.deprecated ? true : null

[bcoe]

@satazor awesome \o/ such a weird bug.