search

dominictarr / npmd

MIT License
450 stars 37 forks source link

Crash on first sync #30

Closed wraithan closed 10 years ago

wraithan commented 10 years ago

I just globally installed npmd and I tried running npmd --sync and I get the below stacktrace.

$ npmd --sync
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
TypeError: Cannot read property 'prerelease' of null
    at exports.pad (/usr/local/lib/node_modules/npmd/node_modules/padded-semver/index.js:28:24)
    at /usr/local/lib/node_modules/npmd/plugins/couch-sync.js:65:34
    at /usr/local/lib/node_modules/npmd/node_modules/level-couch-sync/index.js:84:7
    at Feed.onfollow (/usr/local/lib/node_modules/npmd/node_modules/level-couch-sync/index.js:113:7)
    at Feed.<anonymous> (/usr/local/lib/node_modules/npmd/node_modules/level-couch-sync/node_modules/follow/api.js:23:55)
    at Feed.EventEmitter.emit (events.js:95:17)
    at Feed.on_good_change (/usr/local/lib/node_modules/npmd/node_modules/level-couch-sync/node_modules/follow/lib/feed.js:563:8)
    at Feed.on_change (/usr/local/lib/node_modules/npmd/node_modules/level-couch-sync/node_modules/follow/lib/feed.js:526:17)
    at Feed.on_couch_data (/usr/local/lib/node_modules/npmd/node_modules/level-couch-sync/node_modules/follow/lib/feed.js:410:8)
    at Changes.handle_confirmed_req_event (/usr/local/lib/node_modules/npmd/node_modules/level-couch-sync/node_modules/follow/lib/feed.js:290:30)

/usr/local/lib/node_modules/npmd/plugins/couch-sync.js:80
        throw err
              ^
TypeError: Cannot read property 'prerelease' of null
    at exports.pad (/usr/local/lib/node_modules/npmd/node_modules/padded-semver/index.js:28:24)
    at /usr/local/lib/node_modules/npmd/plugins/couch-sync.js:65:34
    at /usr/local/lib/node_modules/npmd/node_modules/level-couch-sync/index.js:84:7
    at Feed.onfollow (/usr/local/lib/node_modules/npmd/node_modules/level-couch-sync/index.js:113:7)
    at Feed.<anonymous> (/usr/local/lib/node_modules/npmd/node_modules/level-couch-sync/node_modules/follow/api.js:23:55)
    at Feed.EventEmitter.emit (events.js:95:17)
    at Feed.on_good_change (/usr/local/lib/node_modules/npmd/node_modules/level-couch-sync/node_modules/follow/lib/feed.js:563:8)
    at Feed.on_change (/usr/local/lib/node_modules/npmd/node_modules/level-couch-sync/node_modules/follow/lib/feed.js:526:17)
    at Feed.on_couch_data (/usr/local/lib/node_modules/npmd/node_modules/level-couch-sync/node_modules/follow/lib/feed.js:410:8)
    at Changes.handle_confirmed_req_event (/usr/local/lib/node_modules/npmd/node_modules/level-couch-sync/node_modules/follow/lib/feed.js:290:30)

I have tried rerunning the command, and I tried deleting my ~/.npmd and resyncing everything and still got this stacktrace.

ceejbot commented 10 years ago

Just came here to report the same problem. Adding a little logging reveals that semver.parse() is returning null when asked to parse version strings like 0.6.0dev2.

ceejbot commented 10 years ago

Patch to padded-semver incoming.

zhuangya commented 10 years ago

same here.

megamaddu commented 10 years ago

Not sure if this is related at all, but on Windows it gets to 100% and then spins forever (or at least for 12+ hours). It even printed 100.12 and 100.18 at some point. I retried in verbose mode, where it started dumping module descriptions (very slowly). There seemed to be no end so I eventually terminated it.. Here's the output I had at that point.

npmdsyncverbose

ceejbot commented 10 years ago

That's normal operation, I believe. Dominic has patched the bug reported here in his latest version, so what you're seeing is what happens when it chugs through all of the nom metadata. I think.

megamaddu commented 10 years ago

12+ hours can't be normal operation. There's also no chugging going on, as CPU is at 0%. Memory usage seemed a little high, at 1.3 GB, but it wasn't really leaking beyond that (at least not very fast).

dominictarr commented 10 years ago

this is the intended behavior. It's runs forever in the background, first syncing old modules, then syncing new ones as they are published. If you let it run the the background then your local database will always be up to date.

Once it is synced the modules it's logging are the modules that are being updated. It's only slow because people are not writing modules fast enough. :(

There is a memory thing, that is really a problem about scheduling leveldb map-reduce and indexing stuff... which is a separate problem. leveldb seems bad at not deallocating memory.

The real problem here how can I make the output and the documentation clearer so that the intended behavior is more obvious?

megamaddu commented 10 years ago

Ohhh I see. I was thinking I needed to wait for the initial sync to finish before I could use it.

balupton commented 10 years ago

Crashed on the initial sync here too;

|....:....|....:....|....:....|....:....|....:....| 65.20%   
|....:....|....:....|....:....|....:....|....:....| 62.96%   

/Users/balupton/.nvm/v0.10.22/lib/node_modules/npmd/plugins/couch-sync.js:27
      if(doc._deleted) return
            ^
TypeError: Cannot read property '_deleted' of null
    at /Users/balupton/.nvm/v0.10.22/lib/node_modules/npmd/plugins/couch-sync.js:27:13
    at /Users/balupton/.nvm/v0.10.22/lib/node_modules/npmd/node_modules/level-couch-sync/index.js:96:7
    at Feed.onfollow (/Users/balupton/.nvm/v0.10.22/lib/node_modules/npmd/node_modules/level-couch-sync/index.js:128:7)
    at Feed.<anonymous> (/Users/balupton/.nvm/v0.10.22/lib/node_modules/npmd/node_modules/level-couch-sync/node_modules/follow/api.js:23:55)
    at Feed.EventEmitter.emit (events.js:95:17)
    at Feed.on_good_change (/Users/balupton/.nvm/v0.10.22/lib/node_modules/npmd/node_modules/level-couch-sync/node_modules/follow/lib/feed.js:563:8)
    at Feed.on_change (/Users/balupton/.nvm/v0.10.22/lib/node_modules/npmd/node_modules/level-couch-sync/node_modules/follow/lib/feed.js:526:17)
    at Feed.on_couch_data (/Users/balupton/.nvm/v0.10.22/lib/node_modules/npmd/node_modules/level-couch-sync/node_modules/follow/lib/feed.js:410:8)
    at Changes.handle_confirmed_req_event (/Users/balupton/.nvm/v0.10.22/lib/node_modules/npmd/node_modules/level-couch-sync/node_modules/follow/lib/feed.js:290:30)
    at Changes.EventEmitter.emit (events.js:95:17)
dominictarr commented 10 years ago

that is strange, okay, added a check for that... try npmd@0.20.1 or later.

revathskumar commented 10 years ago

@dominictarr I didn't find npmd@0.20.1 on npm/github.

dominictarr commented 10 years ago

pushed!

dominictarr commented 10 years ago

In the new rewrite this will all be reorganized so that syncing is optional, and so you can install a module without syncing first. (it will request couch docs and store them in the cache like npm classic)

The main thing that will change is how the cache works, so you'll be able to offline install from cache even for modules that depend on a url (i.e. a git or http url)

wraithan commented 10 years ago

@dominictarr Installed the latest version and --sync is working for me now without crashing, so I'll go ahead and close this ticket.