Improve NPM/Github-polling

[rexxars]

Currently there is a simple setInterval doing some fetching from NPM/Github. I don't like this. It will create trouble when we try to up the number of procs running the server, as well as being sort of ghetto.

Maybe we should make this a cron job which either writes data to disk or to memcached or similar.

[brigand]

Are you open to something with a real database and message queue?

• n workers are running (say 3) idle most of the time
• Cron job publishes refresh message to the queue
• Worker picks up message and fetches the keyword(s) listings from npm
  • Worker publishes npm-refresh message for each pacakge
• Worker picks up npm-refresh message
  • Worker gets and stores new npm data in db
  • Worker publishes gh-refresh message
• Worker picks up gh-refresh message
  • Worker gets and stores new github data in db

It's a very simple, scalable (n processes, n machines), worker stateless solution.

[rexxars]

Absolutely. I've actually done quite a bit of changes to the process, but have not had time to work on it for a while. It's a bit simpler than what you are proposing, though, but it's a starting point.

Let me spend an hour or two on this this evening, and I'll see how far I get. Thanks for the interest - I'll get back to you.

[rexxars]

OK, I had a look at what I had done earlier and it wasn't really working. LevelDB was not a great choice, since you would be reading and writing to the files from separate processes and whatnot.

I decided to go for Redis, instead. Hopefully things should be more stable from now on. Implemented in #23.

Thanks for the interest, @brigand - you made me finally get this out of the way :-)

[brigand]

Awesome, and I love the site, it's very helpful for both publishers and users.