jsha
commented
9 years ago This was fixed but an upgrade to Node v0.12. It turned out Node v0.10 had an issue with some timer-checking method when there were a large number of TCP connections open: it was n^2 instead of linear or something like that. stream.js is now at around 10% CPU for both processes.
Right now stream.js takes up about 50% CPU across two processes, and more when there's a lot of traffic. It processes about 3700 TCP streams for 3700 users, but most users aren't receiving any data at any given time.
Profile stream.js and figure out where the CPU consumption is. Is it in TCP handling? JSON parsing? If it's JSON parsing, maybe avoid node-twitter-api, go straight to the oauth library, and filter JSON lines for interesting topics (replies, blocks, unblocks) before parsing them.