Concierge process runs out of memory and crashes

[parasdsingh]

In raising this issue, I confirm to the best of my ability the following (please check boxes, e.g. [X]):

• [x] There are no duplicate issues.
• [x] The problem does not lie with an upstream dependency.
• [x] An installed module/integration/service/other is not causing your problem.
• [x] This is reproducible on the latest codebase with up to date NPM dependencies.
• [x] I have added appropriate labels to this issue.

Expected Behaviour: Continues to run indefinitely without using up all system resources Actual Behaviour: I believe the process ended up using all of the system's RAM after running for a week. It was ran in an AWS t2.micro instance inside a screen process. The system was only running concierge and no other user-process. The crash happened after failing to login to facebook for a while (FB had blocked the account) Here's the log.

<--- Last few GCs --->

[11447:0x31c9070] 490403670 ms: Scavenge 739.2 (808.3) -> 725.7 (808.8) MB, 21.2 / 0.0 ms allocation failure [11447:0x31c9070] 490403731 ms: Scavenge 739.8 (808.8) -> 725.5 (809.8) MB, 20.6 / 0.0 ms allocation failure [11447:0x31c9070] 490406480 ms: Scavenge 740.7 (809.8) -> 726.7 (809.8) MB, 21.3 / 0.0 ms allocation failure [11447:0x31c9070] 490406543 ms: Scavenge 740.8 (809.8) -> 726.9 (810.8) MB, 21.7 / 0.0 ms allocation failure

<--- JS stacktrace ---> Cannot get stack trace in GC. FATAL ERROR: NewSpace::Rebalance Allocation failed - process out of memory 1: node::Abort() [/usr/local/bin/node] 2: 0x12b82ac [/usr/local/bin/node] 3: v8::Utils::ReportOOMFailure(char const, bool) [/usr/local/bin/node] 4: v8::internal::V8::FatalProcessOutOfMemory(char const, bool) [/usr/local/bin/node] 5: 0xa98e0b [/usr/local/bin/node] 6: v8::internal::MarkCompactCollector::EvacuateNewSpaceAndCandidates() [/usr/local/bin/node] 7: v8::internal::MarkCompactCollector::CollectGarbage() [/usr/local/bin/node] 8: v8::internal::Heap::MarkCompact() [/usr/local/bin/node] 9: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node] 10: v8::internal::Heap::CollectGarbage(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*, v8::GCCallbackFlags) [/usr/local/bin/node] 11: v8::internal::Heap::HandleGCRequest() [/usr/local/bin/node] 12: v8::internal::StackGuard::HandleInterrupts() [/usr/local/bin/node] 13: v8::internal::Runtime_StackGuard(int, v8::internal::Object*, v8::internal::Isolate) [/usr/local/bin/node] 14: 0x26b539e8ed46 !CORE! critical error was unhandled (null, SIGABRT). Please report this to developers.

Steps to Reproduce: Unsure if this will work, but let facebook login continuously fail for a long time? The bot was started with the debug and log arguments

[drkno]

Ummm.... we had facebook running on AWS for a few months (at least 100 modules installed) without an out of memory crash. Some questions to help us track it down:

• NodeJS version? This looks like Node.JS failing to GC rather than Concierge itself.
• Given you had the log enabled, do you have any more of the log from before this happening?
• How many and which modules were installed?
• Do you have auto_restart installed - on the latest FB integration it is no longer needed and could cause this kind of problem if left for too long.

Some side notes:

• One way to reduce the frequency of account blocking on FB is to marry it to another account
• Have you considered using tmux instead of screen? Its under more active development and objectively provides better features.

[parasdsingh]

• Node version: v7.10.0

• I don't have the log file anymore but I did whizz through and most of it was just messages from the facebook module failing to login

• Here are the installed kpm modules

• config

• help

• ping

• kpm

• restart

• update

• shutdown

• creator

• abstract

• humming

• uptime

• urban

• slap

• lmgtfy

• pun

• pickUp

• joke

• insult

• currency

• cricket

• advice

• 9gag

• boss

• auto_restart

• facebook

• test

• I'll remove auto-restart and see if that makes a difference.

One way to reduce the frequency of account blocking on FB is to marry it to another account

Can you expand more on this 😮

• And yes I'm thinking of switching to tmux though my usage with screen has been very straightforward so far and it's served it's purpose well.

[drkno]

Given auto_restart never gives the event loop an opportunity to GC (its entirely sync) I'm going to put this down to that. If it happens again, re-open the bug.

Can you expand more on this 😮

For some reason we discovered that if your bot account is in a relationship with another account on FB, your bot account will not get banned (when changing countries it will still require a once-off manual verification, e.g. changing your hosting from aws to local if you're not located in the US).