search

Netflix-Skunkworks / Scumblr

Web framework that allows performing periodic syncs of data sources and performing analysis on the identified results
Apache License 2.0
2.64k stars 317 forks source link

Tasks not completing for the last 8 days #181

Open smoebot opened 7 years ago

smoebot commented 7 years ago

Hi,

I have a relatively new install of Scumblr 2. It was working fine up until about 8 days ago. It appears that tasks aren't completing.

I have 3 groups of tasks Group 1 - 2 tasks that each do a google search for some domain names Group 2 - 1 task that does a Twitter mention search Group 3 - 2 tasks that each do a cURL query against results found in Group 1

Running the tasks manually shows: Group 1: running (6bc03767376739cd9736f212) for eternity

If I disable Group 1, the same behavior is replicated on Group 2. If i disable Group 2, the same behavior is replicated on Group 3.

Web interface seems to be working fine

sbehrens commented 7 years ago

@ahoernecke This may be fixed by the the async-refactor branch?

smoebot commented 7 years ago

Hi, is there any more insight on this one?

ahoernecke commented 7 years ago

Hi @smoebot,

We have been working on a pretty large minor release in the async-refactor branch. The code in that branch isn't and is actively being worked on, but it's what we're using internally and has a number of enhancement and improvements, including to the task handling. You might want to try that branch and see if you get better results.

Have you tried anything that has resolved the issue? (For example, is the issue temporarily resolved if the sidekiq is restarted)? Can you give me an idea of how long each task/group ran before they stopped working properly?

Thanks

smoebot commented 7 years ago

Hi @ahoernecke.

Tried pulling that branch. No dice. Nothing is resolved if sidekiq is restarted.

The task groups seemed to be running fine for about a month until they stopped working.

sbehrens commented 7 years ago

Hi @smoebot,

Did you happen to notice any memory issues (rails or sidekiq taking up most system memory) when the task groups stopped executing?