search

Netflix-Skunkworks / Scumblr

Web framework that allows performing periodic syncs of data sources and performing analysis on the identified results
Apache License 2.0
2.64k stars 317 forks source link

"No URL for result:" Error with Rails Analyzer #164

Open davereber opened 7 years ago

davereber commented 7 years ago

When I try to run the rails analyzer I always get an error for all results stating "No URL for result: ". This occurs for results that I add manually and those that are added via the Github sync. I have my github and tmp download variables set i /config/initializers/scumblr.rb with my GitHub API key and /tmp/. I have been having no luck finding this error documented anywhere.

ahoernecke commented 7 years ago

Hi @davereber,

It looks like the rails analyzer was expecting a field to be added that wasn't be added through the github sync. Can you re-run your github sync then run the rails analyzer again and see if that helps?

davereber commented 7 years ago

Hi @ahoernecke

I just re-ran the github sync. I also ensured that an api.github.com was configured as the api endpoint. I am still getting the same error message. Is there a way to view more detailed logging via the rails console? I am new to using Scumblr.

I did confirm that a URL field exists...but I am assuming that it is looking for something else.

Thanks.

ahoernecke commented 7 years ago

Did you update your code to the latest from master? (sorry forgot to tell you to do so in my original response)?

davereber commented 7 years ago

I did the install (following the wiki instructions) yesterday.

ahoernecke commented 7 years ago

Got it, I pushed a fix for your issue this morning. You'll need to update to the latest code. From your scumblr folder:

git pull origin master

Restart your services (including sidekiq) and rerun the sync.

davereber commented 7 years ago

Looks like it is running :) I will let you know when it finishes the job if there are any errors.

Thank you!

ahoernecke commented 7 years ago

Sounds good, thanks!

davereber commented 7 years ago

Now it seems to execute the Rails Analyzer task, but the job never completes. When I look up in the running tasks it says "No Tasks Running" and I do not see a completed entry in the events. When I try to launch another task nothing will start until I restart the server. I let it run for 4x the amount of time that it took me to manually run brakeman against the same repo.

ahoernecke commented 7 years ago

Hi @davereber,

Any chance you can look at the memory on your system for the time period the scan was running and see if it was being exhausted? We're actively working on fixing some memory issues that would cause this behavior that seemed to be triggered by a combination some inefficient code on our end, a possible memory leak in brakeman, and (at least for us) a couple repos that brakeman really didn't like and would never finished parsing. Your comment about new tasks not launching makes me suspect you may be running into something similar.

We're actively committing to the async-refactor branch working on fixing these memory issues. Obviously it hasn't been promoted to master yet so YMMV, but so far we've seen some pretty significant improvements.