search

stewartmckee / cobweb

Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
MIT License
227 stars 45 forks source link

Cobweb gem causes Rails app to run 10x slower #44

Closed SimonBirrell closed 8 years ago

SimonBirrell commented 8 years ago

Hi,

I have a very bare-bones Rails App, and when I add 

gem 'cobweb'

to the Gemfile, run 'bundle' and restart, then all web requests take around 15s to execute instead of 1s. The weird thing is that I haven't even started calling Cobweb code at all.

There are no errors and MiniProfiler tells me that no time is begin spent in SQL. It's just much slower.

Any idea on where to start looking? Is the Gem initialising somehow, even if I'm not explicitly calling it?

Using Webrick and PostGres. Ruby 2.2.3 and Rails 4.2.4.

Thanks!

Simon

SimonBirrell commented 8 years ago

Whatever it was, I seem to have fixed it by deleting the Gem and reinstalling it. Deleted it using JosephL's advice here:

http://stackoverflow.com/questions/12188833/how-to-reinstall-a-gem-using-bundler

SimonBirrell commented 8 years ago

I take it back. This is still an issue. When the gem is properly reinstalled the web server speed drops from 2s to 50s.

Same if you force Webrick. Same if you use heroku local. I don't know what the gem is doing on initialization - I can't figure out why this should happen.

But it does.

SimonBirrell commented 8 years ago

Another datapoint. If you add the gems that cobweb depends on to the main application Gemfile, there's no problem. It's only when cobweb itself is added.

stewartmckee commented 8 years ago

Thanks Simon,

Cant think off hand what it could be, i'll take a look later and see if i find anything, on a train just now.

Stewart.

stewartmckee commented 8 years ago

had a quick look on the phone, first a couple questions... is it always the same amount of time delay? there are a couple monkey patch bits of code that should be considered. Also do you have a gemfile you are using (and lock) that i can plug into a test rails app, oh and version of rails too. There is a require that loads all rb files that are in the same or sub folders from lib/cobweb.rb, if its missbehaving, could be related. Nothing obvious though, if you can send the above i'll het a test app running to try to replicate. Also, what version of cobweb your on too.

thanks, Stewart.

stewartmckee commented 8 years ago

Thanks, after a bit of wrestling with recent el capitan update, i've been able to get rails installed and can replicate this locally with the rubygems gem. I'm going to now try to replicate with local copy and see what its up to.

stewartmckee commented 8 years ago

Ok, think i've found it, try version 1.0.29.

stewartmckee commented 8 years ago

Evil, Evil, Evil monkey patching on the String class with method_missing....

Glad its working now.

thanks, Stewart.

SimonBirrell commented 8 years ago

Great gem, thanks a lot for the fix!