rubycdp / vessel

Fast high-level web crawling Ruby framework
https://vessel.rubycdp.com
MIT License
645 stars 11 forks source link

Inherit settings #19

Closed milk1000cc closed 1 year ago

milk1000cc commented 2 years ago

I want to inherit the settings of the parent class.

In the current master version, the following code will run in headless mode.

class ApplicationSpider < Vessel::Cargo
  driver :ferrum, headless: false
end

class ExampleCom < ApplicationSpider
  start_urls 'https://example.com/'

  def parse
    puts page.at_css('h1').text
  end
end

ExampleCom.run
Mifrill commented 2 years ago

well, afaik it's good to avoid the using of inheritance in this way. take a look at these notes: https://github.com/serodriguez68/poodr-notes#coding-technique-6-create-shallow-hierarchies By the way, the case when we need shared settings is simpler to handle by this way:

settings = { driver_options: { headless: false } }
ExampleCom.run(settings)
ExampleCom2.run(settings)
milk1000cc commented 2 years ago

I understood it's not a good idea to use inheritance in this way. However, passing the shared settings to the #run method every time doesn't seem like a good idea either.

Is this better way?

class ApplicationSpider < Vessel::Cargo
  DEFAULT_SETTINGS = {
    driver_options: { headless: false }
  }

  def self.settings
    @settings ||= super.merge(DEFAULT_SETTINGS)
  end
end

class ExampleCom < ApplicationSpider
  start_urls 'https://example.com/'

  def parse
    puts page.at_css('h1').text
  end
end
Mifrill commented 2 years ago

Is this better way?

yeah, probably...

Or this way:

      class ApplicationSpider < Vessel::Cargo
        driver :ferrum, headless: false

        def self.settings
          @settings ||= super.merge(ApplicationSpider.settings)
        end
      end

      class ExampleCom < ApplicationSpider
        start_urls 'https://example.com/'

        def parse
          puts page.at_css('h1').text
        end
      end
      ExampleCom.run
milk1000cc commented 2 years ago

Oh, thanks!

route commented 2 years ago

I think I had an idea of inheriting all the settings from parent class, so if it doesn't work it's a bug. But I don't remember because I didn't finish full implementation of the driver mode. I'll check