search

elixir-crawly / crawly

Crawly, a high-level web crawling & scraping framework for Elixir.
https://hexdocs.pm/crawly
Apache License 2.0
976 stars 115 forks source link

fix: request_options middleware overriding spider options #208

Closed suzdalnitski closed 2 years ago

suzdalnitski commented 2 years ago

Currently, any options set in the Crawly.Request will get overriden by the RequestOptions middleware. This PR fixes that to allow merging of both request options.

Crawly.Request.new(
  "http://example.com",
  headers,
  # the body option here will get overriden by the middleware from the config
  body: body
)
config :crawly,
  middlewares: [
    {Crawly.Middlewares.RequestOptions, proxy: {'https://proxy.com', 3128}}
  ],
oltarasenko commented 2 years ago

Formatted the code and merged as the other pr