search

elixir-crawly / crawly

Crawly, a high-level web crawling & scraping framework for Elixir.
https://hexdocs.pm/crawly
Apache License 2.0
988 stars 116 forks source link

Does Crawly support requests using the POST method? #281

Closed oallanmendes closed 7 months ago

oallanmendes commented 1 year ago

I've encountered a challenge related to passing the "next_requests" using the POST method. It appears that Crawly.Request doesn't have built-in support for handling HTTP methods. Could someone kindly assist me in resolving this issue?

zastrixarundell commented 9 months ago

The only thing I can think of is probably setting a custom fetcher which can either do post or get methods.

You can look at this implementation for it. The second argument regarding the client_options is only set during the config phase so it's not something you can specify in your crawler on a per-request basis (maybe if you manage to sync it with the override settings config).

I'd recommend putting the method into options and then just switch between post and get.