search

elixir-crawly / crawly

Crawly, a high-level web crawling & scraping framework for Elixir.
https://hexdocs.pm/crawly
Apache License 2.0
953 stars 112 forks source link

Does Crawly support requests using the POST method? #281

Closed oallanmendes closed 3 months ago

oallanmendes commented 8 months ago

I've encountered a challenge related to passing the "next_requests" using the POST method. It appears that Crawly.Request doesn't have built-in support for handling HTTP methods. Could someone kindly assist me in resolving this issue?

zastrixarundell commented 5 months ago

The only thing I can think of is probably setting a custom fetcher which can either do post or get methods.

You can look at this implementation for it. The second argument regarding the client_options is only set during the config phase so it's not something you can specify in your crawler on a per-request basis (maybe if you manage to sync it with the override settings config).

I'd recommend putting the method into options and then just switch between post and get.