search

juftin / camply

camply, the campsite finder ⛺️ - a tool to find campsites at sold out campgrounds through sites like recreation.gov
https://juftin.com/camply/
MIT License
464 stars 89 forks source link

403 Client Error - ReserveCalifornia blocked on cloud hosting providers #319

Open DavydLiu opened 6 months ago

DavydLiu commented 6 months ago

Describe the bug I got the following error when I run the below command from a linode machine:

HTTPError: 403 Client Error: Forbidden for url: https://calirdr.usedirect.com/rdr/rdr/search/filters

Original Camply Command (with --debug)

camply campsites --provider ReserveCalifornia --campground 597 --campground 598 --start-date 2024-02-10 --end-date 2024-02-11 --search-forever  --notifications pushover --polling-interval 5 --debug

Expected behavior Running the same command on my laptop works fine

Console Output (with --debug)

[2024-01-31 02:00:45] CAMPLY camply, the campsite finder ⛺️
[2024-01-31 02:00:45] DEBUG Setting up camply debugging
[2024-01-31 02:00:45] DEBUG Camply Version: 0.32.1
[2024-01-31 02:00:45] DEBUG Python Version: 3.11.6
[2024-01-31 02:00:45] DEBUG Platform: linux
[2024-01-31 02:00:45] INFO Using Camply Provider: "ReserveCalifornia"
[2024-01-31 02:00:45] INFO 1 booking nights selected for search, ranging from 2024-02-10 to 2024-02-10
[2024-01-31 02:00:45] DEBUG Refreshing UseDirect Metadata: filters.json
[2024-01-31 02:00:45] DEBUG Starting new HTTPS connection (1): calirdr.usedirect.com:443
[2024-01-31 02:00:45] DEBUG https://calirdr.usedirect.com:443 "GET /rdr/rdr/search/filters HTTP/1.1" 403 520
[2024-01-31 02:00:45] CAMPLY Exiting camply 👋

Additional context

juftin commented 6 months ago

Oh no! I'll take a look. Hopefully ReserveCalifornia isn't cracking down on tools like camply

juftin commented 6 months ago

EDIT: I'm seeing that this is only occurring when coming from linode servers and not from local. The command from above worked for me as well.

This might just be an unfortunate blocking of linode IP addresses by the ReserveCalifornia servers. The best I can do in this case is update camply's user agents to a more recent version of fake-useragent

DavydLiu commented 6 months ago

I actually tried Linode and DigitalOcean, both had the same issue.

DavydLiu commented 6 months ago

Tried vultr this morning, still the same error.

juftin commented 6 months ago

😞 I've updated fake-useragent to the latest version and enforced a random User-Agent for each request to ReserveCalifornia. Sadly this is the best I think I'll be able to do - I wouldn't be surprised if ReserveCalifornia universally blocks all major cloud hosting provider's IP ranges.

Let me know if this resolves the issue for you - if not you may want to consider a VPN or similar solution to mask your IP address.

DavydLiu commented 6 months ago

Thanks, let me git it a try. I'm also curious which cloud provider you'd recommend for hosting camply?

juftin commented 6 months ago

I don't have a recommendation sadly - my cloud provider is a small linux server running in my basement that I also use for my homelab. A raspberry pi would probably also work nicely.

This is actually the first I've heard of this cloud provider blocking issue so you'd probably only run into this problem on ReserveCalifornia and not the other camply providers (for now).

DavydLiu commented 6 months ago

Just tried 0.32.2. Unfortunately, it's still blocked for HTTPError: 403 Client Error: Forbidden for url: https://calirdr.usedirect.com/rdr/rdr/search/filters. Recreation.gov works perfectly. I'm curiously what ReserveCalifornia does differently.

Raspberry PI will be my last resort.

justin-outdoorithm commented 6 months ago

I had this issue on Digital Ocean and ended up using a proxy

DavydLiu commented 6 months ago

I had this issue on Digital Ocean and ended up using a proxy

Which proxy did you use?

justin-outdoorithm commented 6 months ago

Smartproxy

I had this issue on Digital Ocean and ended up using a proxy

Which proxy did you use?

Smartproxy

DavydLiu commented 4 months ago

Smartproxy

I had this issue on Digital Ocean and ended up using a proxy

Which proxy did you use?

Smartproxy

Thanks, Justin, do you mind share which proxy you were using. Took a look yesterday but unsure which product I should buy.

justin-outdoorithm commented 4 months ago

I've been using the Residential Pay As You Go plan

On Mon, Apr 1, 2024 at 10:12 AM David Liu @.***> wrote:

Smartproxy

I had this issue on Digital Ocean and ended up using a proxy

Which proxy did you use?

Smartproxy

Thanks, Justin, do you mind share which proxy you were using. Took a look yesterday but unsure which product I should buy.

— Reply to this email directly, view it on GitHub https://github.com/juftin/camply/issues/319#issuecomment-2030177754, or unsubscribe https://github.com/notifications/unsubscribe-auth/BCRKPZBP5OIRUD4TOY7Z6XDY3GIPJAVCNFSM6AAAAABCSIENPGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZQGE3TONZVGQ . You are receiving this because you commented.Message ID: @.***>

botlahan commented 2 weeks ago

I've been using the Residential Pay As You Go plan … On Mon, Apr 1, 2024 at 10:12 AM David Liu @.> wrote: Smartproxy I had this issue on Digital Ocean and ended up using a proxy Which proxy did you use? Smartproxy Thanks, Justin, do you mind share which proxy you were using. Took a look yesterday but unsure which product I should buy. — Reply to this email directly, view it on GitHub <#319 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/BCRKPZBP5OIRUD4TOY7Z6XDY3GIPJAVCNFSM6AAAAABCSIENPGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZQGE3TONZVGQ . You are receiving this because you commented.Message ID: @.>

I’m having the same issue while trying to use camply on a Google cloud virtual console. Any idea on how to setup a third party residential proxy such as smartproxy on Google cloud compute engines?