Server returned a status code of 403 while downloading https://poe. com

Akut0 commented 1 year ago

INFO:root:Setting up session...
INFO:root:Downloading next_data...
WARNING:root:Server returned a status code of 403 while downloading https://poe.
com. Retrying (1/10)...
WARNING:root:Server returned a status code of 403 while downloading https://poe.
com. Retrying (2/10)...
WARNING:root:Server returned a status code of 403 while downloading https://poe.
com. Retrying (3/10)...
WARNING:root:Server returned a status code of 403 while downloading https://poe.
com. Retrying (4/10)...
WARNING:root:Server returned a status code of 403 while downloading https://poe.
com. Retrying (5/10)...
WARNING:root:Server returned a status code of 403 while downloading https://poe.
com. Retrying (6/10)...
WARNING:root:Server returned a status code of 403 while downloading https://poe.
com. Retrying (7/10)...
WARNING:root:Server returned a status code of 403 while downloading https://poe.
com. Retrying (8/10)...
WARNING:root:Server returned a status code of 403 while downloading https://poe.
com. Retrying (9/10)...
WARNING:root:Server returned a status code of 403 while downloading https://poe.
com. Retrying (10/10)...
Traceback (most recent call last):
  File "start_bot.py", line 4, in <module>
    from plugin_chatgpt import GPT
  File "C:\MM Bot\plugin_chatgpt.py", line 74, in <module>
    poe_client = poe.Client(poe_token)
  File "C:\Python390\lib\site-packages\poe\__init__.py", line 156, in __init__
    self.connect_ws()
  File "C:\Python390\lib\site-packages\poe\__init__.py", line 446, in connect_ws

    self.setup_connection()
  File "C:\Python390\lib\site-packages\poe\__init__.py", line 182, in setup_conn
ection
    self.next_data = self.get_next_data(overwrite_vars=True)
  File "C:\Python390\lib\site-packages\poe\__init__.py", line 244, in get_next_d
ata
    r = request_with_retries(self.session.get, self.home_url)
  File "C:\Python390\lib\site-packages\poe\__init__.py", line 84, in request_wit
h_retries
    raise RuntimeError(f"Failed to download {url} too many times.")
RuntimeError: Failed to download https://poe.com too many times.

Toaster496 commented 1 year ago

same here

cmldyu commented 1 year ago

same problem

445022889 commented 1 year ago

一样的遭遇。

media1demo commented 1 year ago

yeah same here RuntimeError: Failed to download https://poe.com too many times.

bolavefasfas commented 1 year ago

Its hapenning on older versions as well i was in v0.4.x Its sad i had lots of texts to summerize

freQuensy23-coder commented 1 year ago

Same problem(

ghost commented 1 year ago

same here!

Milou4Dev commented 1 year ago

SAME

chaney1030 commented 1 year ago

same

Hafi-Ansari commented 1 year ago

Unfortunately same :(

double-zinc commented 1 year ago

same here :(

31702160136 commented 1 year ago

我经过排查后发现使用GET访问https://poe.com提示403，Enable JavaScript and cookies to continue。我使用selenium库去访问https://poe.com可以成功获取到数据，但需要提前写入cookie：p-b、__cf_bm、cf_clearance才可以

Hafi-Ansari commented 1 year ago

"After investigating, I found that accessing https://poe.com/ using GET method prompts a 403 error, 'Enable JavaScript and cookies to continue.' I can successfully retrieve data by using the Selenium library to access https://poe.com/, but it requires writing the cookies: p-b, __cf_bm, and cf_clearance in advance."

is what @31702160136 is saying, but what exactly does this mean?

31702160136 commented 1 year ago

"After investigating, I found that accessing https://poe.com/ using GET method prompts a 403 error, 'Enable JavaScript and cookies to continue.' I can successfully retrieve data by using the Selenium library to access https://poe.com/, but it requires writing the cookies: p-b, __cf_bm, and cf_clearance in advance."

is what @31702160136 is saying, but what exactly does this mean?

我对这方面不太熟悉，我认为是需要加载JavaScript后才可以获得到数据，我们通常使用代码进行GET请求页面是无法加载JavaScript的，但是使用浏览器就可以加载JavaScript

Hafi-Ansari commented 1 year ago

"After investigating, I found that accessing https://poe.com/ using GET method prompts a 403 error, 'Enable JavaScript and cookies to continue.' I can successfully retrieve data by using the Selenium library to access https://poe.com/, but it requires writing the cookies: p-b, __cf_bm, and cf_clearance in advance." is what @31702160136 is saying, but what exactly does this mean?

我对这方面不太熟悉，我认为是需要加载JavaScript后才可以获得到数据，我们通常使用代码进行GET请求页面是无法加载JavaScript的，但是使用浏览器就可以加载JavaScript

Have you confirmed that this method works?

445022889 commented 1 year ago

我经过排查后发现使用GET访问https://poe.com提示403，Enable JavaScript and cookies to continue。我使用selenium库去访问https://poe.com可以成功获取到数据，但需要提前写入cookie：p-b、__cf_bm、cf_clearance才可以

昨天用golang的chromedp访问好像也没问题，只需要加入p-b就行

31702160136 commented 1 year ago

我经过排查后发现使用GET访问https://poe.com提示403，启用JavaScript和cookie才能继续。我使用selenium去库访问https://poe.com可以成功获取到数据，但需要提前写入cookie ：pb、__cf_bm、cf_clearance才可以

昨天用golang的chromedp访问没什么问题，只需要加入pb就行

我试了一下，确实只需要pb，之前我需要加入__cf_bm和cf_clearance是因为浏览器有缓存

31702160136 commented 1 year ago

“经过调查，我发现使用 GET 方法访问https://poe.com/会提示 403 错误，'启用 JavaScript 和 cookies 即可继续'。我可以使用 Selenium 库访问https://poe.com/成功检索数据，但需要提前写入cookie：pb、__cf_bm 和 cf_clearance。” 是什么@31702160136是这么说的，但这到底是什么意思呢？

我对这方面还不太熟悉，我认为是需要加载JavaScript后才能获得数据，通常使用我们的代码进行GET请求页面是无法加载JavaScript的，但是使用浏览器就可以加载JavaScript

你确认这个方法有效吗？

确实能有效获得到数据，但是又遇到了新的问题，提示’document‘ is not defined，可能是formkey算法变了

tejas36026 commented 1 year ago

me too getting the same

mahmuod711 commented 1 year ago

Same Problem

YUFOHON commented 1 year ago

same

sycapple commented 1 year ago

same

RamyGomaa commented 1 year ago

same on an older version too, looks like it's a general problem, not a version specific one.

asce0 commented 1 year ago

does anyone get any idea about how to solve this error?

kraravind commented 1 year ago

getting the same issue, i went to poe, got the token value through inspect and ran the poe.Client and passed the token. This is the error below, please help: File ~\anaconda3\Lib\site-packages\poe__init.py:156, in Client.init__(self, token, proxy, headers, device_id, client_identifier, formkey) 153 self.formkey_salt = None 154 self.formkey = formkey --> 156 self.connect_ws()

File ~\anaconda3\Lib\site-packages\poe__init__.py:446, in Client.connect_ws(self, timeout) 443 self.ws_connected = False 445 if self.connect_count % 5 == 0: --> 446 self.setup_connection() 448 self.connect_count += 1 450 ws = websocket.WebSocketApp( 451 self.get_websocket_url(), 452 header={"User-Agent": user_agent}, (...) 456 on_close=self.on_ws_close 457 )

File ~\anaconda3\Lib\site-packages\poe__init__.py:182, in Client.setup_connection(self) 179 self.setup_count += 1 181 self.ws_domain = f"tch{random.randint(1, 1e6)}" --> 182 self.next_data = self.get_next_data(overwrite_vars=True) 183 self.channel = self.get_channel_data() 185 self.gql_headers = { 186 "poe-formkey": self.formkey, 187 "poe-tchannel": self.channel["channel"], 188 }

File ~\anaconda3\Lib\site-packages\poe__init.py:244, in Client.get_next_data(self, overwrite_vars) 241 def get_next_data(self, overwrite_vars=False): 242 logger.info("Downloading next_data...") --> 244 r = request_with_retries(self.session.get, self.home_url) 245 json_regex = r'<script id="__NEXT_DATA" type="application\/json">(.+?)' 246 json_text = re.search(json_regex, r.text).group(1)

File ~\anaconda3\Lib\site-packages\poe__init__.py:84, in request_with_retries(method, *args, **kwargs) 81 raise RuntimeError("Invalid or missing token.") 82 logger.warn(f"Server returned a status code of {r.status_code} while downloading {url}. Retrying ({i+1}/{attempts})...") ---> 84 raise RuntimeError(f"Failed to download {url} too many times.")

RuntimeError: Failed to download https://poe.com/ too many times.

mak448a commented 1 year ago

Please stop bombing this thread unless if you actually have more information. It'll make it easier on the developers. Also, if you have the same issue, place a thumbs up on the first comment.

teclectic commented 1 year ago

Hi Guys,

Started getting this error on Saturday, I sent an email to Poe and they said that the specific IP has been blocked for 7 days. I have tried using my same token from another IP, I tried getting another account with a different token and all have come up with the same 403 error. I think that they must have some antiscraping software running which is flagging the API requests. Has anyone tried using sock5 proxies ? Any better ?

bolavefasfas commented 1 year ago

Hi Guys,

Started getting this error on Saturday, I sent an email to Poe and they said that the specific IP has been blocked for 7 days. I have tried using my same token from another IP, I tried getting another account with a different token and all have come up with the same 403 error. I think that they must have some antiscraping software running which is flagging the API requests. Has anyone tried using sock5 proxies ? Any better ?

Did you try new account with new ip ? And if they are banning or they have some mechanism in place and even if new ip with new account works it will be useless as it will be limited again right ?

I have tried with diffrent ip and diffrent account key still the error

4781121 commented 1 year ago

作者也不出来说两句, 也不知道是能解决还是不能解决

akl7777777 commented 1 year ago

解决应该是能解决,但是一解决就会被盯上,估计也是难

teclectic commented 1 year ago

Hi Guys, Started getting this error on Saturday, I sent an email to Poe and they said that the specific IP has been blocked for 7 days. I have tried using my same token from another IP, I tried getting another account with a different token and all have come up with the same 403 error. I think that they must have some antiscraping software running which is flagging the API requests. Has anyone tried using sock5 proxies ? Any better ?

Did you try new account with new ip ? And if they are banning or they have some mechanism in place and even if new ip with new account works it will be useless as it will be limited again right ?

I have tried with diffrent ip and diffrent account key still the error

Yes still get the same error. Any luck with proxies or selenium?

ghost commented 1 year ago

Hi Guys, Started getting this error on Saturday, I sent an email to Poe and they said that the specific IP has been blocked for 7 days. I have tried using my same token from another IP, I tried getting another account with a different token and all have come up with the same 403 error. I think that they must have some antiscraping software running which is flagging the API requests. Has anyone tried using sock5 proxies ? Any better ?

Did you try new account with new ip ? And if they are banning or they have some mechanism in place and even if new ip with new account works it will be useless as it will be limited again right ? I have tried with diffrent ip and diffrent account key still the error

Yes still get the same error. Any luck with proxies or selenium?

have you tried to set a Custom Device ID with new IP and account ? from the README:

Setting a Custom Device ID:

If you want to change the device ID that is being spoofed, you can use the poe.set_device_id, which accepts the following arguments:
user_id - The user ID of the account you want to change the device ID for. The user ID can be found at client.viewer["poeUser"]["id"].
device_id - The new device ID. This is a 32 character UUID string in the following format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
poe.set_device_id("UGMlVXqlcLYyMOATMDsKNTMz", "6d659b04-043a-41f8-97c7-fb7d7fe9ad34")

The device IDs are saved to ~/.config/poe-api/device_id.json on Unix-like systems, and C:\Users\\AppData\Roaming\poe-api\device_id.json on Windows.

Additionally, the poe.get_device_id function or client.device_id can be used to retrieve the saved device ID.

poe.get_device_id("UGMlVXqlcLYyMOATMDsKNTMz")

6d659b04-043a-41f8-97c7-fb7d7fe9ad34

client.device_id

6d659b04-043a-41f8-97c7-fb7d7fe9ad34"

pythonhook commented 1 year ago

How to resolve this, guys

wangyu1997 commented 1 year ago

感觉解决不了放弃了一个月了作者好像没有要解决的意思换个库吧兄弟

pythonhook commented 1 year ago

有什么好的库推荐下呗

---Original--- From: "Wang @.> Date: Wed, Aug 9, 2023 15:20 PM To: @.>; Cc: @.**@.>; Subject: Re: [ading2210/poe-api] Server returned a status code of 403 whiledownloading https://poe. com (Issue #218)

感觉解决不了放弃了一个月了作者好像没有要解决的意思换个库吧兄弟

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

445022889 commented 1 year ago

目前自己golang用的无头浏览器，劫持ws消息，勉强能用吧，不用一堆js，希望作者早日修复bug

有什么好的库推荐下呗 … ---Original--- From: "Wang @.> Date: Wed, Aug 9, 2023 15:20 PM To: @.>; Cc: @.**@.>; Subject: Re: [ading2210/poe-api] Server returned a status code of 403 whiledownloading https://poe. com (Issue #218) 感觉解决不了放弃了一个月了作者好像没有要解决的意思换个库吧兄弟 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

teclectic commented 1 year ago

Is it worth building the headless selenium solution in your opinion into poe-api or otherwise? Did you also use proxies?

On Wed, 9 Aug 2023, 09:18 Djing, @.***> wrote:

目前自己golang用的无头浏览器，劫持ws消息，勉强能用吧，不用一堆js，希望作者早日修复bug

有什么好的库推荐下呗 … <#m-2824118766624701009> ---Original--- From: "Wang @.> Date: Wed, Aug 9, 2023 15:20 PM To: @.>; Cc: *@@.*>; Subject: Re: [ading2210/poe-api] Server returned a status code of 403 whiledownloading https://poe. com (Issue #218 https://github.com/ading2210/poe-api/issues/218) 感觉解决不了放弃了一个月了作者好像没有要解决的意思换个库吧兄弟 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @ .>

— Reply to this email directly, view it on GitHub https://github.com/ading2210/poe-api/issues/218#issuecomment-1670880196, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHHFWJ22NPLATJYFQPWGZZ3XUNBUJANCNFSM6AAAAAA3ELAPUI . You are receiving this because you commented.Message ID: @.***>

ongspxm commented 1 year ago

curl 'https://poe.com/' \
  -H 'authority: poe.com' \
  -H 'cookie: p-b=<token>' \
  -H 'user-agent: <copy user-agent>' --http2

after testing with curl

found that both "authority" and "user-agent" header required
http1.1 doesnt work (tested with --http1.1 on curl and it gave the error), http2 does

but...

requests is a 1.1 only library, maybe httpx might work?

can anybody confirm this?

claell commented 1 year ago

@ading2210 You added the help wanted label. Is there anything you need right now? Or are you just busy with other stuff right now?

huyremy commented 1 year ago

@ading2210 You added the help wanted label. Is there anything you need right now? Or are you just busy with other stuff right now?

and...I feel something like dead project.

fanxyu commented 1 year ago

希望作者能注意到这个问题，解决一下

ading2210 commented 1 year ago

Sorry for the lack of an update, but I'm currently busy right now and won't be able to work on this for at least a week. Although if someone submits a PR that fixes this issue then I'll merge it and publish a new release.

ading2210 commented 1 year ago

My current understanding of this bug is that Poe has started enforcing some sort of cloudflare-based verification which requires a full web browser.

claell commented 1 year ago

Alright, thanks for the update! If your understanding is correct, that would be rather sad to see.

@canxin121 I assume you experience the same problem with your async client?

canxin121 commented 1 year ago

not the same promblem, i can successfuly download the poe.com, but i can't send query successfullt.

canxin121 commented 1 year ago

async poe client didn't have this problem, but it could't extarct formkey, "document is not defined", it requires a browser environment to run the script? I could't find a way to do that except for playwright. So i set formkey a must currently.But all other fuctions run well.

mak448a commented 1 year ago

My current understanding of this bug is that Poe has started enforcing some sort of cloudflare-based verification which requires a full web browser.

Maybe selenium would work for this project?

claell commented 1 year ago

@mak448a see #195

claell commented 1 year ago

@canxin121 That's interesting findings. Apparently, a web browser might only be required to retrieve the formkey, then (if even that)? That might leave some room for a hybrid solution where the formkey can get extracted through browser automation and following that the API can still be used.

4781121 commented 1 year ago

@ading2210 可以参考这个项目, 他解决了目前的问题, https://github.com/canxin121/Async-Poe-Client

ading2210 / poe-api

Server returned a status code of 403 while downloading https://poe. com #218

6d659b04-043a-41f8-97c7-fb7d7fe9ad34

6d659b04-043a-41f8-97c7-fb7d7fe9ad34"