aio-libs / aiohttp

Asynchronous HTTP client/server framework for asyncio and Python
https://docs.aiohttp.org
Other
15.15k stars 2.02k forks source link

Under mac os, ClientSession fails to request the stream interface. #7987

Closed marconiho closed 11 months ago

marconiho commented 11 months ago

Describe the bug

Under mac os, ClientSession fails to request the stream interface. This problem does not exist under Windows.

To Reproduce

async def stock_intraday_em(symbol: str = "000001") -> pd.DataFrame: code_id_map_em = code_to_em_symbol(symbol) url = "https://70.push2.eastmoney.com/api/qt/stock/details/sse" params = { "fields1": "f1,f2,f3,f4", "fields2": "f51,f52,f53,f54,f55", "mpi": "2000", "ut": "bd1d9ddb04089700cf9c27f6f7426281", "fltt": "2", "pos": "-0", "secid": f"{code_id_map_em}", "wbp2u": "|0|0|0|web", }

semaphore = asyncio.Semaphore(1000)
timeout = aiohttp.ClientTimeout(total=30)
async with semaphore:
    async with aiohttp.ClientSession(timeout=timeout) as session:
        async with session.get(url, params=params) as res:

            print(res)  # The problem lies here
            big_df = pd.DataFrame()  # 创建一个空的 DataFrame
            data = b''
            async for chunk in res.content:
                if chunk != b'\n':
                    data += chunk
                    event = chunk.decode()
                    event_json = json.loads(event.replace("data: ", ""))
                    temp_df = pd.DataFrame(
                        [item.split(",") for item in event_json["data"]["details"]]
                    )
                    big_df = pd.concat([big_df, temp_df], ignore_index=True)
                    break
            return big_df

df = await stock_intraday_em('300498')

Expected behavior

fix this bug

Logs/tracebacks

none

Python Version

$ python --version
3.11

aiohttp Version

$ python -m pip show aiohttp
Name: aiohttp
Version: 3.9.1
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: 
Author-email: 
License: Apache 2
Location: /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages
Requires: aiosignal, attrs, frozenlist, multidict, yarl
Required-by: aiohttp-socks

multidict Version

$ python -m pip show multidict
Name: multidict
Version: 6.0.4
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages
Requires: 
Required-by: aiohttp, yarl

yarl Version

$ python -m pip show yarl
Name: yarl
Version: 1.9.2
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache-2.0
Location: /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages
Requires: idna, multidict
Required-by: aiohttp

OS

macOs Sonoma

Related component

Client

Additional context

No response

Code of Conduct

Dreamsorcerer commented 11 months ago

There's pretty much nothing to go on from your report. What actually happens on Mac OS? Do you get a response, or it hits a timeout or what?

The code appears to just do a simple GET request, so it's unlikely an aiohttp problem... There could be an issue with a firewall or any number of things.

marconiho commented 11 months ago

async with session.get(url, params=params) as res There is no response when the program runs to this line of code.But in Windows, I can receive the expected results normally.Or maybe I can receive the results in a different way even under macOS,Like this: ` def __event_stream(url, params):

response = requests.get(url, params=params, stream=True)
event_data = ""
for line in response.iter_lines():
    if line:
        event_data += line.decode() + "\n"
    elif event_data:
        yield event_data
        event_data = ""

code_id_map_em = code_to_em_symbol('300498') url = "https://70.push2.eastmoney.com/api/qt/stock/details/sse" params = { "fields1": "f1,f2,f3,f4", "fields2": "f51,f52,f53,f54,f55", "mpi": "2000", "ut": "bd1d9ddb04089700cf9c27f6f7426281", "fltt": "2", "pos": "-0", "secid": f"{code_id_map_em}", "wbp2u": "|0|0|0|web", } big_df = pd.DataFrame() for event in __event_stream(url, params): event_json = json.loads(event.replace("data: ", "")) temp_df = pd.DataFrame( [item.split(",") for item in event_json["data"]["details"]] ) big_df = pd.concat([big_df, temp_df], ignore_index=True) break print(big_df) `

Dreamsorcerer commented 11 months ago

But, what does "no response" mean? What actually happens? Either res has been set to something, or you would get an exception (maybe a timeout after waiting a bit).

marconiho commented 11 months ago

I found that there was an issue with adding an HTTPS request, and when I switched to an HTTP request, it worked fine