aio-libs / aiohttp

Asynchronous HTTP client/server framework for asyncio and Python
https://docs.aiohttp.org
Other
15.03k stars 2k forks source link

Under mac os, ClientSession fails to request the stream interface. #7987

Closed marconiho closed 9 months ago

marconiho commented 9 months ago

Describe the bug

Under mac os, ClientSession fails to request the stream interface. This problem does not exist under Windows.

To Reproduce

async def stock_intraday_em(symbol: str = "000001") -> pd.DataFrame: code_id_map_em = code_to_em_symbol(symbol) url = "https://70.push2.eastmoney.com/api/qt/stock/details/sse" params = { "fields1": "f1,f2,f3,f4", "fields2": "f51,f52,f53,f54,f55", "mpi": "2000", "ut": "bd1d9ddb04089700cf9c27f6f7426281", "fltt": "2", "pos": "-0", "secid": f"{code_id_map_em}", "wbp2u": "|0|0|0|web", }

semaphore = asyncio.Semaphore(1000)
timeout = aiohttp.ClientTimeout(total=30)
async with semaphore:
    async with aiohttp.ClientSession(timeout=timeout) as session:
        async with session.get(url, params=params) as res:

            print(res)  # The problem lies here
            big_df = pd.DataFrame()  # 创建一个空的 DataFrame
            data = b''
            async for chunk in res.content:
                if chunk != b'\n':
                    data += chunk
                    event = chunk.decode()
                    event_json = json.loads(event.replace("data: ", ""))
                    temp_df = pd.DataFrame(
                        [item.split(",") for item in event_json["data"]["details"]]
                    )
                    big_df = pd.concat([big_df, temp_df], ignore_index=True)
                    break
            return big_df

df = await stock_intraday_em('300498')

Expected behavior

fix this bug

Logs/tracebacks

none

Python Version

$ python --version
3.11

aiohttp Version

$ python -m pip show aiohttp
Name: aiohttp
Version: 3.9.1
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: 
Author-email: 
License: Apache 2
Location: /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages
Requires: aiosignal, attrs, frozenlist, multidict, yarl
Required-by: aiohttp-socks

multidict Version

$ python -m pip show multidict
Name: multidict
Version: 6.0.4
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages
Requires: 
Required-by: aiohttp, yarl

yarl Version

$ python -m pip show yarl
Name: yarl
Version: 1.9.2
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache-2.0
Location: /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages
Requires: idna, multidict
Required-by: aiohttp

OS

macOs Sonoma

Related component

Client

Additional context

No response

Code of Conduct

Dreamsorcerer commented 9 months ago

There's pretty much nothing to go on from your report. What actually happens on Mac OS? Do you get a response, or it hits a timeout or what?

The code appears to just do a simple GET request, so it's unlikely an aiohttp problem... There could be an issue with a firewall or any number of things.

marconiho commented 9 months ago

async with session.get(url, params=params) as res There is no response when the program runs to this line of code.But in Windows, I can receive the expected results normally.Or maybe I can receive the results in a different way even under macOS,Like this: ` def __event_stream(url, params):

response = requests.get(url, params=params, stream=True)
event_data = ""
for line in response.iter_lines():
    if line:
        event_data += line.decode() + "\n"
    elif event_data:
        yield event_data
        event_data = ""

code_id_map_em = code_to_em_symbol('300498') url = "https://70.push2.eastmoney.com/api/qt/stock/details/sse" params = { "fields1": "f1,f2,f3,f4", "fields2": "f51,f52,f53,f54,f55", "mpi": "2000", "ut": "bd1d9ddb04089700cf9c27f6f7426281", "fltt": "2", "pos": "-0", "secid": f"{code_id_map_em}", "wbp2u": "|0|0|0|web", } big_df = pd.DataFrame() for event in __event_stream(url, params): event_json = json.loads(event.replace("data: ", "")) temp_df = pd.DataFrame( [item.split(",") for item in event_json["data"]["details"]] ) big_df = pd.concat([big_df, temp_df], ignore_index=True) break print(big_df) `

Dreamsorcerer commented 9 months ago

But, what does "no response" mean? What actually happens? Either res has been set to something, or you would get an exception (maybe a timeout after waiting a bit).

marconiho commented 9 months ago

I found that there was an issue with adding an HTTPS request, and when I switched to an HTTP request, it worked fine