lexiforest / curl_cffi

Python binding for curl-impersonate via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
https://curl-cffi.readthedocs.io/
MIT License
1.9k stars 235 forks source link

[Feature] Simulate aiohttp interface for async operations #202

Open Snusmumr1000 opened 8 months ago

Snusmumr1000 commented 8 months ago

Is your feature request related to a problem? Please describe. aiohttp is a widely used library, which includes connection via HTTP and WebSockets. It would be extremely helpful to have a aiohttp-like interface for this library as it will:

  1. Flatten the learning curve.
  2. Make the integration in existing projects much easier.
  3. Probably will attract more people to use the library.

Describe the solution you'd like Adapter-layer would be enough that would simulate the aiohttp behaviour.

Describe alternatives you've considered Write an own one is a possible solution.

However, if we have a requests library interface simulation, why not implement an aiohttp interface as a standard for async version?

Snusmumr1000 commented 8 months ago

I can help you implement this feature if you agree with this proposal

dolfies commented 8 months ago

In terms of requests, the AsyncSession interface is already similar enough to aiohttp that I don't think it matters

perklet commented 8 months ago

For the asyncio interface, I trid to mimic what httpx does if possible. As @dolfies said, this is similar to what aiohttp does. Could you please describe in details about what is missing?

Snusmumr1000 commented 8 months ago

Sure, here are the features I meant:

WebSocket

Using async / sync generator for receiving messages.

IMO, aiohttp has quite a convenient interface for receiving messages. Example:

async with session.ws_connect(ws_url) as ws:  # not necessary
    async for msg in ws:  # exactly what I mean
        # do something with msg

Unfortunately, aiohttp does not provide a way of choosing the "generated" messages type, so maybe it would also be nice to add a type of received messages (bytes / str / json).

Specific datatypes wrapper methods:

aiohttp has wrapper methods for sending and receiving specific datatypes, they encapsulate some boilerplate conversion code and make library UX more user-friendly.

async def send_str(self, data: str, compress: Optional[int] = None) -> None:
    ...

async def send_bytes(self, data: bytes, compress: Optional[int] = None) -> None:
    ...

async def send_json(
    self,
    data: Any,
    compress: Optional[int] = None,
    *,
    dumps: JSONEncoder = DEFAULT_JSON_ENCODER,
) -> None:
    ...

async def receive_str(self, *, timeout: Optional[float] = None) -> str:
    ...

async def receive_bytes(self, *, timeout: Optional[float] = None) -> bytes:
    ...

async def receive_json(
    self,
    *,
    loads: JSONDecoder = DEFAULT_JSON_DECODER,
    timeout: Optional[float] = None,
) -> Any:
    ...

HTTP Requests

Single request proxy passing

For me, it seems quite convenient (at least in terms of some quick prototyping or testing) to pass a proxy to a single request as a uri:

session.get("http://python.org",
            proxy="http://user:pass@some.proxy.com")

However, I understand, that it might not correlate well with already chosen way of handling proxies.

perklet commented 8 months ago

All of these are valid requests, I guess. I'm happy to merge PRs if someone would be kind enough to implement them. But for the asyncio websockets, be aware that the underlying mechanism is sync-based.

dolfies commented 8 months ago

I can definitely look into implementing some of these

Snusmumr1000 commented 8 months ago

Perfect, I'm glad to hear that the proposal seemed somehow useful to you