gdcc / pyDataverse

Python module for Dataverse Software (dataverse.org).
http://pydataverse.readthedocs.io/
MIT License
63 stars 41 forks source link

Async requests #175

Closed JR-1991 closed 2 months ago

JR-1991 commented 4 months ago

Changes this PR introduces

This pull request adds async support to pyDataverse based on the previously merged switch from requests to httpx. Asynchronous requests significantly improve pyDataverse's performance by using concurrent requests. The implementation provides a seamless usage to work either sync or async by the usage of a context manager.

Small benchmark

Overview

Example

Synchronous

native_api = NativeApi("https://demo.dataverse.org")
response = native_api.get_info_version()

Asynchronous

import asyncio
from pyDataverse.api import NativeApi

async def run_async(native_api):
    async with native_api:
        tasks = [native_api.get_info_version() for _ in range(100)]
        responses = await asyncio.gather(*tasks)

if __name__ == "__main__":
    native_api = NativeApi("https://demo.dataverse.org")
    asyncio.run(run_async(native_api))
JR-1991 commented 2 months ago

@haeussma thanks for the review! I have added urllib3 to the list of dependencies.