supermanzer / NDBC

Repository for housing Python code for fetching, parsing, and loading NDBC data into a local Python object for analysis.
MIT License
13 stars 4 forks source link

Use async methods for data retrieval #4

Open supermanzer opened 5 years ago

supermanzer commented 5 years ago

Data requests that span either multiple months or multiple years will require multiple web requests. To improve performance for larger data sets chain asynchronous methods for fetching and processing individual data packets.

e.g.

async def chain_fetch_process(self):
    data = await self.fetch_data()
    await self.process_data(data) 

While performance gains will likely be negligible for just a few requests, some NDBC buoys have measurements going back multiple decades. Speeding up data retrieval in these cases may help make long term oceanographic analyses faster and easier during the exploratory phase for researchers.

Since we will have a Python datetime value as one of the fields, we can simply order our data by that field once all retrieval has completed.