search

earthobservations / wetterdienst

Open weather data for humans.
https://wetterdienst.readthedocs.io/
MIT License
358 stars 55 forks source link

Improve runtime/bootstrap efficiency on `import wetterdienst` #791

Closed amotl closed 1 year ago

amotl commented 1 year ago

Hi there,

I discovered this problem a time ago already, but finally wanted to take the chance to report about it.

With kind regards, Andreas.

Report

Currently, it takes 4 seconds to run import wetterdienst. I think this detail should be improved earlier than later, because it will get worse with a growing number of adapters and submodules.

time python -c 'import wetterdienst'

real    0m3.111s
user    0m3.962s
sys 0m0.773s

Analysis

The reason is, due to the current module / API structure, we are loading the whole module and all submodules into memory - at import-time - right?

https://github.com/earthobservations/wetterdienst/blob/ce9eadaf8d8c649875709143c08bd798ebdd503f/wetterdienst/api.py#L20-L39

Proposal

Without knowing too many details yet, my proposal would be to make the ApiEndpoints module registry to evaluate its members at runtime instead. Implementing it is not too difficult, however we may use a minimal kind of plugin system to decouple those modules just a little bit more.

Thoughts

Would it break any sort of editor/autocompletion support, when bringing in such a change? What do you think about it?

amotl commented 1 year ago

Any upvotes or thoughts on this topic? ;]

gutzbenj commented 1 year ago

Dear @amotl ,

I agree this is something that should be improved. Regarding the ApiEndpoints we are just using it as sort of a mapping for cli/restapi/explorer where we want to have access to all implementations. But when running it as a standalone library this is not required as I believe everybody is using the sole implementations.

amotl commented 1 year ago

When running it as a standalone library this is not required as I believe everybody is using the sole implementations.

This is valuable information, because then we will not have to put any energy into keeping backwards compatibility in this regard. Thank you.

Regarding the ApiEndpoints we are just using it as sort of a mapping for cli/restapi/explorer where we want to have access to all implementations.

Excellent. So it will be no problem to change that mapping/registry to a component which resolves the corresponding implementation at runtime instead. I will try to submit a corresponding proposal.

amotl commented 1 year ago

My proposal GH-849 shaves quite a bunch of runtime.

time python -c 'import wetterdienst'

real    0m0.633s
user    0m1.200s
sys 0m0.253s