datopian / metastore-lib

🗄️ Library for storing dataset metadata, with versioning support and pluggable backends including GitHub.
https://tech.datopian.com/versioning/
MIT License
10 stars 1 forks source link

[performance] Investigate generating Organization and Repository objects vs. fetching from GitHub #24

Open shevron opened 4 years ago

shevron commented 4 years ago

See if we can instantiate these objects based on data we already have (the org name and package ID) instead of fetching them from GitHub via API. This can reduce a lot of repeating calls in a bunch of different methods.

Both objects have the ability to "lazy-fetch themselves" ("complete") from GitHub if a property of them that was not pre-populated is needed, so this technique can probably work and be harmless, but we should still see if it can be done and if it can actually improve performance, or are we going to call __complete in any case regardless, and then the added complexity is not worth it.