search

microsoft / gr-azure

GNU General Public License v3.0
24 stars 5 forks source link

File Caching with Blobs #11

Open pomeroy3 opened 2 years ago

pomeroy3 commented 2 years ago

Some users may want to cache blob file contents if they intend to re-run the same processing multiple times without re-downloading the same file over and over.

It may be useful to include some pointers in the documentation on how to handle these use cases, possibly by using something like a Lustre file system and file source blocks if users want to flexibly work with a large number of blobs, or using AzCopy to manually copy smaller numbers of blobs and again using file source blocks.

https://azure.microsoft.com/en-us/updates/lustre-hsm-tools-now-available-to-import-from-or-export-to-azure-storage/

https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10

AB#10542

MarcusSorealheis commented 1 year ago

Could the best solution here be an open source database targeting embedded, like Realm?