openworm / owmeta-core

Core library for owmeta
MIT License
3 stars 2 forks source link

BitTorrent directory loader download caching #4

Open mwatts15 opened 5 years ago

mwatts15 commented 5 years ago

Directory loaders are given a directory where their downloaded files should be stored and this directory is returned to the loader each time we attempt a directory load. Although we have logic to check whether a given data source's directory has already been loaded irrespective of the loader that loaded it, it is possible for multiple data sources to share the same directory and the BitTorrent loader should find these cases naturally since it uses content-based identifiers for torrents. When two data sources with the same files associated are downloaded in succession, the loader should recognize this and not attempt to re-download something it already has.