I changed the whole pipeline to create and manage cloud objects without the necessity of keys prefixes at all.
main changes:
update the project for recent PyWren changes (for the next PyWren version 1.5.2).
add support in cloud objects for the whole pipeline.
add cache cloud objects mechanism - to make the pipeline easy to be developed and more stable. the cache mechanism stores and manages cobjects - if a single stage failed for some reason, we dont have to restart everything from the beginning, the pipeline will load the required data from cache and will continue to run from the same point automatically.
add cloud objects cleansing by pipeline.clean(). this method will also delete cached data.
changes for the next PR:
add support in cloud objects for the database builder
add support in cache from the database builder
recover read_object_with_retry() instead of storage.get_cobject()
I changed the whole pipeline to create and manage cloud objects without the necessity of keys prefixes at all.
main changes:
pipeline.clean(). this method will also delete cached data.changes for the next PR:
read_object_with_retry()instead ofstorage.get_cobject()