A portable Pythonic Data Catalog API powered by Ray that brings exabyte-level scalability and fast, ACID-compliant, change-data-capture to your big data workloads.
Apache License 2.0
166
stars
23
forks
source link
Revert "Clear object store after every round" #359
This PR reverts the code change (https://github.com/ray-project/deltacat/pull/357/files) performed to clear object store after every round. Clearing entire object store will also prevent any partitions that are running in parallel to complete. Hence, a dedicated task will be taken up to address this use-case.
This PR reverts the code change (https://github.com/ray-project/deltacat/pull/357/files) performed to clear object store after every round. Clearing entire object store will also prevent any partitions that are running in parallel to complete. Hence, a dedicated task will be taken up to address this use-case.