Closed felix-reichel closed 1 week ago
New:
Largest week offers data, that is ~ 760mb
(52+26)*760 ~ 60gb <= 128gb (S)
L (512gb),XL (1tb) , 2XL (2tb)
Goal: Batch process 5yrs needs 1yr (52 pre-weeks) overlap then, for 4 iterations covering 20-4 years then. (2023-2007)=16. 4 batches. XL should be feasible for offers and clicks only.
Introduced a tables config and basic table caching using duckdb's PRAGMA in faf9fe46ecfbfbfab5b49579cb8cd44c2129a6b9
https://github.com/felix-reichel/price-search-engine-seals-analysis/issues/27
can be properly covered in https://github.com/felix-reichel/price-search-engine-seals-analysis/issues/28 (Generalized dynamic db loaders 2.0)
New:
Largest week offers data, that is ~ 760mb
(52+26)*760 ~ 60gb <= 128gb (S)
L (512gb),XL (1tb) , 2XL (2tb)
Goal: Batch process 5yrs needs 1yr (52 pre-weeks) overlap then, for 4 iterations covering 20-4 years then. (2023-2007)=16. 4 batches. XL should be feasible for offers and clicks only.