-
I think BatchedDataLoader is dealing with the case files are larger than memory, so it streams rows from disk into memory, and shuffles data in the meanwhile.
However, if in-memory cache option is …
-
I've been looking for ways to upload data other than from disk:
- **Data
-
### **Advantages of JSON-Based Tenant Loading**
1. **Simplicity**:
- Easier to implement for small-scale applications or during the development phase.
- No need to set up and manage a databas…
-
I loaded an 18 gig CSV file and the app used... 18 gig of RAM :)
There does seem to be a reference to 'out of core processing' here for how you might be able to address this with DuckDB:
https:/…
-
In one of our production clusters, the real memory usage of LRUQueryCache could be 10GB, almost 40 times larger than the estimated value (247MB) in `_nodes/stats` .
I have met this problem a few tim…
-
We're attempting to use this projects as a replacement for our homegrown memory/disk cache built around Moka and Cacache. We're seeing an issue with memory growing unbound over time, eventually leadin…
-
### Describe the bug
Reading a 2GB parquet file with a single row group:
![image](https://github.com/user-attachments/assets/dfa36531-9638-405d-82d0-77a52ba8a4a9)
Reading a 1-row parquet file:
…
-
We have several nodes running on Linux machines with the version https://github.com/ton-blockchain/ton/releases/tag/v2024.09
After start they getting in sync and consuming memory up to 100G and mor…
-
Regarding to https://github.com/redis/redis/issues/11915
Shall we add this feature to valkey?
In https://github.com/redis/redis/pull/11987, @hwware add three options to randomkeycommand, t…
-
Excellent work! I just wonder if there is any way to load the model in more than one GPU because even the 7B model consumes more than 20G memory, which is larger than memory of one GPU.