-
### Description
Make `ray.data.from_arrow` efficiently support iterators of Arrow tables. Currently, `from_arrow` loads all of the data in memory.
### Use case
If your data can't fit in memor…
-
### System Info
- `transformers` version: 4.41.2
- Platform: Linux-5.10.0-1.0.0.28-x86_64-with-glibc2.31
- Python version: 3.10.14
- Huggingface_hub version: 0.24.6
- Safetensors version: 0.4.4…
-
I have tried using both dense_hash_map and sparse_hash_map for inserting millions of entries into the hash table in a distributed environment, where each process holds its own private hash table. Unfo…
-
**Describe the bug**
With [PR](https://github.com/rapidsai/cuml/pull/4300) we enabed training single GPU cuML models using Dask DataFrames and Series but we use `compute` there which brings data t…
-
Flamingo model initialized with 23461888 trainable parameters
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/jovyan/taoheng/work/Multimodal-GPT/mmgpt/tr…
-
At present, `DiskDataset` is our workhorse class for large datasets. This class is pretty nicely optimized with a cache and everything, and I've been able to use it on 50GB datasets without too much t…
-
Currently you can link to an anchor tag _within_ a `Doc` term with the anchor tag link syntax, but we've encountered a few cases where it would be useful to link to an anchor tag within a different `D…
-
Subsequent Blockwise layers are currently fused into a single layer. This reduces the number of tasks, the overhead and is very generally a good thing to do. Currently, the fused output does not gener…
-
Thanks for you code.
However, when I run "CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --master_port 10025 --nproc_per_node=1 tools/relation_train_net.py --config-file "configs/SHA_GCL_…
-
### Missing functionality
# Support for modin framework to make the EDA on larger datasets much faster
I would like to request the addition of support for the Modin framework to our EDA tools. As …