-
The issue is specific to GPUs with 24GB of memory. I have to set batch_size to 1 during distributed training. But in the function `create_data_loaders`, the validation batch size is `batch_size=batch_…
-
### Motivation
CUDA_VISIBLE_DEVICES=3,4 lmdeploy serve api_server /home/ma/work/modelscope/glm-4-9b-chat-GPTQ-Int4 --backend turbomind --model-format gptq --server-port 11231 --tp 2 --session-len 165…
-
I am trying to save all coherence and phase data into a GeoTIFF using sbas.export_geotiff.
decimator = sbas.decimator(resolution=15)
corr_sbas = decimator(ds_sbas.correlation)
sbas.export_geotiff…
-
We will need to write some cards for the "Distributed Memory Systems" slides. Anyone up for the task can assign this issue to him/herself.
-
Normally disk I/O is the most time consuming part in distributed system. My suggestion is try our best to avoid touching disk on the critical path of data processing. To achieve that, we can use some…
-
Thanks for this work.
I was trying to train the model using the conda environment:
```
pytorch 2.1.2 py3.11_cuda11.8_cudnn8.7.0_0 pytorch
pytorch-cuda …
-
for mimir namespace the resourcequota and usage looks like this (per default)
```
kubectl get resourcequota -n mimir
NAME AGE REQUEST …
-
I've encountered a small issue while using Torchtune for distributed training across multiple GPUs. The problem occurs when resuming training from an unsharded recipe_state, resulting in extremely hig…
-
Operator (as distributed via OLM) has default memory limit set to 550MiB and memory request to 20MiB.
Our monitoring shows that the memory usage spikes a bit higher, on our relatively small cluster…
-
**Describe the bug**
The log settings defined by `logging_on()`, and therefore by any trollflow2 process, are not inherited by tasks scheduled using dask.distributed when called inside an `if __nam…