-
-
Dear authors,
Thank you for the great work.
I can‘t fully download the dataset. Can you give me some advice on how to create my own dataset。what data structure should I process?
Thanks
-
I was wondering if there are sample packing approaches defined somewhere for preprocessing and tokenization of datasets? I looked through different prepare_*.py, but couldn't find anything related to …
-
1. I tried vanilla pytorch training loop using bfloat16, the loss got overflow, https://github.com/mesolitica/malaya/blob/5.1/pretrained-model/mamba/causallm-130m-bf16.ipynb
2. so I tried vanilla pyt…
-
-
Hello, there was an issue during the training. Is this a data reading issue? Thanks！！
The error is as follows：
upr-base => val step: 1: 104/119; time: 0.00+0.27
upr-base => val step: 1: 105/119; ti…
-
Hi all, I'm new to xformers, I'm learning the `examples/llama_inference/generate.py` file.
I traced it here:
```python
def _memory_efficient_attention_forward(
inp: Inputs, op: Optional[Type…
-
I try to train data2vec on music data (the FMA dataset). I've made some modifications to the feature extractor ConvNet (I've made it a small ResNet essentially), and reduced the size of the transforme…
-
In https://github.com/mlfoundations/open_lm/pull/125, we had to switch our gradient accumulation tests from SGD to AdamW to make gradient accumulation tests pass. It's unclear why this is the case; an…
-
Hi.
I have several questions about saving models to a local folder.
1, When I run the self-calibration code, in which folders are trained models saved to?
2, Which part of config files should …