-
- https://arxiv.org/abs/2006.15595
- 2021 ICLR
本研究では、言語の事前学習(例:BERT)で使用される位置符号化法を調査し、既存の定式化におけるいくつかの問題点を明らかにする。
まず、絶対位置エンコーディングにおいて、位置エンベッディングと単語エンベッディングに適用される加算演算は、2つの異種の情報資源の間に混在した相関をもたらすことを示す…
e4exp updated
3 years ago
-
Hi Deepspeed team,
I run DeepSpeedExamples/BingBertSquad on my machine with 2 GPUs. I follow the instruction [https://www.deepspeed.ai/tutorials/bert-finetuning/](https://www.deepspeed.ai/tutorials…
-
Hi,
Thank you for this amazing repository. I am trying to replicate your model by running the default command in README
```
python pretrain.py name=amp_b8192_cb_o4_final arch=crammed-bert train…
-
@mavenlin @SivilTaram @P2333 @chenxwh @Boyu-Mi i want to evaluate my pruned llama-7b model (saved by torch.save), but the following error happend:
╭─────────────────────────────── Traceback (most rec…
-
- [ ] Create philosophical shorts for why LLM may actually "understand"
- [ ] Create a weekly target
- [ ] Reflect on how I would trickle from year to daily vision
- [ ] Create gigs on fastwork
- [ ] …
-
### Describe the bug
According to the [documentation](https://huggingface.co/docs/datasets/v2.14.5/loading#slice-splits) is should be possible to run the following command:
`train_test_ds = data…
-
As @severo reported in an internal discussion (https://github.com/huggingface/moon-landing/issues/5929):
Now we show the dataset size:
- from the dataset card (in the side column)
- from the data…
-
We get:
```
Note: This error originates from the build backend, and is likely not a problem with poetry but with kenlm (0.2.0 https://github.com/kpu/kenlm/archive/master.zip) not supporting PEP 51…
-
Hi, Deepspeed team! I am trying to run Bert-Pretraining with deepspeed. After preprocessing the wikipedia_en dataset and bookscorpus dataset, I specified the path in bert_large_lamb_nvidia_data.json, …
-
Running this on my Mac with CPU gives:
```
Dataset bookcorpus downloaded and prepared to /Users/arhamkhan/.cache/huggingface/datasets/bookcorpus/plain_text/1.0.0/eddee3cae1cc263a431aa98207d4d27fd8…