-
Hello,
Thanks a lot for you great work ! We were really impressed !
We wanted to know where we could download Naccos ? It seems that the link is broken: https://github.com/qanastek/DrBERT/blob/main
…
-
do you by any chance still have the dataset split (train/val/test set) that was used to pretrain ProtT5 UniRef50? I am trying to investigate data leakage for down stream tasks.
-
Hello, thanks for the awesome work you did! Could you please clarify whether ICAE uses the **entire Pile dataset** during its pretraining phase, or if it only utilizes a **subset** of it?
-
I am trying to use the framework to continue pretraining llama3-8B. I have converted the HF checkpoint into nanotron format and the generated tokens seem reasonable.
I use the following setting to…
-
Hey,
I was looking into the paper as I want to replicate the work. In the data preparation step it is mentioned **PMC** and **IDL** data is used for pre-training the model, are both data sources us…
-
Hi @jhclark-google and @dhgarrette,
I would like to know if there's any chance to get the pretraining code for CANINE.
It's been a long time since the readme was updated and I'm highly intereste…
-
Hello, thanks very much for the excellent work on this repo.
There are several examples showing how to create a question-response style dataset, but I can't immediately tell how to continue pretrai…
-
Follow your steps to pre-train T5, the code won't run. Is there any error in the code in the repository.
Thank you for your answer.
-
I'm not sure about the effect of EEG pretraining. I have directly train a shallow convolution network on `eeg_5_95_std.pth` as used in your paper. And the test accuracy according to max validation is …
-
follow the instructions on
https://github.com/HabanaAI/Model-References/tree/master/MLPERF3.1/Training/benchmarks
to execute comamnd:
`python3 pack_pretraining_data_pytorch.py --input_dir=$PYT…