-
Will the pretraining datasets and corresponding code be open-sourced?
Thanks!
-
While trying out INT8 mixed precision pretraining (#748) with torchtitan, I came across an issue that if the model is FSDP-sharded, `quantize_()` won't work. The fix would be adding an extra logic to …
-
Hi! I'm trying to pretrain VindLU using 5M data, can you provide the pretraining logs for reference? Thanks!
-
im getting this error for my training script,
` File "/usr/local/lib/python3.10/dist-packages/peft/tuners/lora.py", line 356, in forward
result = F.linear(x, transpose(self.weight, self.fan_in_…
-
In Section 4.1, it seems that only IN1K is used for pretraining. But in Table 1, both SA-1b and IN1K are used in pretraining. Which is the correct one?
Cbtor updated
9 months ago
-
Hi,
I have about 500 labelled images and 1500 unlabelled images.
I wonder if it is possible to pretrain a model using those unlabelled images and then use the pretrained weights to initialize the …
-
Is there a way to run the pretraining of the model from a checkpoint? I'm unable to run it all at once and I don't see a way to do this.
-
Thanks for the great paper and code! I have a query about the moco v3 encoder- in the paper it mentions the latent representations are regularized on a hyper-sphere. I am fairly new to moco v3, can yo…
-
Hi, it seems this work, the pretraining scheme, is very important.
How can I do the pre-training?
-
After pretraining the model on KITTI and plotting the loss, I noticed lots of fluctuations between ~min of 0.1 to ~max of 0.8, it appears that the loss does not stabilize nor saturate. Is this a usual…