Closed Jackmin801 closed 1 year ago
Yes, the models use no biases in general and no element-wise affine transformations in layer norms by design. Can you check if the biases are present in the 6.7B checkpoints that we published or if the biases in your checkpoints are non-zero?
All the biases in 6.7B checkpoint are 0
I've checked the other model checkpoints. All of them have bias tensors that are all zero (except 30b which has no bias tensors).
This is the info I have about the checkpoints so far and some notable differences in the 30b checkpoint:
Size | Parameters | Disk Usage | Bytes / Parameter ratio | Sum(layer.numels) | Data type of tensors |
---|---|---|---|---|---|
mini |
125 M | 480M | 4.0265 | 163,430,400 | {torch.float32: 197} |
base |
1.3 B | 5.0G | 4.1298 | 1,417,601,024 | {torch.float32: 389} |
standard |
6.7 B | 26G | 4.1667 | 6,862,159,872 | {torch.float32: 517} |
large |
30 B | 56G | 2.0043 | 29,968,103,424 | {torch.float16: 290} |
huge |
120 B | 453G | 4.0534 | 121,853,747,200 | {torch.float32: 1541} |
The checkpoint is now fixed as part of https://huggingface.co/facebook/galactica-30b/discussions/6. All the checkpoints are now fully compatible with OPT architecture, use float16 weights, with layer norm weights set to ones and all the biases set to zero.
The 30b model pickles seem to have no biases.
This is opposed to the 6.7b model which contains a lot of biases.
I do not believe I am missing any pickles because the disk usage of the cloned repository tallies with what is displayed by the huggingface site (note that du outputs gibibytes which is likely the cause of the slight discrepancy in raw numbers).