search

google-research / big_vision

Official codebase used to develop Vision Transformer, SigLIP, MLP-Mixer, LiT and more.
Apache License 2.0
2.04k stars 140 forks source link

Can I convert paligemma npz model to pytorch to safetensors? #107

Closed sdip15fa closed 1 month ago

sdip15fa commented 1 month ago

Hi, I have tried to fine-tune paligemma using the notebook provided and got .npz models. Is it possible I can convert them to the format as in google/paligemma-3b-pt-224 so I can use them with transformers? Thanks.

merveenoyan commented 1 month ago

@sdip15fa hello! I think you can use this script: https://github.com/huggingface/transformers/blob/main/src/transformers/models/paligemma/convert_paligemma_weights_to_hf.py

sdip15fa commented 1 month ago

@merveenoyan Thanks a lot!

AhmedMasryKU commented 4 weeks ago

@merveenoyan I tried the script that you shared, but it's giving me this error. Do you know how to fix it?

Traceback (most recent call last): File "/content/convert_paligemma.py", line 340, in convert_paligemma_checkpoint( File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/content/convert_paligemma.py", line 255, in convert_paligemma_checkpoint state_dict_transformers = slice_state_dict(state_dict, config) File "/content/convert_paligemma.py", line 202, in slice_state_dict state_dict[key] = torch.from_numpy(value) TypeError: can't convert np.ndarray of type numpy.void. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint64, uint32, uint16, uint8, and bool.