search

chengchingwen / Pickle.jl

An experimental package for loading and saving object in Python Pickle format.
MIT License
50 stars 9 forks source link

`keras.src.saving.pickle_utils.deserialize_model_from_bytecode` not found, results in Defer #38

Open IanButterworth opened 6 months ago

IanButterworth commented 6 months ago

I'm guessing this means some keras support needs to be added.

julia> params = Pickle.Torch.THload("foo.pth")
Defer(:reduce, Defer(:keras.src.saving.pickle_utils.deserialize_model_from_bytecode), UInt8[0x50, 0x4b, 0x03, 0x04, 0x14, 0x00, 0x00, 0x00, 0x00, 0x00  …  0xb2, 0x00, 0x00, 0x00, 0xc6, 0xa5, 0xee, 0x02, 0x00, 0x00])

If so, can you provide guidance. I'm happy to PR

chengchingwen commented 6 months ago

You would need to implement a julia function that perform keras.src.saving.pickle_utils.deserialize_model_from_bytecode on UInt8[0x50, 0x4b, 0x03, 0x04, 0x14, 0x00, 0x00, 0x00, 0x00, 0x00 … 0xb2, 0x00, 0x00, 0x00, 0xc6, 0xa5, 0xee, 0x02, 0x00, 0x00] and register that to the unpickler (TorchPickler().mt["keras.src.saving.pickle_utils.deserialize_model_from_bytecode"] = julia_impl_deserialize_model_from_bytecode). Though I'm not sure what exactly keras.src.saving.pickle_utils.deserialize_model_from_bytecodedo.

chengchingwen commented 6 months ago

https://github.com/keras-team/keras/blob/601488fd4c1468ae7872e132e0f1c9843df54182/keras/saving/pickle_utils.py#L46

https://github.com/keras-team/keras/blob/601488fd4c1468ae7872e132e0f1c9843df54182/keras/saving/saving_lib.py#L233

The bytes seem to be raw bytes of a zipfile, which contains the configurations and weights stored in either H5 or Npz. The load_model function would directly construct the keras model from the configurations and weights.