irmen / pickle

Java and .NET implementation of Python's pickle serialization protocol
MIT License
78 stars 5 forks source link

Load in JSON #14

Open PowerICT opened 11 months ago

PowerICT commented 11 months ago

I'm trying to load a wide serie of .sav files using your Nuget, but my attempts ended in this error.

expected zero arguments for construction of ClassDict (for numpy.dtype). This happens when an unsupported/unregistered class is being unpickled that requires construction arguments. Fix it by registering a custom IObjectConstructor for this class

Is it possible to set a default translation in a KeyValuePair<string, string> for the unregistered/unsupported classes, or add a specific constructor for numpy.dtype class?

irmen commented 11 months ago

Yes. try Unpickler.registerConstructor(...)

However you may find that you need to add a constructor for all numpy types that might be pickled into your ".sav" file. It could be easier to not store the numpy data in your pickles, but convert it to normal python datatypes before writing the pickle file.

PowerICT commented 11 months ago

However you may find that you need to add a constructor for all numpy types that might be pickled into your ".sav" file. It could be easier to not store the numpy data in your pickles, but convert it to normal python datatypes before writing the pickle file.

It might be difficult to edit all our .sav files (not all of them are created by us).

We've defined a JSONConstructor():

public class JSONConstructor : IObjectConstructor
{
    public object construct(object[] args)
    {
        try
        {
            string json = JsonConvert.SerializeObject(args);

            return json;
        }
        catch (Exception x)
        {
            throw new PickleException("problem constructing object", x);
        }
    }
}

But we encounter the following exception:

  | Nome | Valore | Tipo -- | -- | -- | -- ▶ | e | {"failed to __setstate__()"} | System.Exception {Razorvine.Pickle.PickleException}

We think the Exception is caused by the missing setstate() method for a primitive object (string).

We even tried to use the Unpickler.registerConstructor with the ClassDictConstructor() method but we encounter the following exception:

  | Nome | Valore | Tipo -- | -- | -- | -- ▶ | e | {"expected zero arguments for construction of ClassDict (for numpy.dtype). This happens when an unsupported/unregistered class is being unpickled that requires construction arguments. Fix it by registering a custom IObjectConstructor for this class."} | System.Exception {Razorvine.Pickle.PickleException} ▶ | e | {"expected zero arguments for construction of ClassDict (for numpy.core.multiarray._reconstruct). This happens when an unsupported/unregistered class is being unpickled that requires construction arguments. Fix it by registering a custom IObjectConstructor for this class."} | System.Exception {Razorvine.Pickle.PickleException}

Can you please give us some instructions to resolve the Exceptions?

Thank you!

irmen commented 11 months ago

Try to use the source code to figure out why you still get these errors. For example, here are a couple of unit tests that utilize the custom class constructor registry https://github.com/irmen/pickle/blob/master/java/src/test/java/net/razorvine/pickle/test/UnpicklerComplexTest.java

also you already see for example that you encounter other numpy types such as numpy.core.multiarray that you'll have to deal with.