System.InvalidCastException when .NET type is `object` and Python type is float.

[sherryyshi]

Here is a simple repro:

C# code:

namespace PandasNetTest
{
    public class Test
    {
        object _x;

        public Test(object x)
        {
            _x = x;
        }
    }
}

Python code:

# PythonNet setup code redacted

import pandasnet
import numpy as np
from PandasNetTest import Test

t = Test(np.float64(0.9))

This causes an exception to be thrown here ((Func<PyObject, T>)decoder)(pyObj); (full stack trace in Details section below).

This is because at runtime, decoder is of type (Func<PyObject, double>) and T is type object, and the cast doesn't work.

Unfortunately, when this exception is hit, the entire Python process gets killed. There is no way to catch it as a consumer of this package.

Proposed solution One solution is to cast the return type of decoder to T instead. However, this is not trivial because the type of decoder is object at compile time, and the compiler would not allow you to call it as a function without doing the type casting that is currently in place. One way to work around this would be to not use the _decoders dictionary, but to use a switch/case block instead:

        public bool TryDecode<T>(PyObject pyObj, out T value)
        {
            string type = pyObj.GetPythonType().ToString();

            if (!_decoders.TryGetValue(type, out var decoder))
            {
                value = default;
                return false;
            }

            object decoded = type switch
            {
                "<class 'numpy.ndarray'>" => NumpyCodec.Decode(pyObj),
                "<class 'numpy.bool_'>" => pyObj.As<bool>(),
                "<class 'numpy.uint8'>" => pyObj.As<byte>(),
                "<class 'numpy.uint16'>" => pyObj.As<ushort>(),
                "<class 'numpy.uint32'>" => pyObj.As<uint>(),
                "<class 'numpy.uint64'>" => pyObj.As<ulong>(),
                "<class 'numpy.int8'>" => pyObj.As<sbyte>(),
                "<class 'numpy.int16'>" => pyObj.As<short>(),
                "<class 'numpy.int32'>" => pyObj.As<int>(),
                "<class 'numpy.int64'>" => pyObj.As<long>(),
                "<class 'numpy.float32'>" => pyObj.As<float>(),
                "<class 'numpy.float64'>" => pyObj.As<double>(),
                "<class 'datetime.date'>" => DateTimeCodec.DecodeDate(pyObj),
                "<class 'datetime.datetime'>" => DateTimeCodec.DecodeDateTime(pyObj),
                "<class 'datetime.time'>" => DateTimeCodec.DecodeTime(pyObj),
                "<class 'datetime.timedelta'>" => DateTimeCodec.DecodeTimeDelta(pyObj),
                "<class 'pandas.core.frame.DataFrame'>" => PandasCodec.DecodeDataFrame(pyObj),
                "<class 'pandas.core.series.Series'>" => PandasCodec.DecodeSeries(pyObj),
                "<class 'pandas._libs.tslibs.timestamps.Timestamp'>" => PandasCodec.DecodeTimestamp(pyObj),
                "<class 'pandas._libs.tslibs.timedeltas.TimeDelta'>" => DateTimeCodec.DecodeTimeDelta(pyObj),
                _ => pyObj
            };

            if (decoded == pyObj)
            {
                value = default;
                return false;
            }

            try
            {
                value = (T)decoded;
                return true;
            }
            catch (InvalidCastException)
            {
                value = default;
                return false;
            }
        }

Details

Full stack trace: ``` Unhandled exception. System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidCastException: Unable to cast object of type 'System.Func`2[Python.Runtime.PyObject,System.Double]' to type 'System.Func`2[Python.Runtime.PyObject,System.Object]'. at PandasNet.Codecs.TryDecode[T](PyObject pyObj, T& value) in /home/runner/work/pandasnet/pandasnet/dotnet/PandasNet/Codecs.cs:line 54 at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor) at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr) --- End of inner exception stack trace --- at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr) at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters) at Python.Runtime.PyObjectConversions.<>c__DisplayClass9_0.g__TryDecode|0(BorrowedReference pyHandle, Object& result) in /tmp/build-via-sdist-h3j67_wc/pythonnet-3.0.3/src/runtime/Codecs/PyObjectConversions.cs:line 109 at Python.Runtime.PyObjectConversions.TryDecode(BorrowedReference pyHandle, BorrowedReference pyType, Type targetType, Object& result) in /tmp/build-via-sdist-h3j67_wc/pythonnet-3.0.3/src/runtime/Codecs/PyObjectConversions.cs:line 87 at Python.Runtime.Converter.ToManagedValue(BorrowedReference value, Type obType, Object& result, Boolean setError) in /tmp/build-via-sdist-h3j67_wc/pythonnet-3.0.3/src/runtime/Converter.cs:line 382 at Python.Runtime.Converter.ToManaged(BorrowedReference value, Type type, Object& result, Boolean setError) in /tmp/build-via-sdist-h3j67_wc/pythonnet-3.0.3/src/runtime/Converter.cs:line 273 at Python.Runtime.MethodBinder.TryConvertArgument(BorrowedReference op, Type parameterType, Object& arg, Boolean& isOut) in /tmp/build-via-sdist-h3j67_wc/pythonnet-3.0.3/src/runtime/MethodBinder.cs:line 666 at Python.Runtime.MethodBinder.TryConvertArguments(ParameterInfo[] pi, Boolean paramsArray, BorrowedReference args, Int32 pyArgCount, Dictionary`2 kwargDict, ArrayList defaultArgList, Int32& outs) in /tmp/build-via-sdist-h3j67_wc/pythonnet-3.0.3/src/runtime/MethodBinder.cs:line 629 at Python.Runtime.MethodBinder.Bind(BorrowedReference inst, BorrowedReference args, Dictionary`2 kwargDict, MethodBase[] methods, Boolean matchGenerics) in /tmp/build-via-sdist-h3j67_wc/pythonnet-3.0.3/src/runtime/MethodBinder.cs:line 407 at Python.Runtime.MethodBinder.Bind(BorrowedReference inst, BorrowedReference args, BorrowedReference kw, MethodBase info, MethodBase[] methodinfo) in /tmp/build-via-sdist-h3j67_wc/pythonnet-3.0.3/src/runtime/MethodBinder.cs:line 366 at Python.Runtime.MethodBinder.Invoke(BorrowedReference inst, BorrowedReference args, BorrowedReference kw, MethodBase info, MethodBase[] methodinfo) in /tmp/build-via-sdist-h3j67_wc/pythonnet-3.0.3/src/runtime/MethodBinder.cs:line 868 at Python.Runtime.MethodObject.Invoke(BorrowedReference target, BorrowedReference args, BorrowedReference kw, MethodBase info) in /tmp/build-via-sdist-h3j67_wc/pythonnet-3.0.3/src/runtime/Types/MethodObject.cs:line 77 at Python.Runtime.MethodBinding.tp_call(BorrowedReference ob, BorrowedReference args, BorrowedReference kw) in /tmp/build-via-sdist-h3j67_wc/pythonnet-3.0.3/src/runtime/Types/MethodBinding.cs:line 166 at Python.Runtime.Runtime.PyObject_Call(BorrowedReference pointer, BorrowedReference args, BorrowedReference kw) in /tmp/build-via-sdist-h3j67_wc/pythonnet-3.0.3/src/runtime/Runtime.cs:line 919 at Python.Runtime.ManagedType.Init(BorrowedReference obj, BorrowedReference args, BorrowedReference kw) in /tmp/build-via-sdist-h3j67_wc/pythonnet-3.0.3/src/runtime/Types/ManagedType.cs:line 136 at Python.Runtime.ClassBase.Init(BorrowedReference obj, BorrowedReference args, BorrowedReference kw) in /tmp/build-via-sdist-h3j67_wc/pythonnet-3.0.3/src/runtime/Types/ClassBase.cs:line 634 at Python.Runtime.MetaType.tp_call(BorrowedReference tp, BorrowedReference args, BorrowedReference kw) in /tmp/build-via-sdist-h3j67_wc/pythonnet-3.0.3/src/runtime/Types/MetaType.cs:line 255 ```

[fdieulle]

Hi

Thanks for your question.

Why your function Test in C# doesn't define a double as argument ? By defining an object, the decoder cannot infer the recipient type. In addition it generates boxing allocations which can be harmful in an heavy usage. In your example you trigger a boxing (aka: native value wrapped in an object) You could reproduce this boxing manually by defining your own types. For example:

interface IBoxing { object Value {get; } } class DoubleBox : IBoxing { double Value { get; set; } object IBoxing.Value => Value; } class Int32Box : IBoxing { int Value { get; set; } object IBoxing.Value => Value; }

or more flexible: class ObjectBox : IBoxing { object Value { get; set; } object IBoxing.Value => Value; }

void Test(IBoxing boxing) { _x = boxing?.Value; } ...

Let me know if it helped.

Regards