Handling None -> Missing conversions in matrices

[fipelle]

Hi,

I have noticed that trying to convert a numpy array containing None to a Matrix{Union{Missing, Float64}} does not work smoothly. Indeed, it gives the following error: TypeError('must be real number, not NoneType').

Have you ever encountered this issue before? Are there any working solutions?

[fipelle]

I have written down the following workaround:

"""
    ndarray_to_matrix(X::Matrix{PyCall.PyObject})

Converts `X` into a Matrix{Union{Missing, Float64}} handling `None` appropriately.

    ndarray_to_matrix(X::Matrix{Float64})

Converts `X` into a Matrix{Union{Missing, Float64}} for internal consistency.
"""
function ndarray_to_matrix(X::Matrix{PyCall.PyObject})
    out = zeros(size(X)) |> Matrix{Union{Missing, Float64}};
    for j in axes(out, 2), i in axes(out, 1)
        try
            out[i,j] = convert(Float64, X[i,j]);
        catch e
            if "$(e.val)" == "PyObject TypeError('must be real number, not NoneType')"
                out[i,j] = missing;
            else
                throw(e);
            end
        end
    end

    return out;
end

ndarray_to_matrix(X::Matrix{Float64}) = convert(Matrix{Union{Missing, Float64}}, X);

Not sure if there are better ways to handle this issue.

[stevengj]

I thought numpy arrays don't have a missing value? Can you give an example of what you have on the Python side?

[fipelle]

Yes, of course. I am writing a wrapper for a public package (happy to pass along the references if it may help). I have attached a short screenshot to highlight what's happening on the Python side.