search

crystal-data / num.cr

Scientific computing in pure Crystal
MIT License
151 stars 12 forks source link

Cannot Parse Rank-One Tensors from .npy Files #93

Closed dhwa-kyle closed 4 months ago

dhwa-kyle commented 4 months ago

Problem description

Trying to parse a rank-one tensor from a .npy file currently results in a JSON::ParseException. The exception is raised when trying to parse the JSON header, and is specifically due to a trailing comma that is left behind only in the rank-one case after the gsub calls here.

Example

require "num"

tensor = Tensor.from_array([1, 2, 3, 4])
tensor.to_npy("single_rank_tensor.npy")
Tensor(Int32, CPU(Int32)).from_npy("single_rank_tensor.npy") # Raises JSON::ParseException

Workaround

One can pad the shape with one or mores 1s before serialization and reshape back to rank-one after de-serialization.

Proposed Fix

Change the second gsub on line 51 in io/npy.cr from gsub(/,]|],/, "]") to gsub(/,],|],/, "]")

christopherzimmerman commented 4 months ago

@dhwa-kyle thanks for the issue, this was just merged into master, and should be fixed now!