search

rogersce / cnpy

library to read/write .npy and .npz files in C/C++
MIT License
1.31k stars 300 forks source link

npy file size limit #98

Open erichnau opened 7 months ago

erichnau commented 7 months ago

I am sucessfully using cnpy for the conversion of a custom .fld format to .npy, however, i have lately observed issues with very large files. my npy files are written to a maximum size of 6.81GB - then writing stops but the process continues without any errors. the resulting npy is faulty because the array size specified in the header does not match the data.

Juelin-Liu commented 5 months ago

Hi, you can try to change lines 85 and 133 in the "cnpy.cpp" file to: 

shape.push_back(std::stoul(sm[0].str()));

The original implementation uses std::stoi which cannot support shape dimension > 2,147,483,647.

erichnau commented 2 months ago

Great. Thanks guys, that solved my issue!