search

fhs / NPZ.jl

A Julia package that provides support for reading and writing Numpy .npy and .npz files
Other
117 stars 16 forks source link

Via Stackoverflow: npzread(filename.npz) very slow #19

Closed mschauer closed 5 years ago

mschauer commented 6 years ago

Timings degrade critically when .npz contains several variables, see https://stackoverflow.com/questions/46116346/julia-npzreadfilename-npz-very-slow . In

Possibly because of https://github.com/fhs/NPZ.jl/blob/master/src/NPZ.jl#L197

standarddeviant commented 5 years ago

I can confirm that this issue is still relevant

julia> @time load_xy();
122.734489 seconds (26.65 M allocations: 1006.223 MiB, 0.11% gc time)

julia> @time load_x_and_y();
  0.015817 seconds (5.03 k allocations: 9.844 MiB)

I've been using MAT.jl for a project and want to switch over to using NPZ.jl for a few reasons, but this is a show-stopper for me. I'll see if I can deduce what's taking the most time. It could be an issue with ZipFile.Reader.

bhalonen commented 5 years ago

@standarddeviant and @mschauer My pull request should minimize the issue.

bhalonen commented 5 years ago

We should be able to call this resolved @standarddeviant and @mschauer.

https://github.com/fhs/ZipFile.jl/pull/49

fhs commented 5 years ago

Closing due to https://github.com/fhs/ZipFile.jl/pull/49. Reopen if it's still an issue.