quinnj / JSON3.jl

Other
214 stars 47 forks source link

Allow typed `JSON3.read` on `JSON3.Array` #250

Open rafaqz opened 1 year ago

rafaqz commented 1 year ago

In GeoJSON.jl we are loading JSON lazily, which is great over large files.

But the final vectors of points are loaded as arrays of length 2/3 arrays, which is very slow. We need to load these as length 2/3 Tuples instead.

Is it possible to read a JSON3.Array forcing the eltype to be Tuple{Float64,Float64} ? Just loading the whole array at that point as Vector{Tuple{Float64,Float64}}.

evetion commented 1 year ago

Specifically, it's slow to go from JSON3 Array into a nested type like Vector{Vector{Vector{Float64}}} (sometimes deeper).

Reading directly into a struct is fast, but loses the laziness. Ideally we load lazily and then convert subsets of the JSON 3 Object/Array into known types directly.

rafaqz commented 1 year ago

Can we just call read on the codepoints in the tape at the leaf?

Maybe allowing something like JSON3.read(A::JSON.Array, Vector{Tuple{Float64,Float64}}) with these leaf arrays would be possible?

evetion commented 1 year ago

From what I gathered; given an JSON3.Array A, A.tape[A.inds[begin]:A[A.inds[end]] would give you the tape for that array, but it's unclear to me how to parse the tape to find the correct indices into A.buf. It could just take a single utility function.