search

jpjones76 / SeisIO.jl

Julia language support for geophysical time series data
http://seisio.readthedocs.org
Other
47 stars 21 forks source link

Error in opening SeisData object after recent update of SeisIO #21

Closed xtyangpsp closed 5 years ago

xtyangpsp commented 5 years ago

After updating SeisIO to the latest version (to use the functionality of removing responses), I can't open my old SeisData object stored in jld2 file. It seems there is some compatibility issue after the new update. Here are the errors:

`┌ Warning: saved type PZResp is missing field a0 in workspace type; reconstructing └ @ JLD2 ~/.julia/packages/JLD2/KjBIK/src/data.jl:402

ERROR: MethodError: Cannot convert an object of type getfield(JLD2.ReconstructedTypes, Symbol("##PZResp#363")) to an object of type InstrumentResponse Closest candidates are: convert(::Type{T}, ::T) where T at essentials.jl:154 Stacktrace: [1] jlconvert at /Users/xiaotaoyang/.julia/packages/JLD2/KjBIK/src/data.jl:675 [inlined] [2] macro expansion at /Users/xiaotaoyang/.julia/packages/JLD2/KjBIK/src/dataio.jl:70 [inlined] [3] macro expansion at ./simdloop.jl:73 [inlined] [4] read_array!(::Array{InstrumentResponse,1}, ::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadRepresentation{InstrumentResponse,JLD2.RelOffset}) at /Users/xiaotaoyang/.julia/packages/JLD2/KjBIK/src/dataio.jl:68 [5] read_array(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::JLD2.ReadRepresentation{InstrumentResponse,JLD2.RelOffset}, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1}) at /Users/xiaotaoyang/.julia/packages/JLD2/KjBIK/src/datasets.jl:323 [6] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadRepresentation{Any,JLD2.RelOffset}, ::Array{JLD2.ReadAttribute,1}) at /Users/xiaotaoyang/.julia/packages/JLD2/KjBIK/src/datasets.jl:198 [7] macro expansion at /Users/xiaotaoyang/.julia/packages/JLD2/KjBIK/src/datasets.jl:156 [inlined] [8] macro expansion at /Users/xiaotaoyang/.julia/packages/JLD2/KjBIK/src/datatypes.jl:76 [inlined] [9] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::UInt8, ::Int64, ::Int64, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1}) at /Users/xiaotaoyang/.julia/packages/JLD2/KjBIK/src/datasets.jl:152 [10] load_dataset(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.RelOffset) at /Users/xiaotaoyang/.julia/packages/JLD2/KjBIK/src/datasets.jl:92 [11] jlconvert at /Users/xiaotaoyang/.julia/packages/JLD2/KjBIK/src/data.jl:674 [inlined] [12] macro expansion at /Users/xiaotaoyang/.julia/packages/JLD2/KjBIK/src/data.jl:1285 [inlined] [13] jlconvert(::JLD2.ReadRepresentation{SeisData,JLD2.OnDiskRepresentation{(0, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104),Tuple{Int64,Array{String,1},Array{String,1},Array{InstrumentPosition,1},Array{Float64,1},Array{Float64,1},Array{InstrumentResponse,1},Array{String,1},Array{String,1},Array{Dict{String,Any},1},Array{Array{String,1},1},Array{Array{Int64,2},1},Array{Union{Array{Float32,1}, Array{Float64,1}},1},Array{Sockets.TCPSocket,1}},Tuple{Int64,JLD2.RelOffset,JLD2.RelOffset,JLD2.RelOffset,JLD2.RelOffset,JLD2.RelOffset,JLD2.RelOffset,JLD2.RelOffset,JLD2.RelOffset,JLD2.RelOffset,JLD2.RelOffset,JLD2.RelOffset,JLD2.RelOffset,JLD2.RelOffset}}()}, ::JLD2.JLDFile{JLD2.MmapIO}, ::Ptr{Nothing}, ::JLD2.RelOffset) at /Users/xiaotaoyang/.julia/packages/JLD2/KjBIK/src/data.jl:1232 [14] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadRepresentation{SeisData,JLD2.OnDiskRepresentation{(0, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104),Tuple{Int64,Array{String,1},Array{String,1},Array{InstrumentPosition,1},Array{Float64,1},Array{Float64,1},Array{InstrumentResponse,1},Array{String,1},Array{String,1},Array{Dict{String,Any},1},Array{Array{String,1},1},Array{Array{Int64,2},1},Array{Union{Array{Float32,1}, Array{Float64,1}},1},Array{Sockets.TCPSocket,1}},Tuple{Int64,JLD2.RelOffset,JLD2.RelOffset,JLD2.RelOffset,JLD2.RelOffset,JLD2.RelOffset,JLD2.RelOffset,JLD2.RelOffset,JLD2.RelOffset,JLD2.RelOffset,JLD2.RelOffset,JLD2.RelOffset,JLD2.RelOffset,JLD2.RelOffset}}()}, ::Array{JLD2.ReadAttribute,1}) at /Users/xiaotaoyang/.julia/packages/JLD2/KjBIK/src/dataio.jl:37 [15] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::UInt8, ::Int64, ::Int64, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1}) at /Users/xiaotaoyang/.julia/packages/JLD2/KjBIK/src/datasets.jl:149 [16] load_dataset(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.RelOffset) at /Users/xiaotaoyang/.julia/packages/JLD2/KjBIK/src/datasets.jl:92 [17] getindex(::JLD2.Group{JLD2.JLDFile{JLD2.MmapIO}}, ::String) at /Users/xiaotaoyang/.julia/packages/JLD2/KjBIK/src/groups.jl:108 [18] getindex(::JLD2.JLDFile{JLD2.MmapIO}, ::String) at /Users/xiaotaoyang/.julia/packages/JLD2/KjBIK/src/JLD2.jl:330 [19] top-level scope at none:0 `

You can recreate the errors with the following lines:

using SeisIO, JLD2 f=jldopen("cascadiaRMEQtestdatawithgap.jld2") S=f["2011.314/7D.J34A..HHZ"]

Here I attached the data file for your test. cascadiaRMEQtestdatawithgap.jld2.zip

Thanks, Xiaotao

jpjones76 commented 5 years ago

Hi Xiaotao,

Unfortunately, JLD2 is not in the scope of SeisIO, but I can give you some advice and one possible workaround.

Ultimately it's unsafe to use JLD2 with a package whose Types aren't finalized. JLD2 assumes that Types never change; it saves a Type name and a host package name, but doesn't save the low-level Type definition itself. So, if a Type changes, JLD2 can't read the data in again, unless you roll back the package to the version you were using when you created the file.

In this case, I changed PZResp on August 1 and JLD2 is trying to read PZResp objects into the new Type structure, rather than the old one. However, this will happen with any Type change.

There's one possible workaround, but it's tedious:

  1. Create a conversion script for your data. (I can write one for you; in fact, it's better to have me do that, so please tell me if you want one)
  2. Roll back your SeisIO install to the last release version (v0.3).
  3. In SeisIO v0.3, read the old JLD2 files into memory.
  4. Apply the conversion script to write the corrected structures to disk in SeisIO data format. (Don't write to JLD2 in this step, or you'll have the same problem again.)
  5. Reinstall SeisIO from the GitHub "master" branch.
  6. Read the new SeisIO files into memory with rseis.
  7. Write the new files to disk in JLD2 format.

However, I need to repeat my earlier warning, here: this will happen again (possibly many times) if you continue using JLD2. I've stated in many places (including Skype conversations) that not all SeisIO Types are finalized; yet any change to any Type breaks JLD2.

I'm very sorry that I can't be more helpful. A conversion script is the most I can do.

xtyangpsp commented 5 years ago

Hi Josh,

Thanks very much for your response. Sorry to report it as an issue under SeisIO. Clearly, as you stated, it is JLD2’s issue or it was designed that way. You can close this issue on GitHub.

I will discuss with the group to find a long term solution in saving SeisData.

Appreciate your explanations. I will reach out to you if we decide to convert the old files (again, appreciate your thought).

Thanks, Xiaotao On Aug 26, 2019, 21:53 -0400, Joshua Jones notifications@github.com, wrote:

Hi Xiaotao, Unfortunately, JLD2 is not in the scope of SeisIO, but I can give you some advice and one possible workaround. Ultimately it's unsafe to use JLD2 with a package whose Types aren't finalized. JLD2 assumes that Types never change; it saves a Type name and a host package name, but doesn't save the low-level Type definition itself. So, if a Type changes, JLD2 can't read the data in again, unless you roll back the package to the version you were using when you created the file. In this case, I changed PZResp on August 1 and JLD2 is trying to read PZResp objects into the new Type structure, rather than the old one. However, this will happen with any Type change. There's one possible workaround, but it's tedious:

  1. Create a conversion script for your data. (I can write one for you; in fact, it's better to have me do that, so please tell me if you want one)
  2. Roll back your SeisIO install to the last release version (v0.3).
  3. In SeisIO v0.3, read the old JLD2 files into memory.
  4. Apply the conversion script to write the corrected structures to disk in SeisIO data format. (Don't write to JLD2 in this step, or you'll have the same problem again.)
  5. Reinstall SeisIO from the GitHub "master" branch.
  6. Read the new SeisIO files into memory with rseis.
  7. Write the new files to disk in JLD2 format.

However, I need to repeat my earlier warning, here: this will happen again (possibly many times) if you continue using JLD2. I've stated in many places (including Skype conversations) that not all SeisIO Types are finalized; yet any change to any Type breaks JLD2. I'm very sorry that I can't be more helpful. A conversion script is the most I can do. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

jpjones76 commented 5 years ago

Hi Xiaotao,

I'll create a conversion script for you tonight. If you run it in SeisIO v0.3, it'll update the PZResp structure to the current definition. I'll close this issue once the script works, ok?

jpjones76 commented 5 years ago

As it turns out, no workaround script was needed. Check that your copy of SeisIO "master" branch is up-to-date. Then follow this procedure:

  1. ROLLBACK TO SeisIO v0.3
  2. READ IN JLD2 FILE AND SAVE WITH wseis
  3. CHANGE SYMBOLIC LINK IN ~.julia/dev BACK TO SEISIO "MASTER"
  4. READ .SEIS FILE INTO SEISIO WITH RSEIS; CHECK THAT IT WORKED

You can see from the attached console log that following these steps works. After that you can write the SeisData structures to new JLD2 files if you want to.

console.log

I hope this fixes it!