search

jpjones76 / SeisIO.jl

Julia language support for geophysical time series data
http://seisio.readthedocs.org
Other
47 stars 21 forks source link

readmseed can't handle blank space after SEED blockettes #7

Closed anowacki closed 5 years ago

anowacki commented 5 years ago

(Continued from anowacki/Seis.jl#3.)

I get the following error when trying to read the sample miniSEED file distributed with PASSCAL's libmseed library:

julia> import Seis

julia> cd(dirname(pathof(Seis)))

julia> import SeisIO; SeisIO.readmseed("test/test_data/test.mseed")
[ Info: Recompiling stale cache file /Users/nowacki/.julia/compiled/v1.1/SeisIO/sUMCG.ji for SeisIO [e2f1b5a8-5fbb-5d5d-94aa-e39813b81559]
ERROR: No support for Blockette Type 0
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] parserec!(::SeisIO.SeisData, ::IOStream, ::Int64) at /Users/nowacki/.julia/dev/SeisIO/src/Formats/mSEED.jl:233
 [3] parsemseed!(::SeisIO.SeisData, ::IOStream, ::Int64) at /Users/nowacki/.julia/dev/SeisIO/src/Formats/mSEED.jl:392
 [4] #readmseed#193(::Bool, ::Int64, ::Function, ::String) at /Users/nowacki/.julia/dev/SeisIO/src/Formats/mSEED.jl:437
 [5] readmseed(::String) at /Users/nowacki/.julia/dev/SeisIO/src/Formats/mSEED.jl:428
 [6] top-level scope at none:0

I am on commit 3b3a0ad, having deved SeisIO:

julia> cd(dirname(pathof(SeisIO)))

shell> git show
commit 3b3a0ad05383f4ba73b278054f1a48198936b549 (HEAD -> master, tag: v0.1.2, origin/master, cache/heads/master)
Author: Joshua Jones <--->
Date:   Wed Feb 13 02:31:42 2019 -0800

    update README.md

[...]

Julia info:

julia> versioninfo()
Julia Version 1.1.0
Commit 80516ca202 (2019-01-21 21:24 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin14.5.0)
  CPU: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, ivybridge)
Environment:
  JULIA_EDITOR = mate -w

For reference, SAC and ObsPy read the file without any problem.

jpjones76 commented 5 years ago

I'm unable to follow the steps provided. I tried to install Seis.jl to replicate your example using the instructions on your project page, and I got this error:

(v1.1) pkg> add https://github.com/anowacki/Geodesics.jl https://github.com/anowacki/TauPy.jl https://github.com/anowacki/Seis.jl

Updating registry at ~/.julia/registries/General Updating git-repo https://github.com/JuliaRegistries/General.git Cloning git-repo https://github.com/anowacki/Geodesics.jl Updating git-repo https://github.com/anowacki/Geodesics.jl Cloning git-repo https://github.com/anowacki/TauPy.jl Updating git-repo https://github.com/anowacki/TauPy.jl Cloning git-repo https://github.com/anowacki/Seis.jl Updating git-repo https://github.com/anowacki/Seis.jl Resolving package versions... ERROR: Unsatisfiable requirements detected for package SAC [03de775e]: SAC [03de775e] log: ├─SAC [03de775e] has no known versions! └─restricted to versions * by Seis [dd80f4d8] — no versions left └─Seis [dd80f4d8] log: ├─possible versions are: 0.1.1 or uninstalled └─Seis [dd80f4d8] is fixed to version 0.1.1

If I don't import Seis.jl, however, the remainder of your example works correctly:

julia> import SeisIO [ Info: Precompiling SeisIO [e2f1b5a8-5fbb-5d5d-94aa-e39813b81559] julia> SeisIO.readmseed("Code/src/libmseed/example/test.mseed") SeisData with 1 channels (1 shown) " ID: IU.COLA.00.LHZ " => "" " NAME: IU.COLA.00.LHZ " => "" ... " X: -2.319e+05 " => "" -2.284e+05
-2.232e+05
-2.212e+05
...
(nx = 4200)
C: 0 open, 0 total

Confirming we have the same version:

julia> cd(dirname(pathof(SeisIO))) shell> git show commit 3b3a0ad05383f4ba73b278054f1a48198936b549 (HEAD -> master, tag: v0.1.2, origin/master, cache/heads/master) Author: Joshua Jones <---> Date: Wed Feb 13 02:31:42 2019 -0800

You mentioned in anowacki/Seis.jl#3 that you've been using code from SeisIO, particularly some of my mseed functions. Could there be an issue with duplicate function names in both projects?

(edit) I should clarify that the error text string "ERROR: No support for Blockette Type 0" isn't describing the actual problem. There's no Blockette Type 0 in SEEDManual_V2.4, as you know. The number reported in the error is the numeric value of the byte read in as the blockette type; readmseed is reading starting at the wrong position.

I strongly suspect that you're overwriting some part of my readmseed functionality with your own. Try reading in the file without first importing Seis.jl. What happens?

anowacki commented 5 years ago

The difference is the file, I think, not methods being overwritten. I get the same error, regardless of loading Seis, for the file in that repo which I name f1 below.

Probably easiest for debugging would be to just download the file in the repo. (I only did import Seis so that the path to the file could be found programmatically with pathof.)

Like you, I don't have a problem with the file you tried (f2 below).

The following should work okay just copy-pasting and is from a fresh REPL session:

julia> import SeisIO # Haven't loaded Seis
[ Info: Recompiling stale cache file /Users/nowacki/.julia/compiled/v1.1/SeisIO/sUMCG.ji for SeisIO [e2f1b5a8-5fbb-5d5d-94aa-e39813b81559]

julia> cd(dirname(pathof(SeisIO)))

shell> git rev-parse HEAD # Latest master
3b3a0ad05383f4ba73b278054f1a48198936b549

julia> f1 = download("https://github.com/anowacki/Seis.jl/raw/miniseed/test/test_data/test.mseed"); # Example miniSEED file distributed by PASSCAL I found on my computer, now in Seis.jl repo
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   151  100   151    0     0    668      0 --:--:-- --:--:-- --:--:--   665
100  8192  100  8192    0     0  20480      0 --:--:-- --:--:-- --:--:-- 20480

julia> SeisIO.readmseed(f1)
ERROR: No support for Blockette Type 0
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] parserec!(::SeisIO.SeisData, ::IOStream, ::Int64) at /Users/nowacki/.julia/dev/SeisIO/src/Formats/mSEED.jl:233
 [3] parsemseed!(::SeisIO.SeisData, ::IOStream, ::Int64) at /Users/nowacki/.julia/dev/SeisIO/src/Formats/mSEED.jl:392
 [4] #readmseed#193(::Bool, ::Int64, ::Function, ::String) at /Users/nowacki/.julia/dev/SeisIO/src/Formats/mSEED.jl:437
 [5] readmseed(::String) at /Users/nowacki/.julia/dev/SeisIO/src/Formats/mSEED.jl:428
 [6] top-level scope at none:0

julia> f2 = download("https://github.com/iris-edu/libmseed/raw/master/example/test.mseed");
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   143  100   143    0     0    616      0 --:--:-- --:--:-- --:--:--   616
100 18432  100 18432    0     0  12198      0  0:00:01  0:00:01 --:--:-- 15835

julia> SeisIO.readmseed(f2)
SeisData with 1 channels (1 shown)
"    ID: IU.COLA.00.LHZ                                                                    " => ""
"  NAME: IU.COLA.00.LHZ                                                                    " => ""
"   LOC: 0.000, 0.000, 0.000                                                               " => ""
"    FS: 1                                                                                 " => ""
"  GAIN: 1.000e+00                                                                         " => ""
  RESP:                                                                                   

" UNITS:                                                                                   " => ""
"   SRC:                                                                                   " => ""
" NOTES: 1 entries                                                                         " => ""
"  MISC: 0 items                                                                           " => ""
"     T: 2010-02-27T06:50:00 (0 gaps)                                                      " => ""
"     X: -2.319e+05                                                                        " => ""
        -2.284e+05                                                                        
        -2.232e+05                                                                        
        -2.212e+05                                                                        
          ...                                                                             
        (nx = 4200)                                                                       
     C: 0 open, 0 total

Just for reference, 'f1' in the Seis repo should look like:

julia> run(pipeline(`echo r mseed $f1; lh knetwk kstnm khole kcmpnm npts kzdate kztime b delta`, `sac`))
 SEISMIC ANALYSIS CODE [June 25 1991 (Version 10.6d-grh110/116)]  
 Copyright 1991 Regents of the University of California

  FILE: /var/folders/mx/t8fyspqs0rq_bvdgqpn8fn5m0000gp/T/juliarFySzd
  ------------------------------------------------------------------
    knetwk = NL
     kstnm = HGN
     khole = 00
    kcmpnm = BHZ
      npts = 5980
    kzdate = MAY 29 (149), 2003
    kztime = 02:13:22.043
         b = 0.0
     delta = 0.250E-01
Base.ProcessChain(Base.Process[Process(`echo r mseed '/var/folders/mx/t8fyspqs0rq_bvdgqpn8fn5m0000gp/T/juliarFySzd;' lh knetwk kstnm khole kcmpnm npts kzdate kztime b delta`, ProcessExited(0)), Process(`sac`, ProcessExited(0))], Base.DevNull(), Base.DevNull(), Base.DevNull())

Apologies that you had the misfortune of trying to install Seis.jl!

jpjones76 commented 5 years ago

Confirming that my reader currently fails on the file you included:

julia> readmseed("/home/josh/Downloads/test.mseed",v=3,swap=false) Added channel: NL.HGN.00.BHZ Position = 50 Blockette type to parse = 1000 Position = 58 Blockette type to parse = 0 ERROR: No support for Blockette Type 0

I'd been using test.mseed in the current libmseed source, which is the file that I thought you meant; is yours from an older libmseed version?

When I downloaded libmseed two days ago (via git clone https://github.com/iris-edu/libmseed), there was only one test.mseed present, whence the mix-up:

shell> find ~/Code/src/libmseed/ -name "test.mseed" -print /home/josh/Code/src/libmseed/example/test.mseed

I'm glad you found this, though, as I've tested readmseed hundreds of times without incident, from SeedLink live streams to Canadian DRM requests to Steve Malone's digitization of the MSH 1980 seismic data. This is the first serious readmseed bug I've encountered in a couple of years; #4 was a simple deprecation issue with a one-line fix.

jpjones76 commented 5 years ago

Issue identified: readmseed presently can't handle blank space between blockettes. Fix in testing.

anowacki commented 5 years ago

This was just a file included with libmseed from an old install of PQL, and the only miniSEED file I could find on my home computer! What terrible luck. But also fortunate I guess.

I have tracked the original down, and it does indeed come from libmseed. It seems that the example/test.mseed file was changed in iris-edu/libmseed@e48ae4e. The old version from the initial commit is exactly the one now in Seis.jl.

I also found a similar but not identical version in another instance of the libmseed repo in Earthworm's source here, which looks suspiciously like the exact same data, but filtered slightly differently and with a different start time:

f001

jpjones76 commented 5 years ago

OK, could you look up one more thing for me, since I don't have a working SAC install at home: what are the first few data point values in that file?

I want to use that version of test.mseed in my tests, since the structure is much more complicated than the current test file.

anowacki commented 5 years ago

If it's still of use, then an SAC ASCII-format version of the file is here:

example.sacalpha.txt