Download failed (return code 1), No SEED data detected, overlapped data

[chad-earthscope]

For an existing repository I added one day to the end time of this selection:

TA MSTX -- BH? 2012-01-01T00:00:00 2012-02-04T00:00:00

then I ran rover retrieve again and got the error below:

retrieve  DEFAULT: Rover version 0.0.4 - starting retrieve
retrieve  DEFAULT: Status available at http://127.0.0.1:8000
retrieve  DEFAULT: Trying new retrieval (attempt 1 of 3)
retrieve  DEFAULT: Downloading TA_MSTX__BHE (3/5); day 1/2
retrieve  DEFAULT: Downloading TA_MSTX__BHE (3/5); day 2/2
retrieve  DEFAULT: Downloading TA_MSTX__BHN (4/5); day 1/2
retrieve  DEFAULT: Downloading TA_MSTX__BHN (4/5); day 2/2
retrieve  DEFAULT: Downloading TA_MSTX__BHZ (5/5); day 1/2
ERROR: Cannot read /Users/chad/tmp/td/tmp/rover_download_15bac3_86226: No SEED data detected
download.86226 CRITICAL: Command "LIBMSEED_LEAPSECOND_FILE=/Users/chad/tmp/td/leap-seconds.list mseedindex -sqlitebusyto 60000 -sqlite /Users/chad/tmp/td/tmp/rover_ingester_15bac3_86226 /Users/chad/tmp/td/tmp/rover_download_15bac3_86226" failed
download.86226  DEFAULT: See "rover help download"
retrieve    ERROR: Download failed (return code 1)
retrieve  DEFAULT: Downloading TA_MSTX__BHZ (5/5); day 2/2
retrieve  DEFAULT: The latest retrieval attempt completed with 1 errors after 1 attempts. We will retry to check that all data were retrieved

One possibility: that is the error message expected from mseedindex if the file is empty or no SEED data is otherwise detected.

[chad-earthscope]

I was able to reproduce this by performing a rover retrieve request.txt, where request.txt is:

TA MSTX -- BH? 2012-01-01T00:00:00 2012-02-03T00:00:00

Then modify the request to add a single day to the end time, aka:

TA MSTX -- BH? 2012-02-01T00:00:00 2012-02-04T00:00:00

to get:

ERROR: Cannot read /Users/chad/tmp/td/tmp/rover_download_15bac3_88867: No SEED data detected
download.88867 CRITICAL: Command "LIBMSEED_LEAPSECOND_FILE=/Users/chad/tmp/td/leap-seconds.list mseedindex -sqlitebusyto 60000 -sqlite /Users/chad/tmp/td/tmp/rover_ingester_15bac3_88867 /Users/chad/tmp/td/tmp/rover_download_15bac3_88867" failed
download.88867  DEFAULT: See "rover help download"
retrieve    ERROR: Download failed (return code 1)

[chad-earthscope]

The miniSEED data from the reproduction run now contains a single sample overlap for all channels:

$ msi -tg data/TA/2012/03*/*
   Source                Start sample             End sample        Gap  Hz  Samples
TA_MSTX__BHE      2012,032,00:00:00.000000 2012,033,23:59:59.975000  ==  40  6912000
TA_MSTX__BHE      2012,033,23:59:59.975000 2012,034,23:59:59.975000 -0   40  3456001
TA_MSTX__BHN      2012,032,00:00:00.000000 2012,033,23:59:59.975000  ==  40  6912000
TA_MSTX__BHN      2012,033,23:59:59.975000 2012,034,23:59:59.975000 -0   40  3456001
TA_MSTX__BHZ      2012,032,00:00:00.000000 2012,034,23:59:59.975000  ==  40  10368000

[chad-earthscope]

To reproduce:

rover init-repository testrepo
cd testrepo
rover retrieve TA_MSTX_--_BH? 2012-02-02T00:00:00 2012-02-03T00:00:00
rover retrieve TA_MSTX_--_BH? 2012-02-02T00:00:00 2012-02-04T00:00:00

after the first retrieve the miniSEED is:

$ msi -tg data/TA/2012/033/*
   Source                Start sample             End sample        Gap  Hz  Samples
TA_MSTX__BHE      2012,033,00:00:00.000001 2012,033,23:59:59.975000  ==  40  3456000
TA_MSTX__BHN      2012,033,00:00:00.000001 2012,033,23:59:59.975000  ==  40  3456000
TA_MSTX__BHZ      2012,033,00:00:00.000000 2012,033,23:59:59.999998  ==  40  3456001
Total: 3 trace(s) with 3 segment(s)

The second retrieve generates these requests:

1. [200] POST /irisws/availability/1/query HTTP/1.1
2. [200] GET /fdsnws/dataselect/1/query?net=TA&sta=MSTX&loc=--&cha=BHN&start=2012-02-02T23:59:59.975000&end=2012-02-02T23:59:59.999999 HTTP/1.1
3. [200] GET /fdsnws/dataselect/1/query?net=TA&sta=MSTX&loc=--&cha=BHE&start=2012-02-02T23:59:59.975000&end=2012-02-02T23:59:59.999999 HTTP/1.1
4. [204] GET /fdsnws/dataselect/1/query?net=TA&sta=MSTX&loc=--&cha=BHZ&start=2012-02-02T23:59:59.999998&end=2012-02-02T23:59:59.999999 HTTP/1.1
5. [200] GET /fdsnws/dataselect/1/query?net=TA&sta=MSTX&loc=--&cha=BHE&start=2012-02-03T00:00:00.000000&end=2012-02-03T23:59:59.999999 HTTP/1.1
6. [200] GET /fdsnws/dataselect/1/query?net=TA&sta=MSTX&loc=--&cha=BHN&start=2012-02-03T00:00:00.000000&end=2012-02-03T23:59:59.999999 HTTP/1.1
7. [200] GET /fdsnws/dataselect/1/query?net=TA&sta=MSTX&loc=--&cha=BHZ&start=2012-02-03T00:00:00.000000&end=2012-02-03T23:59:59.999999 HTTP/1.1
8. [200] POST /irisws/availability/1/query HTTP/1.1
9. [200] POST /irisws/availability/1/query HTTP/1.1

A couple of problems here:

• The first three requests are erroneously trying to "fill out" day 2012-02-02 even though:
  • they include the time of the last sample already retrieved, this results in duplication
  • the requested time span is less than the a sample period, i.e. the next sample after the last one retrieved cannot be in that window
• The 4th requests results in an HTTP 204, i.e. "no data". rover appears to ignore this code, write an empty file and try to index that file, resulting in the "No SEED data detected" report and failure of mseedindex.

[chad-earthscope]

msi is a tool for parsing/printing/diagnosing miniSEED data.

[andrewcooke-isti]

yes, i googled (i thought i deleted my question - sorry for the bother).