Restart of large download failed with "UNIQUE constraint failed: rover_lock_mseed.pid"

[chad-earthscope]

The setup in an otherwise clean system:

$ cat ta.req 
TA * * LH? 2012-01-01T00:00:00 2012-02-01T00:00:00
TA * * BH? 2012-01-01T00:00:00 2012-02-01T00:00:00
$ rover retrieve ta.req 

Pressed Control-C to cancel the downloads after a minute or so and then attempt to restart the download, results:

$ rover retrieve ta.req 
retrieve  DEFAULT: Status available at http://127.0.0.1:8000
retrieve  DEFAULT: No email status (see --email configuration parameter)
retrieve CRITICAL: UNIQUE constraint failed: rover_lock_mseed.pid
retrieve  DEFAULT: See "rover help retrieve"
$

There are no other rover processes running at this time. There is no rover_lock_mseed.pid file.

The data/index.sql file is attached: index.sql.zip

[rtweekly-iris]

I was able to recreate this error after immediately deleting some data that was just downloaded.

$ rover retrieve TA_A*_*_* 2014-05-01 2014-05-03

$ rm -rf data/TA

$ cat POST_TEST_big.dat 
TA A* * * 2014-05-01T00:00:00 2014-05-03T00:00:00

$ rover retrieve POST_TEST_big.dat 
retrieve  DEFAULT: Rover version 0.0.4 - starting retrieve
retrieve  DEFAULT: Status available at http://127.0.0.1:8000
retrieve CRITICAL: UNIQUE constraint failed: rover_lock_mseed.pid
retrieve  DEFAULT: See "rover help retrieve"

$ rover retrieve POST_TEST_big.dat 
retrieve  DEFAULT: Rover version 0.0.4 - starting retrieve
retrieve  DEFAULT: Status available at http://127.0.0.1:8000
retrieve  WARNING: Cleaning out old entry for PID 79134 on lock rover_lock_mseed with /Users/rtweekly/Testing/rover/datarepo/data/US/1993/269/DUG.US.1993.269 (created 2018-08-15T23:38:59.000000)
retrieve CRITICAL: UNIQUE constraint failed: rover_lock_mseed.pid
retrieve  DEFAULT: See "rover help retrieve"

Additionally, that WARNING about US_DUG data seems to persist through other calls to retrieve method, despite no data from US_DUG being requested.

$ cat POST_TEST_4.dat 
AV OKCF * * 2010-01-05T00:00:00 2010-02-05T00:00:00

$ rover retrieve POST_TEST_4.dat 
retrieve  DEFAULT: Rover version 0.0.5 - starting retrieve
retrieve  DEFAULT: Status available at http://127.0.0.1:8000
retrieve  DEFAULT: Email status will be sent to rtweekly@iris.washington.edu
retrieve  DEFAULT: Trying new retrieval (attempt 1 of 3)
retrieve  DEFAULT: Downloading AV_OKCF__EHZ (1/1); day 1/735
retrieve  DEFAULT: Downloading AV_OKCF__EHZ (1/1); day 2/735
retrieve  DEFAULT: Downloading AV_OKCF__EHZ (1/1); day 3/735
retrieve  DEFAULT: Downloading AV_OKCF__EHZ (1/1); day 4/735
retrieve  DEFAULT: Downloading AV_OKCF__EHZ (1/1); day 5/735
download.83385  WARNING: Cleaning out old entry for PID 79250 on lock rover_lock_mseed with /Users/rtweekly/Testing/rover/datarepo/data/US/1993/269/DUG.US.1993.269 (created 2018-08-15T23:41:08.000000)
download.83385  WARNING: Cleaning out old entry for PID 79250 on lock rover_lock_mseed with /Users/rtweekly/Testing/rover/datarepo/data/US/1993/269/DUG.US.1993.269 (created 2018-08-15T23:41:08.000000)
download.83388  WARNING: Cleaning out old entry for PID 79250 on lock rover_lock_mseed with /Users/rtweekly/Testing/rover/datarepo/data/US/1993/269/DUG.US.1993.269 (created 2018-08-15T23:41:08.000000)
download.83388    ERROR: database is locked
download.83388    ERROR: database is locked
download.83388 CRITICAL: database is locked
download.83388  DEFAULT: See "rover help download"
...

[andrewcooke-isti]

this is tested in 2f6dda204b3ffb589441a8ee90fd284fe1badf50 and fixed (mainly) by 8bf02c9d992c79e15276c455231af7cb3825975b