--resume is broken

[jbrzusto]

in find_tags_motus version 2017-07-32-g4dd8f5f built Fri Dec 22 04:12:22 UTC 2017 e.g. running boot session 389 on SG-1614BBBK1919, splitting files from then into two groups:

2140$ sqlite3 ./SG-1614BBBK1919.motus
sqlite> create table save_files as select * from files where monoBN=389 and ts > (select (min(ts) + max(ts))/2 from files where monoBN=389 and ts > 1e9);
sqlite> delete from files where monoBN=389 and ts > (select (min(ts) + max(ts))/2 from files where monoBN=389 and ts > 1e9);
sqlite> .quit
2140$ /sgm/bin/find_tags_motus --pulses_to_confirm=8 --frequency_slop=0.5 --min_dfreq=0 --max_dfreq=12 --pulse_slop=1.5 --burst_slop=4 --burst_slop_expansion=1 --use_events --max_skipped_bursts=20 --default_freq=166.376 --bootnum=389 --src_sqlite /sgm/cache/motus_meta_db.sqlite ./SG-1614BBBK1919.motus
Warning: malformed line in input
  at line 21:
1504011291.355,1,-m,166.376,0,
Warning: malformed line in input
  at line 23:
/%s/%sp
Max num candidates: 83 at 1507196111.84; now (1507213421.7835): 2
Done.
2141$ sqlite3 ./SG-1614BBBK1919.motus
sqlite> insert into files select * from save_files;
sqlite> drop table save_files;
sqlite> .quit
2142$ /sgm/bin/find_tags_motus --resume --pulses_to_confirm=8 --frequency_slop=0.5 --min_dfreq=0 --max_dfreq=12 --pulse_slop=1.5 --burst_slop=4 --burst_slop_expansion=1 --use_events --max_skipped_bursts=20 --default_freq=166.376 --bootnum=389 --src_sqlite /sgm/cache/motus_meta_db.sqlite ./SG-1614BBBK1919.motus
resumed successfully
Warning: malformed line in input
  at line 208248:
p1,1508402234.0364,3.88
Warning: malformed line in input
  at line 208250:
1,-70.99,-77.26
Warning: malformed line in input
  at line 595657:
p1,1508402234.0364,3.88
Warning: malformed line in input
  at line 595659:
1,-70.99,-77.26
Max num candidates: 0 at 0; now (0): 1
Done.
2143$ /sgm/bin/find_tags_motus --pulses_to_confirm=8 --frequency_slop=0.5 --min_dfreq=0 --max_dfreq=12 --pulse_slop=1.5 --burst_slop=4 --burst_slop_expansion=1 --use_events --max_skipped_bursts=20 --default_freq=166.376 --bootnum=389 --src_sqlite /sgm/cache/motus_meta_db.sqlite ./SG-1614BBBK1919.motus
Warning: malformed line in input
  at line 21:
1504011291.355,1,-m,166.376,0,
Warning: malformed line in input
  at line 23:
-%m-%d�
Warning: malformed line in input
  at line 208248:
p1,1508402234.0364,3.88
Warning: malformed line in input
  at line 208250:
1,-70.99,-77.26
Max num candidates: 130 at 1510084546.0253; now (1510414581.3767): 6
Done.
2144$ sqlite3 ./SG-1614BBBK1919.motus
sqlite> select * from batches where monoBN=389;
19||389|1504011291.644|1507213421.7835|357|1517345914.59739|||
20||389|0.0|0.0|0|1517345919.38726|||
21||389|1504011291.644|1510414581.3767|387|1517345923.34939|||
sqlite> 

Batch 21 has 387 hits, and batch 19 (first half of files) has 357, so batch 20 should have the remaining 30 hits. But it doesn't, and its timestamps are not filled in.

[jbrzusto]

something to do with SG_SQLite_Data_Source::rewind()? middle run in previous comment has duplicate warnings of identical line at different line numbers

[jbrzusto]

This bug was introduced in eeaf9b8a:

• in order to filter records with bogus timestamps, we added the field Clock_Repair::max_ts which was initialized to the current real time plus 5 minutes of slop
• the field was set by a static method called from the parameterized Tag_Foray constructor
• it was not called by the default Tag_Foray constructor that is used when deserializing state (i.e. when --resume is specified), leaving the value at its default of 0, which meant all records were being dropped due to overlarge timestamps!

Fix

• ~have the Clock_Repair::Clock_Repair() set max_ts (after all, why was Tag_Foray setting it?)~
• have both Clock_Repair::Clock_Repair(...) ~chain to Clock_Repair::Clock_Repair(), thereby setting max_ts appropriately after deserializing (this chaining requires recompiling with g++ -std=c++13.~ call a common init function()