jbrzusto / TO_DO

sensorgnome / motus TODO list for jbrzusto
0 stars 0 forks source link

default frequency cleanup #195

Open jbrzusto opened 6 years ago

jbrzusto commented 6 years ago

The tag finder needs to know what frequency a receiver is listening at in order to select the set of tags it should be looking for. This normally happens by parsing the frequency-setting parameters in the raw data files, but sometimes those records are absent, typically because users haven't uploaded files dated "2000", which are recorded before the GPS sets the system clock. In this case, the tag finder uses the default frequency specified as the command-line parameter default_freq, which itself defaults to the Western Hemisphere default of 166.376. Many projects use a different default, and these are provided as parameter overrides in the paramOverrides table in the database /sgm_local/motus_meta_db.sqlite. These overrides can be specified by project or by receiver and by time period.

To be fixed:

sqlite> select t1.projectID, t2.nomFreq, count(t2.tagID) as numTags,
datetime(min(t1.tsStart), "unixepoch") as tagTsStart, 
datetime(max(t1.tsEnd), "unixepoch") as tagTsEnd, 
t3.id as ruleID, t3.serno, datetime(t3.tsStart,'unixepoch') as ruleTsStart, 
datetime(t3.tsEnd, 'unixepoch') as ruleTsEnd, t3.paramName, t3.paramVal, t3.why
from tagDeps as t1 join tags as t2 on t1.tagID=t2.tagID
left join paramOverrides as t3 on t1.projectID=t3.projectID 
   and (    (t3.tsStart is null or t3.tsStart < t1.tsEnd) 
        and (t3.tsEnd is null or t3.tsEnd > t1.tsStart)) 
group by t1.projectID, t2.nomFreq 
having t2.nomFreq != 166.38 
order by t1.projectID, t2.nomFreq, t3.id;
projectID nomFreq numTags tagTsStart tagTsEnd ruleID serno ruleTsStart ruleTsEnd paramName paramVal why
8 166.3 204 2013-08-14 12:00:00 2013-10-16 11:48:03 6 SG-0613BB000613 burst_slop 10.0 higher time variance due to frequency switching?
9 166.3 30 2013-08-05 16:44:00 2013-09-30 00:00:00
13 166.3 178 2012-08-01 12:00:00 2013-10-03 16:49:15
28 166.3 6 2013-07-11 18:00:00 2013-12-30 00:00:00
36 166.3 5 2016-09-08 12:00:00 2016-11-30 21:48:16
41 150.1 29 2016-04-01 00:00:00 2016-12-01 03:52:06
41 160.4 2 2016-07-01 00:00:00 2016-09-29 00:00:00
62 166.32 6 2018-04-01 00:00:00 2018-08-10 12:22:40
62 166.36 11 2018-04-01 00:00:00 2018-08-10 12:22:54
76 149.4 5 2015-04-19 14:25:00 2015-09-18 17:41:11 37 2015-08-01 00:00:00 2016-04-05 06:21:40 default_freq 150.336 project frequency for 2015 tags
76 150.1 540 2016-07-01 00:00:00 2018-06-23 17:59:15 2 2016-08-24 15:53:20 default_freq 150.096 project frequency for fall 2016 tags
76 150.34 337 2015-01-01 00:00:00 2017-06-25 15:08:11 1 2016-04-05 06:21:40 2016-08-25 07:26:40 default_freq 150.336 project frequency for spring 2016 tags
76 150.44 2 2015-01-01 00:00:00 2015-04-27 19:53:11
85 160.38 1 2016-12-23 00:00:00 2017-06-29 15:22:59
97 999.999 1 2016-04-01 00:00:00 2016-05-20 07:44:08
108 166.3 26 2016-01-01 00:00:00 2016-07-01 15:37:36
113 150.1 27 2016-07-01 00:00:00 2018-09-16 14:40:03 36 2017-01-01 00:00:00 default_freq 150.1 project frequency for all 2017 tags
114 150.1 20 2016-08-21 15:00:00 2016-10-11 11:51:13 5 2016-01-01 00:00:00 default_freq 150.096 project frequency for all 2016 tags
121 150.1 6 2017-09-23 00:00:00 2017-11-21 01:34:58
123 150.1 1239 2017-01-01 00:00:00 2018-09-03 19:00:28 12 2017-01-01 00:00:00 default_freq 150.096 project frequency for all 2017 tags
170 151.5 27 2017-09-03 09:30:00 2017-11-10 07:14:41
173 150.1 80 2017-10-12 08:27:00 2018-07-05 03:20:50
177 151.5 49 2017-10-01 00:00:00 2020-09-05 05:11:35 59 default_freq 151.5 project frequency for all 2018 tags
182 150.1 11 2018-06-06 00:00:00 2018-07-29 18:31:08
186 150.1 55 2017-07-01 00:00:00 2018-06-30 23:25:15
211 150.1 52 2018-08-01 00:00:00 2018-12-13 18:23:03
jbrzusto commented 6 years ago

I'm adding these paramOverride records:

sqlite> select t1.projectID as projectID, 'find_tags_motus' as progName, 'default_freq' as paramName, 'project listening frequency inferred from tag registrations' as why, t2.nomFreq as paramVal, datetime(min(t1.tsStart), "unixepoch") as tsStart, datetime(max(t1.tsEnd), "unixepoch") as tsEnd from tagDeps as t1 join tags as t2 on t1.tagID=t2.tagID left join paramOverrides as t3 on t1.projectID=t3.projectID and ((t3.tsStart is null or t3.tsStart < t1.tsEnd) and (t3.tsEnd is null or t3.tsEnd > t1.tsStart) ) group by t1.projectID, t2.nomFreq having t2.nomFreq != 166.38 and t3.id is null and t1.projectID != 97 order by t1.projectID, t2.nomFreq, t3.id;
projectID progName paramName why paramVal tsStart tsEnd
9 find_tags_motus default_freq project listening frequency inferred from tag registrations 166.3 2013-08-05 16:44:00 2013-09-30 00:00:00
13 find_tags_motus default_freq project listening frequency inferred from tag registrations 166.3 2012-08-01 12:00:00 2013-10-03 16:49:15
28 find_tags_motus default_freq project listening frequency inferred from tag registrations 166.3 2013-07-11 18:00:00 2013-12-30 00:00:00
36 find_tags_motus default_freq project listening frequency inferred from tag registrations 166.3 2016-09-08 12:00:00 2016-11-30 21:48:16
41 find_tags_motus default_freq project listening frequency inferred from tag registrations 150.1 2016-04-01 00:00:00 2016-12-01 03:52:06
41 find_tags_motus default_freq project listening frequency inferred from tag registrations 160.4 2016-07-01 00:00:00 2016-09-29 00:00:00
62 find_tags_motus default_freq project listening frequency inferred from tag registrations 166.32 2018-04-01 00:00:00 2018-08-10 12:22:40
62 find_tags_motus default_freq project listening frequency inferred from tag registrations 166.36 2018-04-01 00:00:00 2018-08-10 12:22:54
76 find_tags_motus default_freq project listening frequency inferred from tag registrations 150.44 2015-01-01 00:00:00 2015-04-27 19:53:11
85 find_tags_motus default_freq project listening frequency inferred from tag registrations 160.38 2016-12-23 00:00:00 2017-06-29 15:22:59
108 find_tags_motus default_freq project listening frequency inferred from tag registrations 166.3 2016-01-01 00:00:00 2016-07-01 15:37:36
121 find_tags_motus default_freq project listening frequency inferred from tag registrations 150.1 2017-09-23 00:00:00 2017-11-21 01:34:58
170 find_tags_motus default_freq project listening frequency inferred from tag registrations 151.5 2017-09-03 09:30:00 2017-11-10 07:14:41
173 find_tags_motus default_freq project listening frequency inferred from tag registrations 150.1 2017-10-12 08:27:00 2018-07-05 03:20:50
182 find_tags_motus default_freq project listening frequency inferred from tag registrations 150.1 2018-06-06 00:00:00 2018-07-29 18:31:08
186 find_tags_motus default_freq project listening frequency inferred from tag registrations 150.1 2017-07-01 00:00:00 2018-06-30 23:25:15
211 find_tags_motus default_freq project listening frequency inferred from tag registrations 150.1 2018-08-01 00:00:00 2018-12-13 18:23:03
jbrzusto commented 6 years ago

so re-running these projects: