Uploaded Data is Missing

leberrigan commented 6 years ago

Projects: 2 (Taylor Receivers) and 10 (Motus Atlantic Array) Receivers: All from June 1 - Dec 1, 2017

All towers were functioning normally when downloading data and had recorded folders for each dy with decently-sized '.gz'. This data was uploaded by myself (lberrigan) and Dave (david.bell) in from early to mid-November, but there are many large gaps in the uploaded data that don't make sense.

For example, the Pubnico receiver (4001BBBK2010) is missing data from between Aug 28 and Nov 21, 2017, but Dave uploaded the files from this period and the tower had been visited at least twice in this time and showed no apparent issues. https://sgdata.motus.org/download/2/SG-4001BBBK2010-2017_MotusATL_Pubnico_hourly_tags.png

Is this a software issue? I believe they are all running the June or Sept 2016 software release.

jbrzusto commented 6 years ago

Not a receiver issue. Likely a problem with tag finder resuming some batches from stale saved state

Will check off when completed:

leberrigan commented 6 years ago

Here are some additional receivers we have found to have similar data issues:

beaubassin - SG-1614BBBK1915 Black Rock - SG-4000BBBK9170 Blandford - SG-1614BBBK1889 Canso - SG-2313BBBK3402 cape jourimain - SG-1614BBBK1611 Church Pt - SG-4002BBBK1630 clam bay - SG-1614BBBK1919 East Walton - SG-4005BBBK4310 gardner creek - SG-4000BBBK9340 Hampton - SG-4001BBBK1660 Joggins - SG-4002BBBK0360 Keji - SG-4002BBBK2630 Kingsburg2 - SG-4001BBBK2600 Lockhartville - SG-1614BBBK1662 Lookoff - SG-1614BBBK1603 Marshalltown - SG-1614BBBK1832 Matthew's Lake - SG-4001BBBK0910 Miscou - SG-1614BBBK1903 mount thom - SG-4002BBBK1980 point lepreau - SG-4001BBBK7590 Port Joli - SG-4001BBBK2380 port maitland - SG-4001BBBK6020 Prospect 3 - SG-4002BBBK1530 Selma - SG-4000BBBK9550 Sonora - SG-1614BBBK1916 Taylor Head - SG-4001BBBK0350 Truro - SG-4002BBBK2240 Upper Clements - SG-1614BBBK1869 West Quoddy - SG-1614BBBK1956

jbrzusto commented 6 years ago

Just looking at Baccaro SG-1614BBBK1880, these are the relevant batch records:

sqlite> select * from batches where monoBN=403;
batchID     motusDeviceID  monoBN      tsStart         tsEnd            numHits     ts                motusUserID  motusProjectID  motusJobID
----------  -------------  ----------  --------------  ---------------  ----------  ----------------  -----------  --------------  ----------
18          33             403         1491663004.147  1503423910.7997  916         1505234754.94926                               76035     
19          33             403         1491663004.147  1503423910.7997  852         1512769246.72707  2525         2               104048    
20          33             403         0.0             0.0              0           1514529181.45721  2525         2               114330    
21          33             403         0.0             0.0              0           1514604460.32886  504          2               119687    
33          33             403         0.0             0.0              0           1517237512.09842  2525         2               127053    
52          33             403         1491663004.147  1512901622.8546  1437        1517252388.39442                               127509

Batch 20 should have been a resumption of batch 19, since it was processing a bunch of new files covering August 1 to October 22.

The relevant parameter records are:

sqlite> select * from batchParams where paramName="resume";
batchID     progName         paramName   paramVal  
----------  ---------------  ----------  ----------
1           find_tags_motus  resume      1.0       
2           find_tags_motus  resume      0.0       
20          find_tags_motus  resume      1.0       
22          find_tags_motus  resume      0.0       
33          find_tags_motus  resume      1.0       
34          find_tags_motus  resume      0.0

showing that batch 20 was run with "--resume". One possibility: format of saved state changed between batches 19 and 20. Here are the relevant program version records

sqlite> select * from batchProgs;
batchID     progName         progVersion          progBuildTS 
----------  ---------------  -------------------  ------------
1           find_tags_motus  2017-07-16-g1e1306a  1504798324.0
19          find_tags_motus  2017-07-29-g2f2f0b2  1510856315.0
20          find_tags_motus  2017-07-32-g4dd8f5f  1513915942.0

So the program version changed. However, a diff between 2017-07-29-g2f2f0b2 and 2017-07-32-g4dd8f5 doesn't turn up anything that should affect serialization: no serialized class members were added, removed, or re-ordered!

jbrzusto commented 6 years ago

Other possibilities:

--resume is broken

Yes it is: jbrzusto/find_tags#48

jbrzusto commented 6 years ago

Rerunning all the above receivers today - sorry, dropped the ball on this.

jbrzusto / TO_DO

Uploaded Data is Missing #156