eliminate lag between registration of tag and ability to detect it

[jbrzusto]

Currently, we cache metadata such as tag registrations, and update them once per day from motus.org for these reasons:

• we always have a version of the tag registrations to run the tag finder against, regardless of the status of the motus.org server
• we can maintain a record of metadata in the metadata-history repo, so that we can determine exactly what metadata the tagfinder was run with by recording the metadata commit hash as one of the batch parameters
• the tag finder or its R harness would otherwise have to make a call to fetch the tag registration database from motus.org with each run, which would be significant traffic, at times.

The downside is a delay of up to 24 hours between the time a tag is registered and the time it is available for the tag-finder to detect in new runs of data on the sgdata server. This fails for users who want to register tags and then detect them in uploaded test data the same day. (However, those users can see live detections of newly-registered on a sensorgnome's web interface if they download and install the tag database which is generated on sgdata.motus.org when users register tags.)

Options for Improvement

• ~add an API on the sgdata server that can be called by the motus.org server to indicate a new tag has been registered. Requires new code on both servers.~

or

• when handleRegisterTags() registers a batch of tags with motus.org, it can make its own changes to both the meta data DB and the metadata-history repo (with appropriate locking). Only requires new code on sgdata.motus.org

[jbrzusto]

We'll do this on sgdata. refreshMotusMetaDBCache():

• [x] factor out update of metadata for a set of tags into a new function updateMetadataForTags(t, meta, fixBI) with:
  • t: tag data.frame in the format returned by motusSearchTags()
  • meta: safeSQL object to meta database where tables will reside
  • p: data.frame of motus projects, as returned by motusListProjects(); default: meta("select * from projs")
  • fixBI: logical; if TRUE, fix burst intervals, otherwise, don't. This should only be set TRUE when updating all tags, because it uses a majoritarian rule to select "accurate" BIs from those in t. (refreshMotusMetaDBCache() is currently calling cleanTagRegistrations() with the default cleanBI=FALSE anyway, so this parameter is only included in case of future changes)
• [x] factor out commits on metadata-history git repo into new function commitMetadataHistory(meta) with:
  • meta: safeSQL object to meta database with meta table where metadata hash will be stored
• [x] change table overwrites to use insert-or-replace semantics

handleRegisterTags():

• [x] record motusTagIDs of newly-registered tags
• [x] call motusSearchTags(projectID) then subset only these tags
• [x] call new function updateMetadataForTags() with these tags and fixBI=FALSE
• [x] call commitMetadataHistory()

cleanTagRegistrations():

• [x] change table overwrites to use insert-or-replace semantics

motusQuickTagReg():

• [x] call updateMetadataForTags() with the new tag
• [x] call commitMetadataHistory()

[jbrzusto]

testing:

• [x] refreshMotusMetaDBCache() - success: results in identical tables tags, tagsDeps
• [x] handleRegisterTags() - success: two bogus tags registered to project 1

  tags:
  31811,"Lotek","NTQB-3-2","Lotek4",166.38,"332",44.6974983215332
  31812,"Lotek","NTQB-3-2","Lotek4",166.38,"334",44.6972999572754
  tag_deployments:
  31811,1,1538352000,1566128693.30461,1,3
  31812,1,1538352000,1566128667.32612,1,3

  and updated in metadata cache and metadata history.

• [x] motusQuickRegTag() - success: bogus tag 31813 registered to project 1:

  tags:
  31813,"Lotek","NTQB-3-2","Lotek4",166.38,"337",44.6973991394043
  tag_deployments:
  31813,1,1538352000,1566128680.31539,1,3