Allow user to run a script to pull in data

[colinpmillar]

Summary

this would work like:

@Misc{[name],
  originator = {WGEF},
  year       = {2015},
  title      = {Survey indices: IBTS1, IBTS3, Unknown, and UKBTS3},
  period     = {1977-2015},
  source     = {script},
}

the name allows us to trace the directory or file that the record is referencing.

Tasks required

• [ ] scope out ideas on how to implement
• [ ] test and implement in R
• [ ] test and implement on TAF server

Links to other issues?

[colinpmillar]

from @arnima-github: I've created a basic pathway to run a custom bootstrap data script. See current version of icesTAF on GitHub (master branch is currently same as develop). The corresponding test analysis is at https://github.com/ices-taf-dev/bootscript.

[colinpmillar]

some potential ways forward:

1. simple source
2. variations of source(local=TRUE), sys.source(envir=new.env()), local(source()), etc
3. source = {script}, forcing script to be the same name as the data output. All output could be put in a folder of the same name as the script possibly

We could then screen the bootstrap/data folder for files or folders that are not reference in the DATA.bib file and warn or delete them

[colinpmillar]

Based on discussions it seems a combo of 1. and 2. from above would be easiest. For example, the given the following record:

@Misc{datras,
  originator = {ICES},
  year       = {2019},
  title      = {Survey indices: IBTS1, IBTS3, Unknown, and UKBTS3},
  period     = {1977-2019},
  source     = {script},
}

The rules would be:

• Assume the existence of an R script datras.R.
• The script can run in isolation (harder to implement on the users machine - but we can do so on the server side).
• All files created by the script are placed inside bootstrap/data/datras/ folder.
• If someone wants to download this from outside of ICES, they will get a zip of the folder and the metadata will show the type as a "collection".

This should ensure that all files and folders in the bootstrap/data/ folder have a name corresponding to a DATA.bib entry while still being relatively easy to implement in icesTAF::process.bib.

[colinpmillar]

brainstorming:

• the R script produces files in the working directory
• process.bib will set the working directory accordingly

instruction to user: "please do not specify target directory"