Proposal for caching behavior--comments.

[seandavi]

We desire a caching behavior for data coming from the NCI GDC. Features might include:

• Cache is used when a file is requested, optionally disabled if desired
• Option to invalidate any or all of the cache
• Cache location configurable
• Ability to use either GDC transfer mechanism (either data transfer endpoint or via the data transfer tool) as a download mechanism, configurable by user (https://docs.gdc.cancer.gov/API/Users_Guide/Downloading_Files/)
• Downloads use Access Token, if supplied, for controlled-access data

API suggestions:

• Combine functionality of gdcdata and transfer into one method that accepts file UUIDs and serves back files.
• Back the functionality in number 1 by a subclass of BiocFileCache that overrides the update and add methods to use the requested file transfer mechanism +/- Token.

Considerations:

• How much metadata should we mirror in the cache versus making calls to the files() endpoint to gather metadata, on demand?
• Conserve filenames?
• Reporting of what is in cache can be "smart" with respect to metadata (how many files from project x, or of type y.)

[seandavi]

Related to #40.

[seandavi]

For the time being, I have:

• combined the gdcdata and transfer functionality into gdcdata with an extra argument to force the use of gdc-client with the default being the data endpoint.
• created a getter, gdc_cache() and a setter gdc_set_cache() that uses an option to store the path to the "cache directory". The caching behavior is very simple. If the file to be transferred exists in the cache, use it (unless explicitly told not to). Otherwise, download.

At this time, I have not used BiocFileCache, but that is because 1) some of the caching using files was already done using files and 2) because I would like to take advantage of the more interesting features of BiocFileCache (metadata, etc.) in the future, but wanted caching behavior first and 3) because I wanted to treat the uuids as the primary key in the "database", enforcing uniqueness and such. As things stand, I leave the BiocFileCache layer to the user. I think doing so represents a nice balance for the time being.

[seandavi]

Related to #45

[mtmorgan]

can this be closed or are you still tracking this?

[seandavi]

We can close for now. I do want to revisit the BiocFileCache approach, but that is going to require a bit of retooling and can wait for another day.