new file approach

[mdsumner]

This will be another upgrade over the discussion in #3

• [x] REMOVE - default.datadir and RAAD_DIR
• [x] establish branched file-db in each of data, data_local, across PUB, PRIV e.g. data/.raad_admin/filedb
• [x] apply detection logic of available file paths
• [x] set up cron jobs to create all branched data.*/.raad_admin/filedb.rds
• [x] trigger filedb creation for casual user
• [x] delete all datadir <- get_raad_datadir() usage
• [x] make all file listings provide only fullname (not file)
• [x] consider having file listings include root

[mdsumner]

Pretty sure RAAD_DIR was only used to set the environment variable in R CMD check, this code ran and relied on ~/.R/Rbuildenviron and Rcheckenviron

  ## try RAAD_DIR, which may only be available to R CMD check from ~/.R/check.Renviron
  #r <- getOption("repos")
  #dd <- getOption("default.datadir")

  #if (is.null(dd["default.datadir"])) {
  #  dd["default.datadir"] <- Sys.getenv("RAAD_DIR");
  #  options(repos = r, default.datadir = dd);
  #}

[mdsumner]

LOL, including .raad_admin/file_db.rds has the nice side effect of being recursive, and now there's never an empty data.*/ folder

[mdsumner]

Utilities are

• .possiblepaths() paths we know about, explicitly ending in '../data', '../data_local' etc.
• validate_input_paths(), validate_possible_paths checks on candidates and user input paths used by
• set_raad_data_roots() which puts any user-input and auto-found roots into a global variable accessible by get_raadf_data_roots()
• set_raad_filenames() scans the data roots for file cache, applies the actual root, and puts into global variable accessible by get_raad_filenames()
• run_this_function_to_build_raad_cache() applies all that and sets up the files and writes the file caches '.raad_admin/file_db.rds'

[mdsumner]

For setting up, we need to do the blueant / bb_sync thing and then

remotes::install_github("AustralianAntarcticDivision/raadfiles@data-roots")
remotes::install_github("AustralianAntarcticDivision/raadtools@raadfiles-data-root")

my_data_dir <- "C:/temp/data/...etc/"
raadfiles:::set_raad_data_roots(my_data_dir)
raadfiles:::run_this_function_to_build_raad_cache()
raadfiles:::set_raad_filenames()

[mdsumner]

Check all functions in raadfiles and their raadtools counterparts:

• [x] amps-files.R
• [ ] cfafiles.R
• [x] cmip5.R
• [x] ncep2-files.R
• [x] nsidc-files.R
• [x] raad-altimetry-files.R
• [x] raad-ice-amsr-files.R
• [x] raad-sst-files.R
• [x] raad-thelist-files.R
• [x] raad-world-ocean-atlas-files.R
• [x] rema-files.R
• [x] smith-sandwell.R
• [x] topography-files.R

[mdsumner]

Turn off bare directory listing in dir_ls

Done: set to type = c("file", "symlink")

[mdsumner]

Pretty sure this is now all good in data-roots branch. Will play around on rs2 and local as a user for a bit. All servers are still using master.

[mdsumner]

Now committed on bw and aes2, all changes in master in raadtools and raadfiles.