search

kyle-messier / targets_test_update

0 stars 0 forks source link

tar_config_set test #1

Open kyle-messier opened 2 months ago

kyle-messier commented 2 months ago

@sigmafelix @mitchellmanware

If you could test a targets task for me and let me know. It is related to this. Basically, I’m hoping this is an approach for us to use a common location on the ddn to manage the _targets/ folder, its objects, and metadata.

I’ve pushed some updates and new files. Here is the update and what I need to have tested:

  1. Clone or pull this repo
  2. I've added the following piece to the _targets.R file: 
    tar_config_set(
store = "/ddn/gs1/group/set/targets_test/"
)
  3. This creates a new _targets.yaml (rejoice yaml 🚀 )
  4. The idea of config set is that it allows an external location for all of the _targets/
  5. Once you have this pulled/cloned, then run tar_make(). If all works properly, it should skip the entire pipeline since it should be up to date.

I'm sure there will be some kinks or gotchas we may need to work through. For this example, I have the data.csv in my local path. We'll see if that matters, but that would also be resolved with our common data storage. What is nice, as far as I can tell, is that the _targets.R is local as it should be. This is where we want individuals to code and work on new updates and push according to our protocols.

If the config_set works, then for projects that have multiple contributors like amadeus and beethoven then, I think we'll have to also use gittargets. This targets extension would ensure that when someone pushes a modification on a branch, that it doesn't invalidate the main branch, etc. Basically, you can see how that running a dev version could impact the target objects. We'll get to this later, but in the meantime let me know how the first test goes. Thank you!

mitchellmanware commented 2 months ago

Cloning and running while connected to ddn finds the correct path but does not find the the meta/meta file.

library(targets); library(tarchetypes)
> setwd("/Volumes/manwareme/explore/R/targets/targets_test_update/")
> tar_make()
Warning message:
In file(file, ifelse(append, "a", "w")) :
  cannot open file '/ddn/gs1/group/set/targets_test//meta/meta': No such file or directory
Error:
! Error running targets::tar_make()
Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
Debugging guide: https://books.ropensci.org/targets/debugging.html
How to ask for help: https://books.ropensci.org/targets/help.html
Last error message:
    cannot open the connection
Last error traceback:
    tryCatch(withCallingHandlers({ NULL saveRDS(do.call(do.call, c(readRDS("...
    tryCatchList(expr, classes, parentenv, handlers)
    tryCatchOne(tryCatchList(expr, names[-nh], parentenv, handlers[-nh]), na...
    doTryCatch(return(expr), name, parentenv, handler)
    tryCatchList(expr, names[-nh], parentenv, handlers[-nh])
    tryCatchOne(expr, names, parentenv, handlers[[1L]])
    doTryCatch(return(expr), name, parentenv, handler)
    withCallingHandlers({ NULL saveRDS(do.call(do.call, c(readRDS("/var/fold...
    saveRDS(do.call(do.call, c(readRDS("/var/folders/bp/ymbcsbsx7gscrg24fxng...
    do.call(do.call, c(readRDS("/var/folders/bp/ymbcsbsx7gscrg24fxngxc840000...
    (function (what, args, quote = FALSE, envir = parent.frame()) { if (!is....
    (function (targets_function, targets_arguments, options, envir = NULL, s...
    tryCatch(out <- withCallingHandlers(targets::tar_callr_inner_try(targets...
    tryCatchList(expr, classes, parentenv, handlers)
    tryCatchOne(expr, names, parentenv, handlers[[1L]])
    doTryCatch(return(expr), name, parentenv, handler)
    withCallingHandlers(targets::tar_callr_inner_try(targets_function = targ...
    targets::tar_callr_inner_try(targets_function = targets_function, target...
    do.call(targets_function, targets_arguments)
    (function (pipeline, path_store, names_quosure, shortcut, reporter, seco...
    local_init(pipeline = pipeline, meta = meta_init(path_store = path_store...
    self$start()
    self$ensure_meta()
    self$meta$database$preprocess(write = TRUE)
    self$ensure_storage()
    self$reset_storage()
    write(self$produce_line(self$header), self$path)
    cat(x, file = file, sep = c(rep.int(sep, ncolumns - 1), "\n"), append = ...
    file(file, ifelse(append, "a", "w"))
> traceback()
9: stop(fallback)
8: signal_abort(cnd, .file)
7: rlang::abort(message = message, class = class, call = tar_empty_envir)
6: tar_error(message = paste0(...), class = base::union(custom_error_classes(class), 
       c("tar_condition_run", "tar_condition_targets")))
5: tar_throw_run(message, class = class(traced_condition$condition))
4: callr_error(traced_condition = out, fun = fun)
3: if_any(inherits(out, "tar_condition_traced"), callr_error(traced_condition = out, 
       fun = fun), out)
2: callr_outer(targets_function = tar_make_inner, targets_arguments = targets_arguments, 
       callr_function = callr_function, callr_arguments = callr_arguments, 
       envir = envir, script = script, store = store, fun = "tar_make")
1: tar_make()

I can do more debugging on Monday, but I hope this helps. @kyle-messier

kyle-messier commented 2 months ago

@mitchellmanware I think this is because you are connected via Wine and not ssh. Sorry - I should clarify. You could try to change the path in config_set and see if that works. Otherwise, @sigmafelix will need to give you the tutorial he gave me on getting an remote ssh link to the ddn (with a /ddn/gs1/ ... path) instead of the usual wine (/Volumes/set/...) in vscode, if you are not familiar already. All good - thanks!

mitchellmanware commented 2 months ago

Yes - I will need that demonstration. I have tried a few different ways but have not been able to get it to work.

On Fri, May 3, 2024 at 4:46 PM {SET}group @.***> wrote:

@mitchellmanware https://github.com/mitchellmanware I think this is because you are connected via Wine and not ssh. Sorry - I should clarify. You could try to change the path in config_set and see if that works. Otherwise, @sigmafelix https://github.com/sigmafelix will need to give you the tutorial he gave me on getting an remote ssh link to the ddn (with a /ddn/gs1/ ... path) instead of the usual wine (/Volumes/set/...) in vscode, if you are not familiar already. All good - thanks!

— Reply to this email directly, view it on GitHub https://github.com/kyle-messier/targets_test_update/issues/1#issuecomment-2093739046, or unsubscribe https://github.com/notifications/unsubscribe-auth/BB2PI7ALNSB7Y3DTPU3B7ILZAPZR7AVCNFSM6AAAAABHGDBT4SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJTG4ZTSMBUGY . You are receiving this because you were mentioned.Message ID: @.***>

sigmafelix commented 2 months ago

@kyle-messier On triton, I find an error on writing permission:

[songi2@cn040603/triton targets_test_update]$ Rscript run.R
Error:
! Error running targets::tar_make()
Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
Debugging guide: https://books.ropensci.org/targets/debugging.html
How to ask for help: https://books.ropensci.org/targets/help.html
Last error message:
    Permission denied: '/ddn/gs1/group/set/targets_test//meta/tar_temp_1187bd33ef678c'. Unable to create new file for writing (it does not exist already). Do you have permission to write here, is there space on the disk and does the path exist?
Last error traceback:
    base::tryCatch(base::withCallingHandlers({ NULL base::saveRDS(base::do.c...
    tryCatchList(expr, classes, parentenv, handlers)
    tryCatchOne(tryCatchList(expr, names[-nh], parentenv, handlers[-nh]), na...
    doTryCatch(return(expr), name, parentenv, handler)
    tryCatchList(expr, names[-nh], parentenv, handlers[-nh])
    tryCatchOne(expr, names, parentenv, handlers[[1L]])
    doTryCatch(return(expr), name, parentenv, handler)
    base::withCallingHandlers({ NULL base::saveRDS(base::do.call(base::do.ca...
    base::saveRDS(base::do.call(base::do.call, base::c(base::readRDS("/tmp/R...
    base::do.call(base::do.call, base::c(base::readRDS("/tmp/RtmpWR7dtU/call...
    (function (what, args, quote = FALSE, envir = parent.frame()) { if (!is....
    (function (targets_function, targets_arguments, options, envir = NULL, s...
    tryCatch(out <- withCallingHandlers(targets::tar_callr_inner_try(targets...
    tryCatchList(expr, classes, parentenv, handlers)
    tryCatchOne(expr, names, parentenv, handlers[[1L]])
    doTryCatch(return(expr), name, parentenv, handler)
    withCallingHandlers(targets::tar_callr_inner_try(targets_function = targ...
    targets::tar_callr_inner_try(targets_function = targets_function, target...
    do.call(targets_function, targets_arguments)
    (function (pipeline, path_store, names_quosure, shortcut, reporter, seco...
    local_init(pipeline = pipeline, meta = meta_init(path_store = path_store...
    self$start()
    self$ensure_meta()
    self$meta$database$preprocess(write = TRUE)
    self$overwrite_storage(data)
    data.table::fwrite(x = data, file = tmp, sep = database_sep_outer, sep2 ..
kyle-messier commented 2 months ago

@sigmafelix That looks like network drive permissions. I have some info from Frank to help fix the defaults. Nonetheless, I changed group user to read, write, execute, so hopefully it resolves that for you. Thanks!

sigmafelix commented 2 months ago

@kyle-messier Thank you for fixing the permission problem. I think it works as expected since all targets were skipped.

[songi2@cn040603/triton targets_test_update]$ Rscript run.R
✔ skipped target my_files
✔ skipped branch read_data_0dece19324a7b29b
✔ skipped branch read_data_899156f6cc6c1ead
✔ skipped branch read_data_3fa004a9f8de4408
✔ skipped pattern read_data
✔ skipped branch data_format_4df430a1c7128d8e
✔ skipped branch data_format_ea796da70e6abcb1
✔ skipped branch data_format_cc64440bf9b55a74
✔ skipped pattern data_format
✔ skipped branch model_8d208b1f95873784
✔ skipped branch model_213768022d3927af
✔ skipped branch model_fe86297a009d9fed
✔ skipped pattern model
✔ skipped branch model_coefficients_540ed59ad0a3ebb4
✔ skipped branch model_coefficients_f35f072be5f3d3b7
✔ skipped branch model_coefficients_0778655c3a30fd38
✔ skipped pattern model_coefficients
✔ skipped target coefficients_table
✔ skipped pipeline [0.114 seconds]
kyle-messier commented 2 months ago

@sigmafelix Excellent! That was the outcome I was hoping for.

mitchellmanware commented 2 months ago

targets are also being skipped for me when running interactively via fermi.

[manwareme@gn040801/fermi targets_test_update]$ Rscript run.R
✔ skipped target my_files
✔ skipped branch read_data_0dece19324a7b29b
✔ skipped branch read_data_899156f6cc6c1ead
✔ skipped branch read_data_3fa004a9f8de4408
✔ skipped pattern read_data
✔ skipped branch data_format_4df430a1c7128d8e
✔ skipped branch data_format_ea796da70e6abcb1
✔ skipped branch data_format_cc64440bf9b55a74
✔ skipped pattern data_format
✔ skipped branch model_8d208b1f95873784
✔ skipped branch model_213768022d3927af
✔ skipped branch model_fe86297a009d9fed
✔ skipped pattern model
✔ skipped branch model_coefficients_540ed59ad0a3ebb4
✔ skipped branch model_coefficients_f35f072be5f3d3b7
✔ skipped branch model_coefficients_0778655c3a30fd38
✔ skipped pattern model_coefficients
✔ skipped target coefficients_table
✔ skipped pipeline [0.134 seconds]