make is successful but crashes shortly after for dynamic targets

[saadaslam]

I can't necessarily reproduce the issue with ease, and this is for a project I can't share, but I'm running into this error while running make:

Error in FUN(X[[i]], ...) : size > 0L is not TRUE

This seems to occur after make finishes compiling a dynamic target. Afterwards, the error above occurs.

I've experimented with this a couple times and have found that:

• the targets are successfully created and I can read them in w/ loadd or readd
• if I run make again, subsequent dependent targets are kicked off and in the process of being created, although the same error above occurs for dynamic targets
• if i keep repeating make, eventually, I only get the error described above.

Unfortunately, the error persists after numerous attempts running clean(), clean(destroy=TRUE), and even re-installing drake. I have also attempted to force targets to be lists via target(..., subtarget_list = TRUE)

Here's the traceback()

27: stop(simpleError(msg, call = if (p <- sys.parent(1L)) sys.call(p)))
26: stopifnot(size > 0L)
25: FUN(X[[i]], ...)
24: lapply(vars, get_dynamic_size, config = config)
23: unlist(lapply(vars, get_dynamic_size, config = config))
22: subtarget_deps.cross(dynamic, parent, index, config)
21: subtarget_deps(dynamic, parent, index, config)
20: register_dynamic_subdeps(dynamic, spec, index, parent, config)
19: FUN(X[[i]], ...)
18: lapply(seq_along(subtargets), register_subtarget_spec, parent = target, 
        subtargets = subtargets, spec = spec, dynamic = dynamic, 
        config = config)
17: register_in_spec(target, subtargets_all, config)
16: register_subtargets(target, static_ok, dynamic_ok, config)
15: handle_triggers_impl.dynamic_unregistered(target, meta, config)
14: handle_triggers_impl(target, meta, config)
13: handle_triggers(target, meta, config)
12: local_build(target = targets[1], config = config, downstream = targets[-1])
11: loop_check(config)
10: drake_backend_loop(config)
9: drake_backend(config)
8: run_backend(config)
7: process_targets(config)
6: make_impl(config)
5: make(plan, max_expand = 2) at make.R#26
4: eval(ei, envir)
3: eval(ei, envir)
2: withVisible(eval(ei, envir))
1: source("~/comp_model/make.R", echo = TRUE)

And here's the sessionInfo()

R version 3.6.0 (2019-04-26)
Platform: x86_64-conda_cos6-linux-gnu (64-bit)
Running under: SUSE Linux Enterprise Server 12 SP5

Matrix products: default
BLAS:   /export/uddn01_SAS_gpfs/Rtools_prod/CI000000019309925/tools_v1/R-3.6.0/lib/R/lib/libRblas.so
LAPACK: /export/uddn01_SAS_gpfs/Rtools_prod/CI000000019309925/tools_v1/R-3.6.0/lib/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] rsample_0.0.7     broom_0.7.0       tictoc_1.0        furrr_0.1.0       future_1.16.0     mgcv_1.8-31       nlme_3.1-139      stringr_1.4.0     tidyr_1.1.0      
[10] purrr_0.3.4       dplyr_1.0.0       mypackage2_2.5    tibble_3.0.2      drake_7.12.4      data.table_1.12.8

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.3            lubridate_1.7.9       here_0.1              aws.ec2metadata_0.2.0 txtq_0.2.3            lattice_0.20-38       listenv_0.8.0        
 [8] prettyunits_1.0.2     rprojroot_1.3-2       assertthat_0.2.1      digest_0.6.23         packrat_0.5.0         aws.signature_0.5.2   R6_2.4.0             
[15] odbc_1.2.2            backports_1.1.4       httr_1.4.1            pillar_1.4.4          rlang_0.4.6           progress_1.2.0        curl_3.3             
[22] rstudioapi_0.11       blob_1.2.1            Matrix_1.2-17         qs_0.22.1             splines_3.6.0         igraph_1.2.4.1        bit_1.1-15.1         
[29] janitor_1.2.0         compiler_3.6.0        pkgconfig_2.0.2       base64enc_0.1-3       globals_0.12.5        tidyselect_1.1.0      codetools_0.2-16     
[36] fansi_0.4.0           crayon_1.3.4          grid_3.6.0            jsonlite_1.7.0        lifecycle_0.2.0       DBI_1.1.0             magrittr_1.5         
[43] storr_1.2.1           RJDBC_0.2-7.1         cli_2.0.2             stringi_1.4.3         xml2_1.3.2            ellipsis_0.3.0        filelock_1.0.2       
[50] generics_0.0.2        vctrs_0.3.1           stringfish_0.13.3     fst_0.9.2             RApiSerialize_0.1.0   forcats_0.5.0         tools_3.6.0          
[57] bit64_0.9-7           glue_1.4.1            hms_0.5.3             aws.s3_0.3.12.1       parallel_3.6.0        PKI_0.1-5.1           ROracle_1.3-1        
[64] base64url_1.4         rJava_0.9-11 

[saadaslam]

After some more debugging, it turns out that this is due to problematic subsequent dynamic targets where one of the inputs is an empty list and you're using cross

So basically, here's a reprex:

library(drake)

the_plan <- drake_plan(
  cars_split = split(mtcars, mtcars$cyl),
  model_1 = target(
    lm(mpg ~ wt, data = cars_split[[1]]),
    dynamic = map(cars_split)
  ),
  link_f = c("logit", "probit"),
  ## cars_split_400 will be an empty list
  cars_split_400 = Filter(function(x) x$hp >= 400, cars_split),
  model_2 = target(
    glm(am ~ wt, data = cars_split_400, family = binomial(link = link_f)),
    dynamic = cross(link_f, cars_split_400)
  )
)

make(the_plan)
#> ▶ target link_f
#> ▶ target cars_split
#> ▶ target cars_split_400
#> ▶ dynamic model_1
#> > subtarget model_1_e80834f2
#> > subtarget model_1_13ddb898
#> > subtarget model_1_b975270d
#> ■ finalize model_1
#> Error in FUN(X[[i]], ...): size > 0L is not TRUE

^{Created on 2020-08-11 by the reprex package (v0.3.0)}

[saadaslam]

I am closing this because it's not a bug, although a more useful error message would be helpful. For example, telling me that the error came from drake and not some other procedure in my program.

[wlandau]

An improved error message sounds reasonable. Just added one in 00183e118b3efa29e327e7241f797c14aa423744.