Error: protect(): protection stack overflow

etiennebacher commented 1 year ago

Hello, I've been following a bit your repos that use extendr since I'm trying to learn it. I ran some of your examples with more points to see if the performance gains scale well and I came across this error in h3_from_points():

library(h3o)

# works fine with 10,000 points
pnts <- tibble::tibble(
  x = runif(10000, -5, 10),
  y = runif(10000, 40, 50)
) |> 
  sf::st_as_sf(
    coords = c("x", "y"), 
    crs = 4326
  )

foo <- h3_from_points(pnts$geometry, 4)

# error with 100,000 points
pnts <- tibble::tibble(
  x = runif(100000, -5, 10),
  y = runif(100000, 40, 50)
) |> 
  sf::st_as_sf(
    coords = c("x", "y"), 
    crs = 4326
  )

foo <- h3_from_points(pnts$geometry, 4)
#> Error: protect(): protection stack overflow

I know it's still early in development and I'm not using these geospatial features anyway, but I just thought you'd want to know.

Session info if needed

─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.2.2 (2022-10-31 ucrt)
 os       Windows 10 x64 (build 19044)
 system   x86_64, mingw32
 ui       RStudio
 language (EN)
 collate  English_Europe.utf8
 ctype    English_Europe.utf8
 tz       Europe/Paris
 date     2023-03-13
 rstudio  2022.12.0+353 Elsbeth Geranium (desktop)
 pandoc   3.1 @ C:/Users/etienne/AppData/Local/Pandoc/ (via rmarkdown)

─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 package     * version    date (UTC) lib source
 bench         1.1.2      2021-11-30 [1] CRAN (R 4.2.0)
 callr         3.7.3      2022-11-02 [1] CRAN (R 4.2.2)
 class         7.3-21     2023-01-23 [1] CRAN (R 4.2.2)
 classInt      0.4-9      2023-02-28 [1] CRAN (R 4.2.2)
 cli           3.6.0      2023-01-09 [1] CRAN (R 4.2.2)
 clipr         0.8.0      2022-02-22 [1] CRAN (R 4.2.0)
 curl          5.0.0      2023-01-12 [1] CRAN (R 4.2.2)
 DBI           1.1.3      2022-06-18 [1] CRAN (R 4.2.1)
 digest        0.6.31     2022-12-11 [1] CRAN (R 4.2.2)
 dplyr         1.1.0      2023-01-29 [1] CRAN (R 4.2.2)
 e1071         1.7-13     2023-02-01 [1] CRAN (R 4.2.2)
 evaluate      0.20       2023-01-17 [1] CRAN (R 4.2.2)
 fansi         1.0.4      2023-01-22 [1] CRAN (R 4.2.2)
 fastmap       1.1.1      2023-02-24 [1] CRAN (R 4.2.2)
 fs            1.6.0      2023-01-23 [1] CRAN (R 4.2.2)
 generics      0.1.3      2022-07-05 [1] CRAN (R 4.2.1)
 geojsonsf     2.0.3      2022-05-30 [1] CRAN (R 4.2.1)
 glue          1.6.2      2022-02-24 [1] CRAN (R 4.2.0)
 h3jsr         1.3.1      2023-01-21 [1] CRAN (R 4.2.2)
 h3o         * 0.1.0      2023-03-13 [1] Github (JosiahParry/h3o@e50144c)
 htmltools     0.5.4      2022-12-07 [1] CRAN (R 4.2.2)
 jsonlite      1.8.4      2022-12-06 [1] CRAN (R 4.2.2)
 KernSmooth    2.23-20    2021-05-03 [2] CRAN (R 4.2.2)
 knitr         1.42       2023-01-25 [1] CRAN (R 4.2.2)
 lifecycle     1.0.3      2022-10-07 [1] CRAN (R 4.2.1)
 magrittr      2.0.3      2022-03-30 [1] CRAN (R 4.2.0)
 pillar        1.8.1      2022-08-19 [1] CRAN (R 4.2.1)
 pkgconfig     2.0.3      2019-09-22 [1] CRAN (R 4.2.0)
 processx      3.8.0      2022-10-26 [1] CRAN (R 4.2.2)
 profmem       0.6.0      2020-12-13 [1] CRAN (R 4.2.0)
 proxy         0.4-27     2022-06-09 [1] CRAN (R 4.2.0)
 ps            1.7.2      2022-10-26 [1] CRAN (R 4.2.2)
 purrr         1.0.1      2023-01-10 [1] CRAN (R 4.2.2)
 R.cache       0.16.0     2022-07-21 [1] CRAN (R 4.2.1)
 R.methodsS3   1.8.2      2022-06-13 [1] CRAN (R 4.2.0)
 R.oo          1.25.0     2022-06-12 [1] CRAN (R 4.2.0)
 R.utils       2.12.2     2022-11-11 [1] CRAN (R 4.2.1)
 R6            2.5.1      2021-08-19 [1] CRAN (R 4.2.0)
 rappdirs      0.3.3      2021-01-31 [1] CRAN (R 4.2.0)
 Rcpp          1.0.10     2023-01-22 [1] CRAN (R 4.2.2)
 reprex        2.0.2      2022-08-17 [1] CRAN (R 4.2.1)
 rlang         1.0.6.9000 2023-02-20 [1] Github (r-lib/rlang@394204f)
 rmarkdown     2.20       2023-01-19 [1] CRAN (R 4.2.2)
 roxygen2      7.2.3.9000 2023-01-04 [1] Github (r-lib/roxygen2@39d2dbe)
 rstudioapi    0.14       2022-08-22 [1] CRAN (R 4.2.1)
 sessioninfo   1.2.2      2021-12-06 [1] CRAN (R 4.2.0)
 sf            1.0-9      2022-11-08 [1] CRAN (R 4.2.2)
 shrtcts       0.1.2      2022-04-29 [1] Github (gadenbuie/shrtcts@12ef67f)
 stringi       1.7.12     2023-01-11 [1] CRAN (R 4.2.2)
 stringr       1.5.0      2022-12-02 [1] CRAN (R 4.2.2)
 styler        1.9.1      2023-03-04 [1] CRAN (R 4.2.2)
 tibble        3.1.8      2022-07-22 [1] CRAN (R 4.2.1)
 tidyr         1.3.0      2023-01-24 [1] CRAN (R 4.2.2)
 tidyselect    1.2.0      2022-10-10 [1] CRAN (R 4.2.1)
 units         0.8-1      2022-12-10 [1] CRAN (R 4.2.2)
 utf8          1.2.3      2023-01-31 [1] CRAN (R 4.2.2)
 V8            4.2.2      2022-11-03 [1] CRAN (R 4.2.2)
 vctrs         0.5.2.9000 2023-02-20 [1] Github (r-lib/vctrs@303b5dd)
 withr         2.5.0      2022-03-03 [1] CRAN (R 4.2.0)
 xfun          0.37       2023-01-31 [1] CRAN (R 4.2.2)
 xml2          1.3.3      2021-11-30 [1] CRAN (R 4.2.0)
 yaml          2.3.7      2023-01-23 [1] CRAN (R 4.2.2)
 [1] C:/Users/etienne/AppData/Local/R/win-library/4.2
 [2] C:/R/library

JosiahParry commented 1 year ago

Thanks @etiennebacher. I did know of this limitation but not in this context. @CGMossa thinks its related to https://github.com/extendr/extendr/issues/397.

Here's something interesting. The number 50000 has something to do with it.

library(h3o)

# its exactly 50000 that causes an issue
n <- 49999
pnts <- tibble::tibble(
  x = runif(n, -5, 10),
  y = runif(n, 40, 50)
) |> 
  sf::st_as_sf(
    coords = c("x", "y"), 
    crs = 4326
  )

foo <- h3_from_points(pnts$geometry, 4)
#> Error: protect(): protection stack overflow

# set to 50000
n <- 50000
pnts <- tibble::tibble(
  x = runif(n, -5, 10),
  y = runif(n, 40, 50)
) |> 
  sf::st_as_sf(
    coords = c("x", "y"), 
    crs = 4326
  )

# watch error
foo <- h3_from_points(pnts$geometry, 4)
#> Error: protect(): protection stack overflow

https://github.com/extendr/extendr/blob/6e15390d2d4dbcd53878f4b31312c4fb68de25e6/extendr-api/src/ownership.rs#L34

50000 * 2 = 10000. 50000 robjs (Doubles) coming in and 50000 robjs (externalptrs) going out its exactly 10000. Can we turn that number up to 11?

etiennebacher commented 1 year ago

The number 50000 has something to do with it.

I don't understand why, the example also fails with 49,999

JosiahParry commented 1 year ago

The number can change by ± 5 or so I've noticed. But pretty reliably around that limit. But only for lists I've found.

JosiahParry commented 1 year ago

simple R package designed to illustrate the issue https://github.com/JosiahParry/stackoverflow/blob/main/example.R

rstub commented 1 year ago

Using code from this package I have boiled this down to:

code <- 'fn stack_overflow_list(x: List) -> List {
    x
        .into_iter()
        .map(|(_, x)| {
            let dbs = Doubles::from_robj(&x).unwrap();
            [dbs[1].0, dbs[0].0]
        })
        .collect::<List>()
}'
rextendr::rust_function(code)

n <- 49984
pnts <- lapply(1:n, \(x) runif(2))
stack_overflow_list(pnts)

n <- 49993
pnts <- lapply(1:n, \(x) runif(2))
stack_overflow_list(pnts)

Unfortunately, reprex does not render this for me.
Interestingly, the threshold for the failure seems to be even lower.
To me this looks like a bug in rextendr. Has this been reported there?

CGMossa commented 1 year ago

Hello @rstub. This is a bug in extendR. @JosiahParry have helped extensively with troubleshooting it. I wrote a PR for it here https://github.com/extendr/extendr/pull/540, and it got reviewed and approved.. I just need to write one more test to merge it. This will necessitate the use of extendr on github, in order to avoid this. I would say that you can circumvent this, if it is truly necessary. Thanks for investigating it as well!

JosiahParry commented 1 year ago

Thanks both! Yeah, this bothered me for quite some time. For now, the only way I am aware of to avoid it is to pre-allocate the List and assign elements in a for loop using the .elt() method.

On Mon, May 8, 2023 at 04:17 CGMossa @.***> wrote:

Hello @rstub https://github.com/rstub. This is a bug in extendR. @JosiahParry https://github.com/JosiahParry have helped extensively with troubleshooting it. I wrote a PR for it here extendr/extendr#540 https://github.com/extendr/extendr/pull/540, and it got reviewed and approved.. I just need to write one more test to merge it. This will necessitate the use of extendr on github, in order to avoid this. I would say that you can circumvent this, if it is truly necessary. Thanks for investigating it as well!

— Reply to this email directly, view it on GitHub https://github.com/JosiahParry/h3o/issues/3#issuecomment-1537947830, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADHIKLGSAEMSZDPM63MSDMTXFCTZPANCNFSM6AAAAAAVY5D5ZE . You are receiving this because you were mentioned.Message ID: @.***>

JosiahParry commented 1 year ago

@etiennebacher can you install the new version from https://josiahparry.r-universe.dev/h3o and let me know if this is fixed?

etiennebacher commented 1 year ago

Hi @JosiahParry, I don't have this error anymore, thanks

JosiahParry commented 1 year ago

@etiennebacher brilliant :) thanks

JosiahParry / h3o

Error: protect(): protection stack overflow #3