testunit fails: "runStaticServer() throws an error if the requested port is used --"

[nunotexbsd]

Any clues?

Thanks

> library(testthat)
> library(httpuv)
>
> test_check("httpuv")
Using libcurl 8.5.0 with OpenSSL/3.0.12
[ FAIL 1 | WARN 0 | SKIP 11 | PASS 253 ]

== Skipped tests (11) ==========================================================
* (8): 'test-traffic.R:65:3', 'test-traffic.R:82:3', 'test-traffic.R:109:3',
  'test-traffic.R:136:3', 'test-traffic.R:164:3', 'test-traffic.R:192:3',
  'test-traffic.R:221:3', 'test-traffic.R:249:3'
* On CRAN (2): 'test-frame-completion.R:8:3', 'test-staticServer.R:126:3'
* Skipping non-ASCII path tests on UTF-8 Unix system (1):
  'test-static-paths.R:770:3'

== Failed tests ================================================================
-- Failure ('test-staticServer.R:84:3'): runStaticServer() throws an error if the requested port is used --
`runStaticServer(...)` did not throw an error.

[ FAIL 1 | WARN 0 | SKIP 11 | PASS 253 ]
Error: Test failures
Execution halted

[gadenbuie]

Hi @nunotexbsd, can you provide the output from devtools::session_info() please? That will help give context into your system environment.

[gadenbuie]

One more thing to try, can you run this code and copy and paste the output here? Thanks!

vapply(setNames(nm = httpuv:::unsafe_ports), httpuv:::is_port_available, logical(1))

An easy way to do that is to use the copy code button in the block above, and then run reprex::reprex() in an R console. reprex() will run the copied R code and put the results on the clipboard (and will also open the result for a preview).

[nunotexbsd]

Hello!

We don't have 'reprex' ported yet, but for what I've read it looks very interesting and I will work on that today :)

> devtools::session_info()
─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.3.2 (2023-10-31)
 os       FreeBSD leg.home 15.0-CURRENT FreeBSD 15.0-CURRENT #62 main-n267800-001c48b4139f: Sat Jan 27 19:46:25 WET 2024     root@leg.home:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG amd64
 system   amd64, freebsd15.0
 ui       X11
 language (EN)
 collate  C.UTF-8
 ctype    C.UTF-8
 tz       Europe/Lisbon
 date     2024-01-31
 pandoc   NA

─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 package     * version date (UTC) lib source
 brio          1.1.3   2021-11-30 [1] CRAN (R 4.3.2)
 cachem        1.0.8   2023-05-01 [1] CRAN (R 4.3.2)
 callr         3.7.3   2022-11-02 [1] CRAN (R 4.3.2)
 cli           3.6.2   2023-12-11 [1] CRAN (R 4.3.2)
 crayon        1.5.2   2022-09-29 [1] CRAN (R 4.3.2)
 desc          1.4.2   2022-09-08 [1] CRAN (R 4.3.2)
 devtools    * 2.4.2   2021-06-07 [1] CRAN (R 4.3.2)
 ellipsis      0.3.2   2021-04-29 [1] CRAN (R 4.3.2)
 fastmap       1.1.1   2023-02-24 [1] CRAN (R 4.3.2)
 fs            1.6.3   2023-07-20 [1] CRAN (R 4.3.2)
 glue          1.7.0   2024-01-09 [1] CRAN (R 4.3.2)
 httpuv      * 1.6.14  2024-01-26 [1] CRAN (R 4.3.2)
 later         1.3.1   2023-05-02 [1] CRAN (R 4.3.2)
 lifecycle     1.0.4   2023-11-07 [1] CRAN (R 4.3.2)
 magrittr      2.0.3   2022-03-30 [1] CRAN (R 4.3.2)
 memoise       2.0.1   2021-11-26 [1] CRAN (R 4.3.2)
 pkgbuild      1.4.2   2023-06-26 [1] CRAN (R 4.3.2)
 pkgload       1.3.2.1 2023-07-08 [1] CRAN (R 4.3.2)
 prettyunits   1.2.0   2023-09-24 [1] CRAN (R 4.3.2)
 processx      3.8.2   2023-06-30 [1] CRAN (R 4.3.2)
 promises      1.2.1   2023-08-10 [1] CRAN (R 4.3.2)
 ps            1.7.5   2023-04-18 [1] CRAN (R 4.3.2)
 purrr         1.0.2   2023-08-10 [1] CRAN (R 4.3.2)
 R6            2.5.1   2021-08-19 [1] CRAN (R 4.3.2)
 Rcpp          1.0.12  2024-01-09 [1] CRAN (R 4.3.2)
 remotes       2.4.2.1 2023-07-18 [1] CRAN (R 4.3.2)
 rlang         1.1.3   2024-01-10 [1] CRAN (R 4.3.2)
 rprojroot     2.0.4   2023-11-05 [1] CRAN (R 4.3.2)
 sessioninfo   1.2.2   2021-12-06 [1] CRAN (R 4.3.2)
 testthat    * 3.1.10  2023-07-06 [1] CRAN (R 4.3.2)
 usethis     * 2.0.1   2021-02-10 [1] CRAN (R 4.3.2)
 vctrs         0.6.5   2023-12-01 [1] CRAN (R 4.3.2)
 withr         3.0.0   2024-01-16 [1] CRAN (R 4.3.2)

 [1] /usr/local/lib/R/library

> vapply(setNames(nm = httpuv:::unsafe_ports), httpuv:::is_port_available, logical(1))
    1     7     9    11    13    15    17    19    20    21    22    23    25
FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
   37    42    43    53    77    79    87    95   101   102   103   104   109
FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
  110   111   113   115   117   119   123   135   139   143   179   389   427
FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
  465   512   513   514   515   526   530   531   532   540   548   556   563
FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
  587   601   636   993   995  2049  3659  4045  6000  6665  6666  6667  6668
FALSE FALSE FALSE FALSE FALSE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE
 6669  6697
 TRUE  TRUE

[gadenbuie]

🤔 that's interesting, thank you. I'll give a little background. The failing test finds an unavailable port and then tries to start an httpuv server that uses that port, expecting it to error.

https://github.com/rstudio/httpuv/blob/b6f96e50e744a1a02b7a88658d3e975b171ae007/tests/testthat/test-staticServer.R#L76-L86

httpuv has an internal list of known unsafe ports that are typically used by other services, and on your system there are many such ports: 1, 7, 9, 11, 13, etc.

What happens when you run this code?

library(httpuv)

runStaticServer(
  system.file("example-static-site", package = "httpuv"),
  port = 1
)
#> Error in runStaticServer(system.file("example-static-site", package = "httpuv"), : Port 1 is not available on host 127.0.0.1

[nunotexbsd]

I've tested 1, 80 and 8080 ports:

> library(httpuv)

runStaticServer(
  system.file("example-static-site", package = "httpuv"),
  port = 1
)
Error in runStaticServer(system.file("example-static-site", package = "httpuv"),  :
  Port 1 is not available on host 127.0.0.1

runStaticServer(
  system.file("example-static-site", package = "httpuv"),
  port = 80
)
Error in runStaticServer(system.file("example-static-site", package = "httpuv"),  :
  Port 80 is not available on host 127.0.0.1

runStaticServer(
  system.file("example-static-site", package = "httpuv"),
  port = 8080
)
Serving: '/usr/local/lib/R/library/httpuv/example-static-site'
View at: http://127.0.0.1:8080
Error in utils::browseURL(paste0("http://", host, ":", port)) :
  'browser' must be a non-empty character string

[gadenbuie]

Thanks! The error for port 1 is expected and is the code the failing test in the original post should be testing. (Neither port 80 nor 8080 would be checked in that test.) The error when starting a server on port 8080 is related to R on your system not knowing how to open a browser. You can suppress that behavior by setting runStaticServer(..., browse = FALSE).

[nunotexbsd]

The error for port 1 is expected and is the code the failing test in the original post should be testing.

In that case shouldn't test be successfull? In other words, if test is supposed to fail on given port, then it is a test success.

[gadenbuie]

In that case shouldn't test be successfull?

That's right. Which means that I'm not sure why the test is failing when you're running the test suite and unfortunately I'm out of clues. Do you have any ideas?

[nunotexbsd]

expect_error( 
   runStaticServer(path_example_site(), port = find_unsafe_port(), background = TRUE) 
 )

expect_error should show what port was but it mentions port "--"

runStaticServer() throws an error if the requested port is used --

Maybe test did not pick a correct port?

[gadenbuie]

That was also my thought. If find_unsafe_port() doesn't actually find an unsafe port, it returns NULL and the test doesn't catch that case. I can update the test to skip if we can't find an unsafe port, but it's surprising that it worked for you interactively.

[nunotexbsd]

Found it what could be the reason:

Local shell: test OK

vapply(setNames(nm = httpuv:::unsafe_ports), httpuv:::is_port_available, logical(1))
    1     7     9    11    13    15    17    19    20    21    22    23    25
FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
   37    42    43    53    77    79    87    95   101   102   103   104   109
FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
  110   111   113   115   117   119   123   135   139   143   179   389   427
FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
  465   512   513   514   515   526   530   531   532   540   548   556   563
FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
  587   601   636   993   995  2049  3659  4045  6000  6665  6666  6667  6668
FALSE FALSE FALSE FALSE FALSE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE
 6669  6697
 TRUE  TRUE

Jail (poudriere): test NOK

vapply(setNames(nm = httpuv:::unsafe_ports), httpuv:::is_port_available, logical(1))
   1    7    9   11   13   15   17   19   20   21   22   23   25   37   42   43
TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE
  53   77   79   87   95  101  102  103  104  109  110  111  113  115  117  119
TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE
 123  135  139  143  179  389  427  465  512  513  514  515  526  530  531  532
TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE
 540  548  556  563  587  601  636  993  995 2049 3659 4045 6000 6665 6666 6667
TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE
6668 6669 6697
TRUE TRUE TRUE

That's why no port is given ("--").

runStaticServer(
  system.file("example-static-site", package = "httpuv"),
  port = 1
)
Serving: '/usr/local/lib/R/library/httpuv/example-static-site'
View at: http://127.0.0.1:1

Inside freebsd jail, all ports are available and safe to use... Now I really don't know what to think :)

[gadenbuie]

Mystery solved! I'll update the test. In the mean time, are you blocked by this issue in some way?

[nunotexbsd]

Hello!

https://github.com/rstudio/httpuv/pull/396 fixes freebsd jail environment and test is success.

I'm not blocked by this issue and it could be closed. I let it to you to close it now or when pull is merged.

Sorry for misleading you at first as it was my fault post wrong port status on jail and local system.

Thanks!