Monkey test check does not return consistent result #9

Open DivadNojnarg opened 2 years ago

Starting basic test with CrrryProc during CI/CD on GitLab runner:

Running /usr/bin/google-chrome --no-first-run --headless \
  '--user-data-dir=/gitlab-runner/.local/share/r-crrri/chrome-data-dir-nruxnqvk' \
  '--remote-debugging-port=9222' \
  '--proxy-server=http://******' \
  '--proxy-bypass-list=localhost;127.0.0.1;${NO_PROXY};*****' \
  --no-sandbox --no-proxy-server '--window-size=1920,1200'
> headless_app$is_alive()
[1] TRUE

App screenshot taken just to make sure the app is really looking how it should (using this method, described by @RLesur):

screenshot = function(file_name = "app-screenshot.png") {
      private$client$Page$captureScreenshot(
        format = "png",
        fromSurface = TRUE
      ) %...>%
        {
          .$data %>%
            jsonlite::base64_dec() %>%
            writeBin(file_name)
        }
    }

So far so good...

text

Then, it starts to become funny:

headless_app$gremlins_horde()
── Sending hordes of gremlins ──────────────────────────────────────────────────────────────────────────────
Shiny is computing
✔ Shiny is still running
>     Sys.sleep(10)
> headless_app$is_alive()
[1] FALSE
> headless_app$wait_for_shiny_ready()
Shiny is computing
Error: The asynchronous job has not finished in the delay of 30 seconds.

headless_app$gremlins_horde() checks the app status: it returns alive but it is not, which subsequently corrupts the end of the workflow.

I don't have explanations at the moment.

sessionInfo()
sessionInfo()
R version 4.1.0 (2021-05-18)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
[1] promises_1.2.0.1

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.7       attempt_0.3.1    ps_1.6.0         later_1.3.0     
 [5] assertthat_0.2.1 waldo_0.2.5      crrri_0.0.13     rappdirs_0.3.3  
 [9] R6_2.5.1         jsonlite_1.7.2   magrittr_2.0.1   httr_1.4.2      
[13] cli_3.0.1        rlang_0.4.11     curl_4.3.2       renv_0.14.0     
[17] testthat_3.0.4   crrry_0.0.0.9001 tools_4.1.0      glue_1.4.2      
[21] purrr_0.3.4      httpuv_1.6.2     compiler_4.1.0   processx_3.5.2  
[25] websocket_1.4.1

I am pretty sure this is a PROXY issue with the gemlins.js library. I am currently checking with local JS script.

A friend of mine once said: "it's always the proxy" ;)

Another friend told me once:

Caught segfault! Address (nil), cause memory not mapped

which translated into French:

Je peux pas, j'ai poney.

More seriously @ColinFay : did you try to run {crrry} on your ThinkR CI/CD servers?

I realised that my app contains a onSessionEnded callback which closes the app, which means something triggers it on CI/CD but not locally.

I thought it was because of a failure to access the gremlins.js script but the script is found. I am running this from the CI/CD Pod:

plop <- httr::GET("http://127.0.0.1:2811/gremlins/gremlins.min.js")
> plop
Response [http://127.0.0.1:2811/gremlins/gremlins.min.js]
  Date: 2022-02-24 21:34
  Status: 200
  Content-Type: application/javascript
  Size: 231 kB
<BINARY BODY>

A dummy request to check:

> plop <- httr::GET("http://127.0.0.1:2811/prout/jeanjacque.js")
> plop
Response [http://127.0.0.1:2811/prout/jeanjacque.js]
  Date: 2022-02-24 21:35
  Status: 404
  Content-Type: text/html; charset=UTF-8
  Size: 18 B

I dumped the DOM to check and realised that locally I get the gremlins JS code inserted:

<script src="./gremlins/gremlins.min.js"></script></body></html>

Then, I dumped the DOM on the CI/CD side. Surprise ... nothing related to gremlins:

</body></html>

I also checked that I can run basic JS command before running the gremlins, to make sure the app is not already broken. According to the WS trace, this makes sense.

SEND {"config":{"workerId":"","sessionId":"93fcd1bdeaa2a149fe1a00629500b641","user":null}}
RECV {"method":"init","data":{"obs":500,".clientdata_output_distPlot_width":1890,".clientdata_output_distPlot_height":400,".clientdata_output_distPlot_bg":"rgb(255, 255, 255)",".clientdata_output_distPlot_fg":"rgb(51, 51, 51)",".clientdata_output_distPlot_accent":"rgb(51, 122, 183)",".clientdata_output_distPlot_font":{"families":["Helvetica Neue","Helvetica","Arial","sans-serif"],"size":"14px"},".clientdata_output_distPlot_hidden":false,".clientdata_pixelratio":1,".clientdata_url_protocol":"http:",".clientdata_url_hostname":"127.0.0.1",".clientdata_url_port":"2811",".clientdata_url_pathname":"/",".clientdata_url_search":"",".clientdata_url_hash_initial":"",".clientdata_url_hash":"",".clientdata_singletons":""}}
SEND {"busy":"busy"}
SEND {"recalculating":{"name":"distPlot","status":"recalculating"}}
SEND {"recalculating":{"name":"distPlot","status":"recalculated"}}
SEND {"busy":"idle"}
SEND {"errors":{},"values":{"distPlot":{"src":"data:image/png;[base64 data]","width":1890,"height":400,"alt":"Plot object","coordmap":{"panels":[{"domain":{"left":-3.76,"right":3.26,"bottom":-4.12,"top":107.12},"range":{"left":59.0400000000001,"right":1859.76,"bottom":325.56,"top":58.04},"log":{"x":null,"y":null},"mapping":{}}],"dims":{"width":1890,"height":400}}}},"inputMessages":[]}
RECV {"method":"update","data":{"obs":300}}
SEND {"progress":{"type":"binding","message":{"id":"distPlot"}}}
SEND {"busy":"busy"}
SEND {"recalculating":{"name":"distPlot","status":"recalculating"}}
SEND {"recalculating":{"name":"distPlot","status":"recalculated"}}
SEND {"busy":"idle"}
SEND {"errors":{},"values":{"distPlot":{"src":"data:image/png;[base64 data]","width":1890,"height":400,"alt":"Plot object","coordmap":{"panels":[{"domain":{"left":-4.28,"right":3.28,"bottom":-2.88,"top":74.88},"range":{"left":59.0399999999999,"right":1859.76,"bottom":325.56,"top":58.04},"log":{"x":null,"y":null},"mapping":{}}],"dims":{"width":1890,"height":400}}}},"inputMessages":[]}

So the questions is why the following code fails to do its job:

(function() {
            function callback() {
                gremlins.createHorde({
                    species: [gremlins.species.clicker(),gremlins.species.toucher(),gremlins.species.formFiller(),gremlins.species.scroller(),gremlins.species.typer()],
                    mogwais: [gremlins.mogwais.alert(),gremlins.mogwais.fps(),gremlins.mogwais.gizmo()],
                    strategies: [gremlins.strategies.distribution()]
                }).unleash();
            }
            var s = document.createElement("script");
            s.src = "https://unpkg.com/gremlins.js";
            if (s.addEventListener) {
                s.addEventListener("load", callback, false);
            } else if (s.readyState) {
                s.onreadystatechange = callback;
            }
            document.body.appendChild(s);
            })()

@RLesur : would you know how to configure headless Chrome to extract the JavaScript errors/warnings from the running app?

Sorry for the late answer @DivadNojnarg (I missed the notification): exceptions can we caught with Runtime.exceptionThrown. An example here.

ColinFay / crrry

Monkey test check does not return consistent result #9