Bibliome / onto-relation-vizualisation

Interface for the synthetic visualization of relational data with ontologies (Florilege).
Apache License 2.0
2 stars 0 forks source link

déploiement de l'application sur le server shiny de Migale #18

Open mandiayba opened 3 years ago

mandiayba commented 3 years ago

Déploiement de l'application onto-relation-vizualisation sur le server shiny hébergé par Migale. Une application shiny à déployer sur le server shiny de Migale doit être dans une image docker et avoir un script de lancement selon. Le dockerfile et le script de lancement seront localisés dans le projet des apps shiny https://forgemia.inra.fr/migale/shinyproxy-docker.

Étapes

  1. création du script de lancement de l'application
  2. création de la recette docker permettant de construire l'image de l'application (Dockerfile). La recette doit (1) installer les dépendances nécessaires pour l'application, (2) créer un dossier dédié à l'application (/opt/ontoRelationVisualizer) où le code de l'application sera copié, (3) déposer le script de lancement de l'application (run.R) dans le dossier de l'application (4) ajouter une petite doc d'utilisation
  3. création et déploiement de l'image docker de l'application dans la VM du server shiny à partir du projet des apps shiny
mandiayba commented 3 years ago

Le HOME de l'application dans le serveur shiny sera /opt/ontoRelationVisualizer

Le contenu du script de lancement inspiré de la doc de l'appli :

$ cat run.R 
library(shiny)
options(shiny.port = 3838)
options(shiny.host = "0.0.0.0")
runApp(appDir = "/opt/ontoRelationVisualizer/inrae/", launch.browser = FALSE)
mandiayba commented 3 years ago

La modification de l'emplacement du dossier data/ sera utile.

Propose de mettre le dossier data/ sous le HOME de l'application.

/opt/ontoRelationVisualizer/data

Il faudra ainsi éditer les lignes de codes concernées pour cette modification

mandiayba commented 3 years ago

We have the run script and data folder containing the ontologies

The dockerfile :

$ cat Dockerfile 
#1.__ base image 
FROM migale/centos-shiny-migale:latest

#2.__ dependences of the shiny app
RUN mkdir /usr/share/doc/R-3.6.0/html/
RUN R -e "install.packages(c('shiny', 'rmarkdown', 'DT', 'future', 'httr', 'jsonlite', 'promises', 'shiny', 'shinyjs', 'shinyWidgets', 'tidyverse', 'visNetwork', 'ontologyIndex', 'devtools', 'sankeyD3'), repos='https://cloud.r-project.org')"
RUN R -e "install.packages('dplyr', repos='https://cloud.r-project.org')"
RUN R -q -e "remotes::install_github('fbreitwieser/sankeyD3', INSTALL_opts=c('--no-help', '--no-html'))"

#3.__ home directory of the app 
RUN mkdir -p /opt/ontoRelationVisualizer

#4.__ source code of the app
RUN git clone https://github.com/Bibliome/onto-relation-vizualisation.git /opt/ontoRelationVisualizer

#5.__ trigger of the app 
COPY run.R /opt/ontoRelationVisualizer/run.R
COPY data /opt/ontoRelationVisualizer/data

#6.__ metadata of the app 
LABEL software="onto-relation-vizualisation"
LABEL software.version="v0.0.0"
LABEL about.summary="Interface for the synthetic visualization of relational data with ontologies (Florilege)."
LABEL about.home="https://github.com/Bibliome/onto-relation-vizualisation"
LABEL about.documentation="https://github.com/Bibliome/onto-relation-vizualisation"
LABEL extra.identifiers="ontoRelationVisualizer"
LABEL about.tags="data, analysis, 16S, interactive"
LABEL about.license_file="https://github.com/Bibliome/onto-relation-vizualisation/blob/main/LICENSE"
LABEL about.license="GNU GENERAL PUBLIC LICENSE"
LABEL author="Mouhamadou Ba, mouhamadou.ba@inrae.fr"

WORKDIR /opt/ontoRelationVisualizer
mandiayba commented 3 years ago

La modification de l'emplacement du dossier data/ sera utile.

Propose de mettre le dossier data/ sous le HOME de l'application.

/opt/ontoRelationVisualizer/data

Il faudra ainsi éditer les lignes de codes concernées pour cette modification

Revoir globalement les chemins relatifs dans le code qui causent des erreurs. Les chemins semblent résolus au niveau du serveur à partir du HOME de l'application qui aura le contenu suivant :

$ls  /opt/ontoRelationVisualizer/
INRAE-onto-visualization.Rproj
LICENSE
README.md
build.sh
data/
docker
inrae/
run.R
run.sh

Les modifications vont se faire dans la branche deploy-2-migale-server crée pour le deploiement

mandiayba commented 3 years ago

Problème de chemins relatifs résolu avec les modifications dans la branche deploy-2-migale-server

mandiayba commented 3 years ago

L'application apparaît dans la liste des apps

image

mais l'application met un temps long à charger...

image

... avant de répondre par une erreur 502

image

Nous avons cependant au moins pu afficher la page d'accueil en local

image

mandiayba commented 3 years ago

on a modifié container-wait-time: 120000 et ça passe https://shiny.migale.inrae.fr/app/ontoRelationVizualizer

image

mandiayba commented 3 years ago

Application déployée et fonctionne sur l'url https://shiny.migale.inrae.fr/app/ontoRelationVizualizer

Une lenteur constatée, elle est apparemment due au démarrage du conteneur docker à chaque nouvelle utilisation de l'application : le conteneur est arrêté après un moment d'inactivé et un nouveau conteneur est crée au lancement de l'appli.

mandiayba commented 3 years ago

Solutions tentées sans succès pour résoudre le problème de lenteur de lancement de l'app (qui abouti à une erreur 502)

Cela ne résout pas le problème, les logs affichent ceci avec le container qui est activé au bout du 49e essai.

avec container-wait-time : 120000

2021-01-29 14:50:24.202  INFO 1 --- [  XNIO-2 task-6] c.s.docker.client.DefaultDockerClient    : Starting container with Id: f2a0dab12993bd0e4522cce30c61d249292bccc5a7909674677b70b17b372dff
2021-01-29 14:50:26.658  WARN 1 --- [  XNIO-2 task-6] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (2/60): http://f2a0dab12993:3838
...
2021-01-29 14:51:58.737  WARN 1 --- [  XNIO-2 task-6] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (48/60): http://f2a0dab12993:3838
2021-01-29 14:52:00.739  WARN 1 --- [  XNIO-2 task-6] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (49/60): http://f2a0dab12993:3838
2021-01-29 14:52:02.998  INFO 1 --- [  XNIO-2 task-6] e.o.containerproxy.service.ProxyService  : Proxy activated [user: cV6g8Jc7jlVt9OhymFHOpcxdwHwl5PDz81-hfOYA] [spec: ontoRelationVisualizer] [id: f843c3cf-36bd-4544-92eb-206522ec0ddb]

avec container-wait-time : 300000

2021-01-29 17:24:00.842  INFO 1 --- [  XNIO-2 task-3] c.s.docker.client.DefaultDockerClient    : Starting container with Id: 8f5b82297fe13e7324984c87fd2bd821f7e4eadbb6e8c94a8389e5cb1ac56efc
2021-01-29 17:24:03.552  WARN 1 --- [  XNIO-2 task-3] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (2/150): http://8f5b82297fe1:3838
...
2021-01-29 17:25:37.655  WARN 1 --- [  XNIO-2 task-3] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (49/150): http://8f5b82297fe1:3838
2021-01-29 17:25:39.929  INFO 1 --- [  XNIO-2 task-3] e.o.containerproxy.service.ProxyService  : Proxy activated [user: cV6g8Jc7jlVt9OhymFHOpcxdwHwl5PDz81-hfOYA] [spec: ontoRelationVisualizer] [id: 12482c51-140d-4839-b716-9ab2769c2b16]
2021-01-29 17:26:52.558  INFO 1 --- [tiveProxyKiller] e.o.c.service.HeartbeatService           : Releasing inactive proxy [user: cV6g8Jc7jlVt9OhymFHOpcxdwHwl5PDz81-hfOYA] [spec: ontoRelationVisualizer] [id: 12482c51-140d-4839-b716-9ab2769c2b16] [silence: 72649ms]

avec container-wait-time : 3000000

2021-01-29 17:44:20.327  INFO 1 --- [  XNIO-2 task-3] c.s.docker.client.DefaultDockerClient    : Starting container with Id: 08c6b20cc23dcb76df5e0045ef21f9abd92f1f52fc46abc2302381d816edcbf6
2021-01-29 17:44:22.995  WARN 1 --- [  XNIO-2 task-3] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (2/1500): http://08c6b20cc23d:3838
...
2021-01-29 17:45:53.088  WARN 1 --- [  XNIO-2 task-3] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (47/1500): http://08c6b20cc23d:3838
2021-01-29 17:45:55.340  INFO 1 --- [  XNIO-2 task-3] e.o.containerproxy.service.ProxyService  : Proxy activated [user: cV6g8Jc7jlVt9OhymFHOpcxdwHwl5PDz81-hfOYA] [spec: ontoRelationVisualizer] [id: 0ae8938b-f4c8-44c7-a752-85a981ce424a]

avec container-wait-time : 300000000

2021-01-29 17:57:12.025  INFO 1 --- [  XNIO-2 task-3] c.s.docker.client.DefaultDockerClient    : Starting container with Id: 51a927de9dfc939d66d3e92f3befdc7c11fd4c801cb5e79b7d41b26bc0185e78
2021-01-29 17:57:14.656  WARN 1 --- [  XNIO-2 task-3] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (2/150000): http://51a927de9dfc:3838
...
2021-01-29 17:58:44.746  WARN 1 --- [  XNIO-2 task-3] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (47/150000): http://51a927de9dfc:3838
2021-01-29 17:58:47.009  INFO 1 --- [  XNIO-2 task-3] e.o.containerproxy.service.ProxyService  : Proxy activated [user: cV6g8Jc7jlVt9OhymFHOpcxdwHwl5PDz81-hfOYA] [spec: ontoRelationVisualizer] [id: a594b460-2b2b-4dfc-bf1b-2796f6201e08]

L'application fonctionne cependant ensuite si on attend un peu et qu'on rafraîchit la page après l'erreur 502

mandiayba commented 3 years ago

reprise des tests sur un server shiny dev sur migale. Aucune solution trouvée pour la lenteur de orv

Comparaison de temps de lancement entre OntoRelationVisualizer et Easy16S

 ontoRelationVisualizer]# docker run -v /opt/shinyproxy-docker/ontoRelationVisualizer/data:/opt/ontoRelationVisualizer/data -it -p 3838:3838 migale/ontorelationvisualizer:latest

[root@a6d652a27023 ontoRelationVisualizer]# time Rscript --verbose run.R
running
  '/usr/lib64/R/bin/R --slave --no-restore --file=run.R'

Failed to create bus connection: No such file or directory
Warning in system("timedatectl", intern = TRUE) :
  running command 'timedatectl' had status 1

Listening on http://0.0.0.0:3838
^C

Execution halted

real    1m37.827s
user    1m34.093s
sys 0m1.730s

Warning in read.dcf(con) : invalid input found on input connection './easy16S//DESCRIPTION'

Listening on http://0.0.0.0:3838 ^C

Execution halted

real 0m4.554s user 0m0.430s sys 0m0.134s

mandiayba commented 3 years ago

Investigation poussée sur le temps de lancement

Pour une raison que j'ignore le garbage collector prend beaucoup de temps avant le lancement de orv, le lancement se fait trop tardivement comparé à d'autres outils

[root@e00ad1e0100d ontoRelationVisualizer]# R --verbose -e "shiny::runApp('/opt/ontoRelationVisualizer', host = '0.0.0.0', port = 3838)"
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/methods/libs/methods.so") ...

R version 3.6.0 (2019-04-26) -- "Planting of a Tree"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-redhat-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

'verbose' and 'quietly' are both true; being verbose then ..
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/utils/libs/utils.so") ...
Garbage collection 1 = 0+0+1 (level 2) ... 
12.1 Mbytes of cons cells used (35%)
3.1 Mbytes of vectors used (5%)
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/grDevices/libs/grDevices.so") ...
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/graphics/libs/graphics.so") ...
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/stats/libs/stats.so") ...
 ending setup_Rmainloop(): R_Interactive = 0 {main.c}
 >R_ReplConsole(): before "for(;;)" {main.c}
> shiny::runApp('/opt/ontoRelationVisualizer', host = '0.0.0.0', port = 3838)
now dyn.load("/usr/lib64/R/library/digest/libs/digest.so") ...
now dyn.load("/usr/lib64/R/library/rlang/libs/rlang.so") ...
now dyn.load("/usr/lib64/R/library/htmltools/libs/htmltools.so") ...
now dyn.load("/usr/lib64/R/library/Rcpp/libs/Rcpp.so") ...
Garbage collection 2 = 1+0+1 (level 0) ... 
17.4 Mbytes of cons cells used (51%)
4.2 Mbytes of vectors used (6%)
Garbage collection 3 = 2+0+1 (level 0) ... 
20.8 Mbytes of cons cells used (60%)
5.1 Mbytes of vectors used (8%)
now dyn.load("/usr/lib64/R/library/later/libs/later.so") ...
now dyn.load("/usr/lib64/R/library/magrittr/libs/magrittr.so") ...
now dyn.load("/usr/lib64/R/library/promises/libs/promises.so") ...
now dyn.load("/usr/lib64/R/library/httpuv/libs/httpuv.so") ...
now dyn.load("/usr/lib64/R/library/mime/libs/mime.so") ...
now dyn.load("/usr/lib64/R/library/ellipsis/libs/ellipsis.so") ...
now dyn.load("/usr/lib64/R/library/fastmap/libs/fastmap.so") ...
Garbage collection 4 = 3+0+1 (level 0) ... 
21.4 Mbytes of cons cells used (62%)
5.3 Mbytes of vectors used (8%)
Loading required package: shiny
now dyn.load("/usr/lib64/R/library/tools/libs/tools.so") ...
Garbage collection 5 = 4+0+1 (level 0) ... 
23.8 Mbytes of cons cells used (69%)
5.8 Mbytes of vectors used (9%)
now dyn.load("/usr/lib64/R/library/cachem/libs/cachem.so") ...
Garbage collection 6 = 5+0+1 (level 0) ... 
26.0 Mbytes of cons cells used (76%)
7.2 Mbytes of vectors used (11%)
Garbage collection 7 = 6+0+1 (level 0) ... 
27.2 Mbytes of cons cells used (79%)
7.5 Mbytes of vectors used (12%)
Garbage collection 8 = 7+0+1 (level 0) ... 
27.8 Mbytes of cons cells used (81%)
7.6 Mbytes of vectors used (12%)
Garbage collection 9 = 7+1+1 (level 1) ... 
26.7 Mbytes of cons cells used (78%)
7.1 Mbytes of vectors used (11%)
Garbage collection 10 = 8+1+1 (level 0) ... 
27.2 Mbytes of cons cells used (79%)
7.2 Mbytes of vectors used (11%)
Garbage collection 11 = 9+1+1 (level 0) ... 
30.6 Mbytes of cons cells used (89%)
8.3 Mbytes of vectors used (13%)
Garbage collection 12 = 9+2+1 (level 1) ... 
30.7 Mbytes of cons cells used (89%)
8.3 Mbytes of vectors used (13%)
Garbage collection 13 = 9+2+2 (level 2) ... 
30.9 Mbytes of cons cells used (43%)
8.3 Mbytes of vectors used (13%)
Garbage collection 14 = 10+2+2 (level 0) ... 
34.0 Mbytes of cons cells used (48%)
9.0 Mbytes of vectors used (14%)
Garbage collection 15 = 11+2+2 (level 0) ... 
35.3 Mbytes of cons cells used (50%)
9.2 Mbytes of vectors used (14%)
Garbage collection 16 = 12+2+2 (level 0) ... 
37.9 Mbytes of cons cells used (53%)
10.0 Mbytes of vectors used (16%)
Failed to create bus connection: No such file or directory
Warning in system("timedatectl", intern = TRUE) :
  running command 'timedatectl' had status 1
Garbage collection 17 = 13+2+2 (level 0) ... 
43.9 Mbytes of cons cells used (62%)
11.2 Mbytes of vectors used (17%)
Garbage collection 18 = 14+2+2 (level 0) ... 
46.6 Mbytes of cons cells used (66%)
11.8 Mbytes of vectors used (18%)
Garbage collection 19 = 15+2+2 (level 0) ... 
47.3 Mbytes of cons cells used (67%)
12.0 Mbytes of vectors used (19%)
Garbage collection 20 = 16+2+2 (level 0) ... 
48.8 Mbytes of cons cells used (69%)
12.4 Mbytes of vectors used (19%)
Garbage collection 21 = 17+2+2 (level 0) ... 
50.5 Mbytes of cons cells used (71%)
12.8 Mbytes of vectors used (20%)
Garbage collection 22 = 18+2+2 (level 0) ... 
51.6 Mbytes of cons cells used (73%)
13.1 Mbytes of vectors used (20%)
'envir' chosen:<environment: R_GlobalEnv>
encoding = "native.enc" chosen
--> parsed 3 expressions; now eval(.)ing them:

>>>> eval(expression_nr. 1 )
         =================

> invert <- function(list, simplify = T) {
+     new_list <- names(list)
+     quote <- paste0("\"", as.character(list), "\"")
+     unquote <- as.cha .... [TRUNCATED] 
curr.fun: symbol <-
 .. after 'expression(invert <- function(list, simplify = T) {''    new_list <- names(list)''    quote <- paste0("\"", as.character(list), "\"")''    unquote <- as.character(list)''    names(new_list) <- if (simplify) {''        unquote''    }''    else {''        quote''    }''    return(new_list)''})'

>>>> eval(expression_nr. 2 )
         =================

> is.valid <- function(x) {
+     require(shiny)
+     is.null(need(x, message = FALSE))
+ }
curr.fun: symbol <-
 .. after 'expression(is.valid <- function(x) {''    require(shiny)''    is.null(need(x, message = FALSE))''})'

>>>> eval(expression_nr. 3 )
         =================

> listToReactive <- function(list) {
+     require(shiny)
+     do.call("reactiveValues", list)
+ }
curr.fun: symbol <-
 .. after 'expression(listToReactive <- function(list) {''    require(shiny)''    do.call("reactiveValues", list)''})'
Garbage collection 23 = 19+2+2 (level 0) ... 
55.4 Mbytes of cons cells used (78%)
14.8 Mbytes of vectors used (23%)
Garbage collection 24 = 20+2+2 (level 0) ... 
55.7 Mbytes of cons cells used (78%)
14.9 Mbytes of vectors used (23%)
Garbage collection 25 = 20+3+2 (level 1) ... 
55.6 Mbytes of cons cells used (78%)
14.8 Mbytes of vectors used (23%)
Garbage collection 26 = 21+3+2 (level 0) ... 
56.1 Mbytes of cons cells used (79%)
15.2 Mbytes of vectors used (24%)
Garbage collection 27 = 22+3+2 (level 0) ... 
56.8 Mbytes of cons cells used (80%)
15.6 Mbytes of vectors used (24%)
Garbage collection 28 = 23+3+2 (level 0) ... 
57.5 Mbytes of cons cells used (81%)
16.0 Mbytes of vectors used (25%)
Garbage collection 29 = 23+4+2 (level 1) ... 
56.7 Mbytes of cons cells used (80%)
15.8 Mbytes of vectors used (25%)
Garbage collection 30 = 24+4+2 (level 0) ... 
57.1 Mbytes of cons cells used (80%)
16.5 Mbytes of vectors used (26%)
Garbage collection 31 = 24+5+2 (level 1) ... 
62.6 Mbytes of cons cells used (88%)
22.5 Mbytes of vectors used (35%)
Garbage collection 32 = 24+5+3 (level 2) ... 
66.5 Mbytes of cons cells used (46%)
30.9 Mbytes of vectors used (48%)
Garbage collection 33 = 24+6+3 (level 1) ... 
88.6 Mbytes of cons cells used (61%)
54.2 Mbytes of vectors used (85%)
Garbage collection 34 = 24+6+4 (level 2) ... 
97.5 Mbytes of cons cells used (67%)
60.0 Mbytes of vectors used (48%)
Garbage collection 35 = 24+6+5 (level 2) ... 
136.9 Mbytes of cons cells used (46%)
105.8 Mbytes of vectors used (70%)
Garbage collection 36 = 24+6+6 (level 2) ... 
141.7 Mbytes of cons cells used (48%)
109.8 Mbytes of vectors used (49%)
Garbage collection 37 = 25+6+6 (level 0) ... 
152.7 Mbytes of cons cells used (52%)
154.0 Mbytes of vectors used (69%)
Garbage collection 38 = 25+6+7 (level 2) ... 
152.7 Mbytes of cons cells used (52%)
175.4 Mbytes of vectors used (65%)
Garbage collection 39 = 26+6+7 (level 0) ... 
152.7 Mbytes of cons cells used (52%)
207.0 Mbytes of vectors used (77%)
Garbage collection 40 = 26+6+8 (level 2) ... 
152.7 Mbytes of cons cells used (52%)
203.5 Mbytes of vectors used (63%)
Garbage collection 41 = 27+6+8 (level 0) ... 
152.7 Mbytes of cons cells used (52%)
263.2 Mbytes of vectors used (82%)
Garbage collection 42 = 27+7+8 (level 1) ... 
152.7 Mbytes of cons cells used (52%)
266.7 Mbytes of vectors used (83%)
Garbage collection 43 = 27+7+9 (level 2) ... 
268.3 Mbytes of cons cells used (47%)
276.4 Mbytes of vectors used (71%)
Garbage collection 44 = 27+7+10 (level 2) ... 
571.0 Mbytes of cons cells used (46%)
311.0 Mbytes of vectors used (67%)
Garbage collection 45 = 28+7+10 (level 0) ... 
890.5 Mbytes of cons cells used (71%)
368.6 Mbytes of vectors used (79%)
Garbage collection 46 = 29+7+10 (level 0) ... 
890.5 Mbytes of cons cells used (71%)
368.6 Mbytes of vectors used (79%)
Garbage collection 47 = 30+7+10 (level 0) ... 
890.5 Mbytes of cons cells used (71%)
389.7 Mbytes of vectors used (84%)
Garbage collection 48 = 30+8+10 (level 1) ... 
890.5 Mbytes of cons cells used (71%)
368.6 Mbytes of vectors used (79%)
Garbage collection 49 = 31+8+10 (level 0) ... 
890.5 Mbytes of cons cells used (71%)
368.6 Mbytes of vectors used (79%)
Garbage collection 50 = 32+8+10 (level 0) ... 
892.1 Mbytes of cons cells used (71%)
390.5 Mbytes of vectors used (84%)
Garbage collection 51 = 32+9+10 (level 1) ... 
899.2 Mbytes of cons cells used (72%)
394.2 Mbytes of vectors used (85%)
Garbage collection 52 = 32+9+11 (level 2) ... 
906.2 Mbytes of cons cells used (51%)
376.7 Mbytes of vectors used (67%)
Garbage collection 53 = 33+9+11 (level 0) ... 
923.5 Mbytes of cons cells used (52%)
401.7 Mbytes of vectors used (72%)
Garbage collection 54 = 34+9+11 (level 0) ... 
940.3 Mbytes of cons cells used (53%)
410.4 Mbytes of vectors used (73%)
Garbage collection 55 = 35+9+11 (level 0) ... 
957.0 Mbytes of cons cells used (54%)
419.1 Mbytes of vectors used (75%)
Garbage collection 56 = 35+10+11 (level 1) ... 
973.3 Mbytes of cons cells used (55%)
427.9 Mbytes of vectors used (77%)
Garbage collection 57 = 36+10+11 (level 0) ... 
989.3 Mbytes of cons cells used (56%)
436.5 Mbytes of vectors used (78%)
Garbage collection 58 = 37+10+11 (level 0) ... 
1004.9 Mbytes of cons cells used (57%)
444.9 Mbytes of vectors used (80%)
Garbage collection 59 = 38+10+11 (level 0) ... 
1020.1 Mbytes of cons cells used (58%)
453.0 Mbytes of vectors used (81%)
Garbage collection 60 = 38+11+11 (level 1) ... 
1035.1 Mbytes of cons cells used (58%)
461.0 Mbytes of vectors used (83%)
Garbage collection 61 = 38+11+12 (level 2) ... 
1049.7 Mbytes of cons cells used (59%)
460.9 Mbytes of vectors used (69%)
Garbage collection 62 = 39+11+12 (level 0) ... 
1064.1 Mbytes of cons cells used (60%)
468.6 Mbytes of vectors used (70%)
Garbage collection 63 = 40+11+12 (level 0) ... 
1078.1 Mbytes of cons cells used (61%)
476.2 Mbytes of vectors used (71%)
Garbage collection 64 = 41+11+12 (level 0) ... 
1092.0 Mbytes of cons cells used (62%)
484.5 Mbytes of vectors used (72%)
Garbage collection 65 = 42+11+12 (level 0) ... 
1105.6 Mbytes of cons cells used (62%)
492.8 Mbytes of vectors used (73%)
Garbage collection 66 = 43+11+12 (level 0) ... 
1118.9 Mbytes of cons cells used (63%)
501.0 Mbytes of vectors used (75%)
Garbage collection 67 = 44+11+12 (level 0) ... 
1131.9 Mbytes of cons cells used (64%)
509.0 Mbytes of vectors used (76%)
Garbage collection 68 = 45+11+12 (level 0) ... 
1221.2 Mbytes of cons cells used (69%)
565.9 Mbytes of vectors used (84%)
Garbage collection 69 = 45+11+13 (level 2) ... 
406.2 Mbytes of cons cells used (29%)
407.3 Mbytes of vectors used (61%)
Garbage collection 70 = 46+11+13 (level 0) ... 
546.0 Mbytes of cons cells used (38%)
487.0 Mbytes of vectors used (73%)
Garbage collection 71 = 47+11+13 (level 0) ... 
656.6 Mbytes of cons cells used (46%)
580.5 Mbytes of vectors used (86%)
Garbage collection 72 = 47+12+13 (level 1) ... 
521.9 Mbytes of cons cells used (37%)
480.4 Mbytes of vectors used (72%)
Garbage collection 73 = 48+12+13 (level 0) ... 
537.1 Mbytes of cons cells used (38%)
482.6 Mbytes of vectors used (72%)
Garbage collection 74 = 49+12+13 (level 0) ... 
545.9 Mbytes of cons cells used (38%)
519.0 Mbytes of vectors used (77%)
Garbage collection 75 = 50+12+13 (level 0) ... 
545.9 Mbytes of cons cells used (38%)
582.2 Mbytes of vectors used (87%)
Garbage collection 76 = 50+13+13 (level 1) ... 
545.9 Mbytes of cons cells used (38%)
536.5 Mbytes of vectors used (80%)
Garbage collection 77 = 50+13+14 (level 2) ... 
545.9 Mbytes of cons cells used (38%)
535.6 Mbytes of vectors used (66%)
Garbage collection 78 = 51+13+14 (level 0) ... 
597.3 Mbytes of cons cells used (42%)
596.1 Mbytes of vectors used (74%)
Garbage collection 79 = 52+13+14 (level 0) ... 
597.3 Mbytes of cons cells used (42%)
644.3 Mbytes of vectors used (80%)
Garbage collection 80 = 52+14+14 (level 1) ... 
632.0 Mbytes of cons cells used (45%)
574.8 Mbytes of vectors used (71%)
Garbage collection 81 = 53+14+14 (level 0) ... 
695.6 Mbytes of cons cells used (49%)
636.3 Mbytes of vectors used (79%)
Garbage collection 82 = 54+14+14 (level 0) ... 
695.6 Mbytes of cons cells used (49%)
701.6 Mbytes of vectors used (87%)
Garbage collection 83 = 54+15+14 (level 1) ... 
690.4 Mbytes of cons cells used (49%)
611.3 Mbytes of vectors used (76%)
Garbage collection 84 = 55+15+14 (level 0) ... 
734.9 Mbytes of cons cells used (52%)
628.3 Mbytes of vectors used (78%)
Garbage collection 85 = 55+16+14 (level 1) ... 
648.6 Mbytes of cons cells used (46%)
609.4 Mbytes of vectors used (75%)
Garbage collection 86 = 56+16+14 (level 0) ... 
697.8 Mbytes of cons cells used (49%)
643.1 Mbytes of vectors used (80%)
Garbage collection 87 = 57+16+14 (level 0) ... 
706.2 Mbytes of cons cells used (50%)
653.7 Mbytes of vectors used (81%)
Garbage collection 88 = 57+16+15 (level 2) ... 
558.9 Mbytes of cons cells used (39%)
571.3 Mbytes of vectors used (59%)
Garbage collection 89 = 58+16+15 (level 0) ... 
576.4 Mbytes of cons cells used (41%)
591.6 Mbytes of vectors used (61%)
Garbage collection 90 = 59+16+15 (level 0) ... 
595.9 Mbytes of cons cells used (42%)
616.0 Mbytes of vectors used (63%)
Garbage collection 91 = 60+16+15 (level 0) ... 
607.2 Mbytes of cons cells used (43%)
634.4 Mbytes of vectors used (65%)
Garbage collection 92 = 61+16+15 (level 0) ... 
622.3 Mbytes of cons cells used (44%)
654.5 Mbytes of vectors used (67%)

Listening on http://0.0.0.0:3838
mandiayba commented 3 years ago

Garbage collector plus court avant lancement pour easy16S par exemple

[root@5d679f6ea33d easy16S]# R --verbose -e "shiny::runApp('/opt/easy16S/easy16S', host = '0.0.0.0', port = 3838)"
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/methods/libs/methods.so") ...

R version 3.6.0 (2019-04-26) -- "Planting of a Tree"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-redhat-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

'verbose' and 'quietly' are both true; being verbose then ..
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/utils/libs/utils.so") ...
Garbage collection 1 = 0+0+1 (level 2) ... 
12.1 Mbytes of cons cells used (35%)
3.1 Mbytes of vectors used (5%)
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/grDevices/libs/grDevices.so") ...
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/graphics/libs/graphics.so") ...
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/stats/libs/stats.so") ...
 ending setup_Rmainloop(): R_Interactive = 0 {main.c}
 >R_ReplConsole(): before "for(;;)" {main.c}
> shiny::runApp('/opt/easy16S/easy16S', host = '0.0.0.0', port = 3838)
now dyn.load("/usr/lib64/R/library/digest/libs/digest.so") ...
now dyn.load("/usr/lib64/R/library/rlang/libs/rlang.so") ...
now dyn.load("/usr/lib64/R/library/htmltools/libs/htmltools.so") ...
now dyn.load("/usr/lib64/R/library/Rcpp/libs/Rcpp.so") ...
Garbage collection 2 = 1+0+1 (level 0) ... 
17.4 Mbytes of cons cells used (51%)
4.2 Mbytes of vectors used (6%)
Garbage collection 3 = 2+0+1 (level 0) ... 
20.8 Mbytes of cons cells used (60%)
5.1 Mbytes of vectors used (8%)
now dyn.load("/usr/lib64/R/library/later/libs/later.so") ...
now dyn.load("/usr/lib64/R/library/magrittr/libs/magrittr.so") ...
now dyn.load("/usr/lib64/R/library/promises/libs/promises.so") ...
now dyn.load("/usr/lib64/R/library/httpuv/libs/httpuv.so") ...
now dyn.load("/usr/lib64/R/library/mime/libs/mime.so") ...
now dyn.load("/usr/lib64/R/library/ellipsis/libs/ellipsis.so") ...
now dyn.load("/usr/lib64/R/library/fastmap/libs/fastmap.so") ...
Garbage collection 4 = 3+0+1 (level 0) ... 
21.4 Mbytes of cons cells used (62%)
5.3 Mbytes of vectors used (8%)
Loading required package: shiny
now dyn.load("/usr/lib64/R/library/tools/libs/tools.so") ...
Garbage collection 5 = 4+0+1 (level 0) ... 
23.8 Mbytes of cons cells used (69%)
5.8 Mbytes of vectors used (9%)
now dyn.load("/usr/lib64/R/library/cachem/libs/cachem.so") ...
Warning in read.dcf(con) :
  invalid input found on input connection '/opt/easy16S/easy16S/DESCRIPTION'
Warning in loadSupport(appDir, renv = sharedEnv, globalrenv = globalenv()) :
  Loading R/ subdirectory for Shiny application, but this directory appears to contain an R package. Sourcing files in R/ may cause unexpected behavior.

Listening on http://0.0.0.0:3838
Garbage collection 6 = 5+0+1 (level 0) ... 
30.6 Mbytes of cons cells used (89%)
8.8 Mbytes of vectors used (14%)
mandiayba commented 3 years ago

Le cas de MALDIquantTypeR

[root@2f1e3482e36c MALDIquantTypeR]# R --verbose -e "shiny::runApp('/opt/MALDIquantTypeR/MALDIquantTypeR', host = '0.0.0.0', port = 3838)"
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/methods/libs/methods.so") ...

R version 3.6.0 (2019-04-26) -- "Planting of a Tree"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-redhat-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

'verbose' and 'quietly' are both true; being verbose then ..
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/utils/libs/utils.so") ...
Garbage collection 1 = 0+0+1 (level 2) ... 
12.1 Mbytes of cons cells used (35%)
3.1 Mbytes of vectors used (5%)
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/grDevices/libs/grDevices.so") ...
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/graphics/libs/graphics.so") ...
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/stats/libs/stats.so") ...
 ending setup_Rmainloop(): R_Interactive = 0 {main.c}
 >R_ReplConsole(): before "for(;;)" {main.c}
> shiny::runApp('/opt/MALDIquantTypeR/MALDIquantTypeR', host = '0.0.0.0', port = 3838)
now dyn.load("/usr/lib64/R/library/digest/libs/digest.so") ...
now dyn.load("/usr/lib64/R/library/rlang/libs/rlang.so") ...
now dyn.load("/usr/lib64/R/library/htmltools/libs/htmltools.so") ...
now dyn.load("/usr/lib64/R/library/Rcpp/libs/Rcpp.so") ...
Garbage collection 2 = 1+0+1 (level 0) ... 
17.4 Mbytes of cons cells used (51%)
4.2 Mbytes of vectors used (6%)
Garbage collection 3 = 2+0+1 (level 0) ... 
20.7 Mbytes of cons cells used (60%)
5.1 Mbytes of vectors used (8%)
now dyn.load("/usr/lib64/R/library/later/libs/later.so") ...
now dyn.load("/usr/lib64/R/library/magrittr/libs/magrittr.so") ...
now dyn.load("/usr/lib64/R/library/promises/libs/promises.so") ...
now dyn.load("/usr/lib64/R/library/httpuv/libs/httpuv.so") ...
now dyn.load("/usr/lib64/R/library/mime/libs/mime.so") ...
now dyn.load("/usr/lib64/R/library/fastmap/libs/fastmap.so") ...
Garbage collection 4 = 3+0+1 (level 0) ... 
21.4 Mbytes of cons cells used (62%)
5.3 Mbytes of vectors used (8%)
now dyn.load("/usr/lib64/R/library/tools/libs/tools.so") ...
Loading required package: shiny

Listening on http://0.0.0.0:3838
Garbage collection 5 = 4+0+1 (level 0) ... 
27.6 Mbytes of cons cells used (80%)
7.0 Mbytes of vectors used (11%)
Garbage collection 6 = 4+1+1 (level 1) ... 
31.1 Mbytes of cons cells used (90%)
7.7 Mbytes of vectors used (12%)
mandiayba commented 3 years ago

La lenteur est causée par les instructions dans le global.R ==> il faut trouver un moyen de charger les données après l’affichage de la page d’accueil pour que l’application fonctionne sur le server shiny de Migale

test sur un hello world !!!

[root@3eb21322f482 ontoRelationVisualizer]# cat ui.R        
############################# LIBRARIES ################################

library(shiny)
library(shinyjs)
library(shinyWidgets)
library(tidyverse)

################################# UI ###################################

shinyUI(bootstrapPage(
     theme = "styles.css",
     "Hello, world!!!"
)

[root@3eb21322f482 ontoRelationVisualizer]# rm -rf global.R 

[root@3eb21322f482 ontoRelationVisualizer]# ls
INRAE-onto-visualization.Rproj  LICENSE  README.md  build.sh  conf.ini  data  docker  help-ui.R  run.R  run.sh  scripts  server.R  ui.R  www

[root@3eb21322f482 ontoRelationVisualizer]# R --verbose -e "shiny::runApp('/opt/ontoRelationVisualizer', host = '0.0.0.0', port = 3838)"
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/methods/libs/methods.so") ...

R version 3.6.0 (2019-04-26) -- "Planting of a Tree"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-redhat-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

'verbose' and 'quietly' are both true; being verbose then ..
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/utils/libs/utils.so") ...
Garbage collection 1 = 0+0+1 (level 2) ... 
12.1 Mbytes of cons cells used (35%)
3.1 Mbytes of vectors used (5%)
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/grDevices/libs/grDevices.so") ...
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/graphics/libs/graphics.so") ...
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/stats/libs/stats.so") ...
 ending setup_Rmainloop(): R_Interactive = 0 {main.c}
 >R_ReplConsole(): before "for(;;)" {main.c}
> shiny::runApp('/opt/ontoRelationVisualizer', host = '0.0.0.0', port = 3838)
now dyn.load("/usr/lib64/R/library/digest/libs/digest.so") ...
now dyn.load("/usr/lib64/R/library/rlang/libs/rlang.so") ...
now dyn.load("/usr/lib64/R/library/htmltools/libs/htmltools.so") ...
now dyn.load("/usr/lib64/R/library/Rcpp/libs/Rcpp.so") ...
Garbage collection 2 = 1+0+1 (level 0) ... 
17.4 Mbytes of cons cells used (51%)
4.2 Mbytes of vectors used (6%)
Garbage collection 3 = 2+0+1 (level 0) ... 
20.8 Mbytes of cons cells used (60%)
5.1 Mbytes of vectors used (8%)
now dyn.load("/usr/lib64/R/library/later/libs/later.so") ...
now dyn.load("/usr/lib64/R/library/magrittr/libs/magrittr.so") ...
now dyn.load("/usr/lib64/R/library/promises/libs/promises.so") ...
now dyn.load("/usr/lib64/R/library/httpuv/libs/httpuv.so") ...
now dyn.load("/usr/lib64/R/library/mime/libs/mime.so") ...
now dyn.load("/usr/lib64/R/library/ellipsis/libs/ellipsis.so") ...
now dyn.load("/usr/lib64/R/library/fastmap/libs/fastmap.so") ...
Garbage collection 4 = 3+0+1 (level 0) ... 
21.4 Mbytes of cons cells used (62%)
5.3 Mbytes of vectors used (8%)
Loading required package: shiny
now dyn.load("/usr/lib64/R/library/tools/libs/tools.so") ...
Garbage collection 5 = 4+0+1 (level 0) ... 
23.8 Mbytes of cons cells used (69%)
5.8 Mbytes of vectors used (9%)
now dyn.load("/usr/lib64/R/library/cachem/libs/cachem.so") ...

Listening on http://0.0.0.0:3838
Garbage collection 6 = 5+0+1 (level 0) ... 
30.6 Mbytes of cons cells used (89%)
8.8 Mbytes of vectors used (14%)
letailli commented 3 years ago

Les données des ontologies chargées au début de global.R font environ 72 Mo. Et ce, même après les transformations / réductions que nous avions réalisées sur les fichiers bruts (qui étaient autour de 600 Mo).

  habitat = get_ontology("../../data/BioNLP-OST+EnovFood-Habitat.obo"),
  phenotype = get_ontology("../../data/BioNLP-OST+EnovFood-Phenotype.obo"),
  use = get_ontology("../../data/Use_V2.obo"),
  taxon = get_ontology("../../data/microorganisms.obo")

On peut regarder à quel moment il est préférable de les charger mais il faut éviter que cela se fasse au moment où l'utilisateur en a besoin car cela provoquerait de grosses lenteurs (ce que nous avions lors des 1e tests). Etes-vous toujours sur une architecture qui vise à arrêter/redémarrer régulièrement le service ? Je crains que cela soit problématique si on veut éviter de répéter ces chargements longs.

Je suis dispo pour en discuter si besoin.

mandiayba commented 3 years ago

Merci @letailli pour la précision.

Nous voulons installer sur le ShinyProxy de Migale car cela permet de ne pas avoir de limitation sur le nombre d'utilisateurs travaillant en même temps (chaque utilisateur aura son conteneur indépendant) et cela permet de regrouper les applis shiny au même endroit mais effectivement si on n'a pas de solution pour le temps de chargement on peut essayer de déployer autrement.

Je ne connais pas bien R et shiny mais j'ai essayé de regarder le coté traitement asynchrone avec future et promises. Apparemment c'est possible d'exécuter le chargement en background et laisser une interface d'accueil s'afficher et éviter ainsi l'erreur 502. Il semble également possible de changer une fois les données et les rendre accessibles à toutes les sessions (ce que vous faites déja avec global.R). A voir si c'est simple à mettre en place...

En attendant je vais déployer sur une VM en dehors du shinyproxy. La lenteur au démarrage restera mais au moins on aura l'appli qui tourne.

letailli commented 3 years ago

Merci @mandiayba

Je pense qu'il faut éviter de charger ces données pour chaque utilisateur. Cela me semble à la fois lourd pour le serveur et oblige à un temps de chargement pour chaque utilisateur.

As-tu plus d'info sur la possibilité de changer une fois les données et les rendre accessibles à toutes les sessions dans l'archi Migale ? On se prévoir un point la semaine prochaine pour en parler.

mandiayba commented 3 years ago

Ce qu'on appelle ShinyProxy c'est un serveur qui accueille des applications shiny encapsulées dans des images docker. Le proxy surveille en permanence les applis déployées. Il reste donc en permanence à l'écoute et dés qu'un user côté navigateur demande une appli (via une URL dédiée) le proxy se charge de déployer un conteneur dédié à ce user dans le quel tourne une instance de l'appli demandée. L'utilisateur peut ainsi travailler tranquillement avec cette instance indépendante. Si l'instance déployée reste inactive pendant un temps T le proxy se charge de l’arrêter en supprimant le conteneur. On peut configurer le temps T mais il s'applique à toutes les applis d'un shinyproxy. Nous l'avons fixé à moins d'une minute pour éviter qu'il y ait trop d'applis inactives mais on peut l'adapter si besoin.

Je te donne plus de détails la semaine prochaine.

Le sam. 3 avr. 2021 à 11:40, Charles Letaillieur @.***> a écrit :

Merci @mandiayba https://github.com/mandiayba

On utilise déjà future à plusieurs endroits. On peut regarder dans ce sens pour être compliant avec Migale. Pour éviter les lenteurs côté utilisateur, j'ai besoin de comprendre quand est-ce que le service sera redémarré. Régulièrement (chaque jour ? Chaque semaine ?) ou bien en fonction du trafic (il est éteint si il n'y a pas de users)?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Bibliome/onto-relation-vizualisation/issues/18#issuecomment-812841322, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEHRRTS5PII4YHBVVW6OCLTTG3O7BANCNFSM4U4LPWPQ .

mandiayba commented 3 years ago

Je ne sais pas si c'est possible de configurer le chargement des données par le proxy mais c'est une idée intéressante que je vais explorer.

Je pensais initialement à ne pas répéter l'étape de chargement si plusieurs instances tournent en même temps, de mettre une condition qui fait que les données seront chargées par la première instance et les instances suivantes en profite. Une condition du genre...

if_not_exist(GLOBAL_DATA)  GLOBAL_DATA <- load(...)

Le sam. 3 avr. 2021 à 11:45, Charles Letaillieur @.***> a écrit :

Merci @mandiayba https://github.com/mandiayba

Je pense qu'il faut éviter de charger ces données pour chaque utilisateur. Cela me semble à la fois lourd pour le serveur et oblige à un temps de chargement pour chaque utilisateur.

As-tu plus d'info sur la possibilité de changer une fois les données et les rendre accessibles à toutes les sessions dans l'archi Migale ? On se prévoir un point la semaine prochaine pour en parler.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Bibliome/onto-relation-vizualisation/issues/18#issuecomment-812841899, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEHRRTVDRFAICFABZ4ZACL3TG3PVLANCNFSM4U4LPWPQ .