Fix v1.0.0 - Githubissues

SanderDevisscher commented 7 months ago

this is still WIP I'll convert it into draft untill completed

mvarewyck commented 6 months ago

For reference: When this PR is merged, it's good to remove following files in S3 bucket (replaced by parquet files in the data creation pipeline)

data_input_checklist_indicators_processed.RData
eu_concern_species_processed.RData
be_alientaxa_cube_processed.RData
full_timeseries.RData

SanderDevisscher commented 6 months ago

@mvarewyck after changing the R - version to one cope with the infrastructure switch to ARM64 I get the following error in step 9:

 > [ 9/10] RUN R -q -e "options(warn = 2); remotes::install_local('/tmp/package', dependencies=FALSE)":
#13 3.278 ** byte-compile and prepare package for lazy loading
#13 3.786 ERROR: lazy loading failed for package ‘alienSpecies’
#13 3.786 * removing ‘/usr/local/lib/R/site-library/alienSpecies’
#13 3.786 Error in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]) : 
#13 3.786   namespace ‘leaflet’ 2.1.2 is being loaded, but >= 2.2.0 is required
#13 3.786 Calls: <Anonymous> ... withCallingHandlers -> loadNamespace -> namespaceImport -> loadNamespace
#13 3.786 Execution halted
#13 3.797 Error: Failed to install 'alienSpecies' from local:
#13 3.797   (converted from warning) installation of package ‘/tmp/RtmpwMNCDE/file77451be4c/alienSpecies_1.0.0.tar.gz’ had non-zero exit status
#13 3.797 Execution halted

while @TheJenne18 gets the following error when building on the UAT environment:

482.9 checking whether the compiler supports GNU C... yes
482.9 checking whether gcc accepts -g... yes
483.0 checking for gcc option to enable C11 features... none needed
483.0 checking for XML_ParserCreate in -lexpat... yes
483.0 checking for udunits2.h... no
483.0 checking for udunits2/udunits2.h... no
483.1 checking for ut_read_xml in -ludunits2... no
483.1 configure: error: in `/tmp/RtmpDJzrfz/R.INSTALL7f91490ead21/units':
483.1 configure: error: 
483.1 --------------------------------------------------------------------------------
483.1   Configuration failed because libudunits2.so was not found. Try installing:
483.1     * deb: libudunits2-dev (Debian, Ubuntu, ...)
483.1     * rpm: udunits2-devel (Fedora, EPEL, ...)
483.1     * brew: udunits (OSX)
483.1   If udunits2 is already installed in a non-standard location, use:
483.1     --configure-args='--with-udunits2-lib=/usr/local/lib'
483.1   if the library was not found, and/or:
483.1     --configure-args='--with-udunits2-include=/usr/include/udunits2'
483.1   if the header was not found, replacing paths with appropriate values.
483.1   You can alternatively set UDUNITS2_INCLUDE and UDUNITS2_LIBS manually.
483.1 --------------------------------------------------------------------------------
483.1 
483.1 See `config.log' for more details
483.1 ERROR: configuration failed for package ‘units’
483.1 * removing ‘/usr/local/lib/R/site-library/units’
483.1 Error: Failed to install 'ggspatial' from CRAN:
483.1   (converted from warning) installation of package ‘units’ had non-zero exit status
483.1 Execution halted
------
Dockerfile:22
--------------------
  20 |     RUN R -q -e "install.packages('remotes')"
  21 |     
  22 | >>> RUN R -q -e "options(warn = 2); remotes::install_cran(c('shiny', 'data.table', 'dplyr', 'DT', 'ggplot2', 'ggspatial', 'htmlwidgets', 'httr', 'jsonlite', 'leaflet', 'leaflet.extras', 'plotly', 'reshape2', 'rgbif', 'sf', 'shinyjs', 'terra', 'testthat', 'tidyr', 'tidyverse', 'webshot', 'xtable'))"
  23 |     
  24 |     # Specific data format + access to S3 on UAT
--------------------
ERROR: failed to solve: process "/bin/sh -c R -q -e \"options(warn = 2); remotes::install_cran(c('shiny', 'data.table', 'dplyr', 'DT', 'ggplot2', 'ggspatial', 'htmlwidgets', 'httr', 'jsonlite', 'leaflet', 'leaflet.extras', 'plotly', 'reshape2', 'rgbif', 'sf', 'shinyjs', 'terra', 'testthat', 'tidyr', 'tidyverse', 'webshot', 'xtable'))\"" did not complete successfully: exit code: 1

mvarewyck commented 6 months ago

@mvarewyck after changing the R - version to one cope with the infrastructure switch to ARM64 I get the following error in step 9:

@SanderDevisscher which base docker image (&R version) did you use then? The DESCRIPTION file indeed requests leaflet > 2.2.0 while I expect lower R versions will not install this. I would prefer to switch to another repository for the base R image that supports your ARM64 e.g. https://hub.docker.com/_/r-base

I'm also building the image locally to reproduce the error on UAT.

mvarewyck commented 6 months ago

while @TheJenne18 gets the following error when building on the UAT environment:

@TheJenne18 Could you try to build the image without cache (--no-cache) once more? For me the image is builds without issues and from the error you get it seems not to install libudunits correctly while it is listed explicitly in the Dockerfile. Did you change anything to the Dockerfile?

mvarewyck commented 6 months ago

@TheJenne18 @SanderDevisscher You can pull my docker image here: docker pull registry.openanalytics.eu/inbo/alienspecies:1.0.0

mvarewyck commented 6 months ago

@mvarewyck first the good. The startup is significantly faster! The styling is much approved! However there are still some unsolved issues.

The year of last observation is not overwritten with the year of the latest observation. for example Lithobates catesbeianus is supposedly last observed in 2020 while in fact the latest observation in the data is 2023.

@SanderDevisscher It's because I changed the code but not yet applied it to the data. I have now applied it to the parquet file on S3 and seems ok. It's best to run the data pipeline on aspbo again using the latest version of the code, then you can actually test whether the new version of createTabularData() overwrites the year correctly given the input data.

mvarewyck commented 6 months ago

@mvarewyck after changing the R - version to one cope with the infrastructure switch to ARM64 I get the following error in step 9:

@SanderDevisscher which base docker image (&R version) did you use then? The DESCRIPTION file indeed requests leaflet > 2.2.0 while I expect lower R versions will not install this. I would prefer to switch to another repository for the base R image that supports your ARM64 e.g. https://hub.docker.com/_/r-base

I'm also building the image locally to reproduce the error on UAT.

@SanderDevisscher Sorry, I didn't see your commit. Changing base image to later version of R indeed fixed the issue.

TheJenne18 commented 6 months ago

@mvarewyck with the latest commit I still receive this error message when building on ARM64:

457.0 checking for error_at_line... yes
457.0 checking for gcc... gcc
457.0 checking whether the compiler supports GNU C... yes
457.0 checking whether gcc accepts -g... yes
457.0 checking for gcc option to enable C11 features... none needed
457.1 checking for XML_ParserCreate in -lexpat... yes
457.1 checking for udunits2.h... no
457.1 checking for udunits2/udunits2.h... no
457.1 checking for ut_read_xml in -ludunits2... no
457.1 configure: error: in `/tmp/RtmpHeGNGz/R.INSTALL822b4d39c0c4/units':
457.1 configure: error: 
457.1 --------------------------------------------------------------------------------
457.1   Configuration failed because libudunits2.so was not found. Try installing:
457.1     * deb: libudunits2-dev (Debian, Ubuntu, ...)
457.1     * rpm: udunits2-devel (Fedora, EPEL, ...)
457.1     * brew: udunits (OSX)
457.1   If udunits2 is already installed in a non-standard location, use:
457.1     --configure-args='--with-udunits2-lib=/usr/local/lib'
457.1   if the library was not found, and/or:
457.1     --configure-args='--with-udunits2-include=/usr/include/udunits2'
457.1   if the header was not found, replacing paths with appropriate values.
457.1   You can alternatively set UDUNITS2_INCLUDE and UDUNITS2_LIBS manually.
457.1 --------------------------------------------------------------------------------
457.1 
457.1 See `config.log' for more details
457.1 ERROR: configuration failed for package ‘units’
457.1 * removing ‘/usr/local/lib/R/site-library/units’
457.2 Error: Failed to install 'ggspatial' from CRAN:
457.2   (converted from warning) installation of package ‘units’ had non-zero exit status
457.2 Execution halted
------
Dockerfile:22
--------------------
  20 |     RUN R -q -e "install.packages('remotes')"
  21 |     
  22 | >>> RUN R -q -e "options(warn = 2); remotes::install_cran(c('shiny', 'data.table', 'dplyr', 'DT', 'ggplot2', 'ggspatial', 'htmlwidgets', 'httr', 'jsonlite', 'leaflet', 'leaflet.extras', 'plotly', 'reshape2', 'rgbif', 'sf', 'shinyjs', 'terra', 'testthat', 'tidyr', 'tidyverse', 'webshot', 'xtable'))"
  23 |     
  24 |     # Specific data format + access to S3 on UAT
--------------------
ERROR: failed to solve: process "/bin/sh -c R -q -e \"options(warn = 2); remotes::install_cran(c('shiny', 'data.table', 'dplyr', 'DT', 'ggplot2', 'ggspatial', 'htmlwidgets', 'httr', 'jsonlite', 'leaflet', 'leaflet.extras', 'plotly', 'reshape2', 'rgbif', 'sf', 'shinyjs', 'terra', 'testthat', 'tidyr', 'tidyverse', 'webshot', 'xtable'))\"" did not complete successfully: exit code: 1

mvarewyck commented 5 months ago

@TheJenne18 It should work now on ARM64 too.

SanderDevisscher commented 5 months ago

@TheJenne18 It should work now on ARM64 too.

@TheJenne18 can you confirm ?

TheJenne18 commented 5 months ago

The build was successful but we see the following error when launching on ECS by the shiny proxy:


April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:31.434Z INFO 1 --- [ XNIO-1 task-8] e.o.containerproxy.service.UserService : User logged in [user: 5560f92c-124c-4b29-b6ac-95a01ce20656] | exotenportaal-shinyproxy
-- | -- | --
April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:30.838Z INFO 1 --- [ XNIO-1 task-8] e.o.containerproxy.service.UserService : User logged in [user: 2e3e2733-9a29-4802-a3df-f641699be31d] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:30.346Z WARN 1 --- [ProxyService-16] e.o.shinyproxy.ShinyProxyTestStrategy : [user=3020848c-2535-499b-814e-e4d1e4ffbaba proxyId=7abd156f-eadb-4807-9722-f4df7eef78f1 specId=01_exotenportaal] Container unresponsive, trying again (250/310): http://10.30.20.95:3838 | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:30.312Z INFO 1 --- [ XNIO-1 task-8] e.o.containerproxy.service.UserService : User logged in [user: eb373ade-92bf-44b3-9659-fd82251c931a] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:29.835Z INFO 1 --- [ XNIO-1 task-8] e.o.containerproxy.service.UserService : User logged in [user: 3c92b746-036f-462d-a66d-61bd0e624709] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:29.267Z INFO 1 --- [ XNIO-1 task-8] e.o.containerproxy.service.UserService : User logged in [user: 1c4651b5-f992-4f69-87ef-fc364651eeb0] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:28.762Z INFO 1 --- [ XNIO-1 task-8] e.o.containerproxy.service.UserService : User logged in [user: 17792a64-0ae2-4396-a68e-1469445f19d3] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:28.108Z ERROR 1 --- [ XNIO-1 task-8] io.undertow.request : UT005023: Exception handling request to /app/config/parameters.yml | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | jakarta.servlet.ServletException: Request processing failed: java.lang.NullPointerException: Cannot invoke "eu.openanalytics.containerproxy.model.spec.ProxySpec.getContainerSpecs()" because "spec" is null | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1022) ~[spring-webmvc-6.1.3.jar!/:6.1.3] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903) ~[spring-webmvc-6.1.3.jar!/:6.1.3] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:527) ~[jakarta.servlet-api-6.0.0.jar!/:6.0.0] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) ~[spring-webmvc-6.1.3.jar!/:6.1.3] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614) ~[jakarta.servlet-api-6.0.0.jar!/:6.0.0] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) ~[undertow-servlet-2.3.10.Final.jar!/:2.3.10.Final] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) ~[undertow-servlet-2.3.10.Final.jar!/:2.3.10.Final] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at eu.openanalytics.containerproxy.util.AppRecoveryFilter.doFilter(AppRecoveryFilter.java:63) ~[containerproxy-1.1.0-SNAPSHOT.jar!/:1.1.0-SNAPSHOT] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67) ~[undertow-servlet-2.3.10.Final.jar!/:2.3.10.Final] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-2.3.10.Final.jar!/:2.3.10.Final] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:108) ~[spring-web-6.1.3.jar!/:6.1.3] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231) ~[spring-security-web-6.2.1.jar!/:6.2.1] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.security.web.ObservationFilterChainDecorator$FilterObservation$SimpleFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:479) ~[spring-security-web-6.2.1.jar!/:6.2.1] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:340) ~[spring-security-web-6.2.1.jar!/:6.2.1] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.security.web.ObservationFilterChainDecorator.lambda$wrapSecured$0(ObservationFilterChainDecorator.java:82) ~[spring-security-web-6.2.1.jar!/:6.2.1] | exotenportaal-shinyproxy

mvarewyck commented 5 months ago

GAMS should be applied even without selecting any radio buttons. In the current version the GAM graph only appears if both radio buttons are selected. Personally I would like it if a button is provide to "apply the GAM" which overwrites the simple black dot observation/occupancy graph.

Indeed I found out last week there is a conflict between config::get and base::get. I'll push a fix but for me this doesn't occur via docker, e.g. for Abies Alba I get

Screenshot from 2024-04-22 10-43-16

SanderDevisscher commented 5 months ago

Looks better

mvarewyck commented 5 months ago

I do get some warnings stating in my docker log: 2024-04-18 16:40:04 Can't detect correct thread for auto_deleter_background. 2024-04-18 16:40:04 Can't detect correct thread for auto_deleter_main.

I don't see them, but they seem to come from here: https://github.com/rstudio/httpuv/blob/main/src/auto_deleter.h#L30

mvarewyck commented 5 months ago

The GAMs mention there is no data for Tribulus terrestris while there are at least 3 observations in the data. This seems to be a bug.

According to the data processing script this is a species introduced before 1950 and therefore excluded from the timeseries data as datasource for the GAM models.

I agree this can be confusing. So, I suggest we either

update the data processing script such that same filtering is applied to both data sources or
we clearly state the difference between both data sources in some disclaimer in the app and disable the 'Indicators' tab in this case.

@SanderDevisscher What is the preferred behavior from your side?

mvarewyck commented 5 months ago

The build was successful but we see the following error when launching on ECS by the shiny proxy:


April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:31.434Z INFO 1 --- [ XNIO-1 task-8] e.o.containerproxy.service.UserService : User logged in [user: 5560f92c-124c-4b29-b6ac-95a01ce20656] | exotenportaal-shinyproxy
-- | -- | --
April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:30.838Z INFO 1 --- [ XNIO-1 task-8] e.o.containerproxy.service.UserService : User logged in [user: 2e3e2733-9a29-4802-a3df-f641699be31d] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:30.346Z WARN 1 --- [ProxyService-16] e.o.shinyproxy.ShinyProxyTestStrategy : [user=3020848c-2535-499b-814e-e4d1e4ffbaba proxyId=7abd156f-eadb-4807-9722-f4df7eef78f1 specId=01_exotenportaal] Container unresponsive, trying again (250/310): http://10.30.20.95:3838 | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:30.312Z INFO 1 --- [ XNIO-1 task-8] e.o.containerproxy.service.UserService : User logged in [user: eb373ade-92bf-44b3-9659-fd82251c931a] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:29.835Z INFO 1 --- [ XNIO-1 task-8] e.o.containerproxy.service.UserService : User logged in [user: 3c92b746-036f-462d-a66d-61bd0e624709] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:29.267Z INFO 1 --- [ XNIO-1 task-8] e.o.containerproxy.service.UserService : User logged in [user: 1c4651b5-f992-4f69-87ef-fc364651eeb0] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:28.762Z INFO 1 --- [ XNIO-1 task-8] e.o.containerproxy.service.UserService : User logged in [user: 17792a64-0ae2-4396-a68e-1469445f19d3] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:28.108Z ERROR 1 --- [ XNIO-1 task-8] io.undertow.request : UT005023: Exception handling request to /app/config/parameters.yml | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | jakarta.servlet.ServletException: Request processing failed: java.lang.NullPointerException: Cannot invoke "eu.openanalytics.containerproxy.model.spec.ProxySpec.getContainerSpecs()" because "spec" is null | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1022) ~[spring-webmvc-6.1.3.jar!/:6.1.3] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903) ~[spring-webmvc-6.1.3.jar!/:6.1.3] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:527) ~[jakarta.servlet-api-6.0.0.jar!/:6.0.0] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) ~[spring-webmvc-6.1.3.jar!/:6.1.3] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614) ~[jakarta.servlet-api-6.0.0.jar!/:6.0.0] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) ~[undertow-servlet-2.3.10.Final.jar!/:2.3.10.Final] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) ~[undertow-servlet-2.3.10.Final.jar!/:2.3.10.Final] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at eu.openanalytics.containerproxy.util.AppRecoveryFilter.doFilter(AppRecoveryFilter.java:63) ~[containerproxy-1.1.0-SNAPSHOT.jar!/:1.1.0-SNAPSHOT] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67) ~[undertow-servlet-2.3.10.Final.jar!/:2.3.10.Final] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-2.3.10.Final.jar!/:2.3.10.Final] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:108) ~[spring-web-6.1.3.jar!/:6.1.3] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231) ~[spring-security-web-6.2.1.jar!/:6.2.1] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.security.web.ObservationFilterChainDecorator$FilterObservation$SimpleFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:479) ~[spring-security-web-6.2.1.jar!/:6.2.1] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:340) ~[spring-security-web-6.2.1.jar!/:6.2.1] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.security.web.ObservationFilterChainDecorator.lambda$wrapSecured$0(ObservationFilterChainDecorator.java:82) ~[spring-security-web-6.2.1.jar!/:6.2.1] | exotenportaal-shinyproxy

These logs show a correct startup of the shinyproxy. @TheJenne18 Can you send the Amazon logs because we think there sth is failing?

TheJenne18 commented 5 months ago

The build was successful but we see the following error when launching on ECS by the shiny proxy:


April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:31.434Z INFO 1 --- [ XNIO-1 task-8] e.o.containerproxy.service.UserService : User logged in [user: 5560f92c-124c-4b29-b6ac-95a01ce20656] | exotenportaal-shinyproxy
-- | -- | --
April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:30.838Z INFO 1 --- [ XNIO-1 task-8] e.o.containerproxy.service.UserService : User logged in [user: 2e3e2733-9a29-4802-a3df-f641699be31d] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:30.346Z WARN 1 --- [ProxyService-16] e.o.shinyproxy.ShinyProxyTestStrategy : [user=3020848c-2535-499b-814e-e4d1e4ffbaba proxyId=7abd156f-eadb-4807-9722-f4df7eef78f1 specId=01_exotenportaal] Container unresponsive, trying again (250/310): http://10.30.20.95:3838 | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:30.312Z INFO 1 --- [ XNIO-1 task-8] e.o.containerproxy.service.UserService : User logged in [user: eb373ade-92bf-44b3-9659-fd82251c931a] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:29.835Z INFO 1 --- [ XNIO-1 task-8] e.o.containerproxy.service.UserService : User logged in [user: 3c92b746-036f-462d-a66d-61bd0e624709] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:29.267Z INFO 1 --- [ XNIO-1 task-8] e.o.containerproxy.service.UserService : User logged in [user: 1c4651b5-f992-4f69-87ef-fc364651eeb0] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:28.762Z INFO 1 --- [ XNIO-1 task-8] e.o.containerproxy.service.UserService : User logged in [user: 17792a64-0ae2-4396-a68e-1469445f19d3] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | 2024-04-22T07:49:28.108Z ERROR 1 --- [ XNIO-1 task-8] io.undertow.request : UT005023: Exception handling request to /app/config/parameters.yml | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | jakarta.servlet.ServletException: Request processing failed: java.lang.NullPointerException: Cannot invoke "eu.openanalytics.containerproxy.model.spec.ProxySpec.getContainerSpecs()" because "spec" is null | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1022) ~[spring-webmvc-6.1.3.jar!/:6.1.3] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903) ~[spring-webmvc-6.1.3.jar!/:6.1.3] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:527) ~[jakarta.servlet-api-6.0.0.jar!/:6.0.0] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) ~[spring-webmvc-6.1.3.jar!/:6.1.3] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614) ~[jakarta.servlet-api-6.0.0.jar!/:6.0.0] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) ~[undertow-servlet-2.3.10.Final.jar!/:2.3.10.Final] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) ~[undertow-servlet-2.3.10.Final.jar!/:2.3.10.Final] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at eu.openanalytics.containerproxy.util.AppRecoveryFilter.doFilter(AppRecoveryFilter.java:63) ~[containerproxy-1.1.0-SNAPSHOT.jar!/:1.1.0-SNAPSHOT] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67) ~[undertow-servlet-2.3.10.Final.jar!/:2.3.10.Final] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-2.3.10.Final.jar!/:2.3.10.Final] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:108) ~[spring-web-6.1.3.jar!/:6.1.3] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231) ~[spring-security-web-6.2.1.jar!/:6.2.1] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.security.web.ObservationFilterChainDecorator$FilterObservation$SimpleFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:479) ~[spring-security-web-6.2.1.jar!/:6.2.1] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:340) ~[spring-security-web-6.2.1.jar!/:6.2.1] | exotenportaal-shinyproxy
April 22, 2024 at 09:49 (UTC+2:00) | at org.springframework.security.web.ObservationFilterChainDecorator.lambda$wrapSecured$0(ObservationFilterChainDecorator.java:82) ~[spring-security-web-6.2.1.jar!/:6.2.1] | exotenportaal-shinyproxy

These logs show a correct startup of the shinyproxy. @TheJenne18 Can you send the Amazon logs because we think there sth is failing?

2024-04-23T07:23:43.637+02:00 | R version 4.3.2 (2023-10-31) -- "Eye Holes"
-- | --
  | 2024-04-23T07:23:43.637+02:00 | Copyright (C) 2023 The R Foundation for Statistical Computing
  | 2024-04-23T07:23:43.637+02:00 | Platform: aarch64-unknown-linux-gnu (64-bit)
  | 2024-04-23T07:23:43.637+02:00 | R is free software and comes with ABSOLUTELY NO WARRANTY.
  | 2024-04-23T07:23:43.637+02:00 | You are welcome to redistribute it under certain conditions.
  | 2024-04-23T07:23:43.637+02:00 | Type 'license()' or 'licence()' for distribution details.
  | 2024-04-23T07:23:43.637+02:00 | Natural language support but running in an English locale
  | 2024-04-23T07:23:43.637+02:00 | R is a collaborative project with many contributors.
  | 2024-04-23T07:23:43.637+02:00 | Type 'contributors()' for more information and
  | 2024-04-23T07:23:43.637+02:00 | 'citation()' on how to cite R or R packages in publications.
  | 2024-04-23T07:23:43.637+02:00 | Type 'demo()' for some demos, 'help()' for on-line help, or
  | 2024-04-23T07:23:43.637+02:00 | 'help.start()' for an HTML browser interface to help.
  | 2024-04-23T07:23:43.637+02:00 | Type 'q()' to quit R.
  | 2024-04-23T07:23:49.431+02:00 | During startup - Warning messages:
  | 2024-04-23T07:23:49.431+02:00 | 1: replacing previous import ‘ggplot2::last_plot’ by ‘plotly::last_plot’ when loading ‘alienSpecies’
  | 2024-04-23T07:23:49.431+02:00 | 2: replacing previous import ‘data.table::dcast’ by ‘reshape2::dcast’ when loading ‘alienSpecies’
  | 2024-04-23T07:23:49.431+02:00 | 3: replacing previous import ‘data.table::melt’ by ‘reshape2::melt’ when loading ‘alienSpecies’
  | 2024-04-23T07:23:49.434+02:00 | > alienSpecies::runShiny()
  | 2024-04-23T07:23:49.436+02:00 | Loading required package: shiny
  | 2024-04-23T07:23:49.467+02:00 | Error: NotImplemented: Got S3 URI but Arrow compiled without S3 support
  | 2024-04-23T07:23:49.471+02:00 | Execution halted

mvarewyck commented 5 months ago

Thanks @TheJenne18 The application is failing to startup. Something is wrong with loading the parquet files from S3. It's hard for me to debug without access to the server, so I've implemented a fix following this issue Can you try to deploy my latest push?

mvarewyck commented 5 months ago

GAMS should be applied even without selecting any radio buttons. In the current version the GAM graph only appears if both radio buttons are selected. Personally I would like it if a button is provide to "apply the GAM" which overwrites the simple black dot observation/occupancy graph.

With the latest trias (2.1.0) there was a dependency issue for gratia >= 0.9.0 required. This should be fixed now.

mvarewyck commented 5 months ago

I found maps were not rendered correctly in the pdf report, implemented this fix

SanderDevisscher commented 4 months ago

The deployment action fails on the arrow install

SanderDevisscher commented 4 months ago

https://github.com/inbo/alien-species-portal/actions/runs/9109152850

SanderDevisscher commented 4 months ago

Both upload actions in aspbo fail. Upload_direct fails on the tests, see https://github.com/inbo/aspbo/actions/runs/9109126752 While upload_processing fails on the alienspeciesportal installation, see https://github.com/inbo/aspbo/actions/runs/9109130261 Something about loading trias 2.0.6. while 2.0.8. is needed

mvarewyck commented 4 months ago

https://github.com/inbo/alien-species-portal/actions/runs/9109152850

Could it be due memory issues? https://github.com/actions/runner-images/issues/6680

SanderDevisscher commented 4 months ago

https://github.com/inbo/alien-species-portal/actions/runs/9109152850

Could it be due memory issues? actions/runner-images#6680

Ifso what can we do ? just retry ? raise an issue with the arrow - package ?

mvarewyck commented 4 months ago

Both upload actions in aspbo fail. Upload_direct fails on the tests, see https://github.com/inbo/aspbo/actions/runs/9109126752

I think the problem is that the data check was done on an old extract of the DB. Though for UAT it should be fine, because the data were already there. I've pushed a fix in aspbo for this.

mvarewyck commented 4 months ago

While upload_processing fails on the alienspeciesportal installation, see https://github.com/inbo/aspbo/actions/runs/9109130261 Something about loading trias 2.0.6. while 2.0.8. is needed

We'll have to wait for a new release of trias I requested I might have to update the release tag

mvarewyck commented 4 months ago

Ifso what can we do ? just retry ? raise an issue with the arrow - package ?

It might be due to memory issues. It's a wild guess but we could try to restrict memory usage. I've pushed a fix for this. If this is not working, we could try to use a github action runner with more memory, if you have a github subscription?

On a side note: We're trying to install binary R packages (compiled) as they require less memory. However, by using arm instead of intel this effort is lost, because no compiled versions are available for arm. So, moving to intel could also resolve this issue.

mvarewyck commented 4 months ago

@SanderDevisscher all good for a new try! :crossed_fingers:

SanderDevisscher commented 4 months ago

new deploy initiated: https://github.com/inbo/alien-species-portal/actions/runs/9175810680

mvarewyck commented 4 months ago

new deploy initiated: https://github.com/inbo/alien-species-portal/actions/runs/9175810680

I saw the job failed again. As the job is self-hosted, could you provide more ram to the server where this process is running?

SanderDevisscher commented 4 months ago

@TheJenne18 is it possible to, at least for a test, increase the RAM for this runner ?

SanderDevisscher commented 4 months ago

Upload files direct works like it should. Upload files processing still fails with this error:

Error: Error: IOError: When reading information for key 'data_input_checklist_indicators_processed.parquet' in bucket 'inbo-exotenportaal-uat-***-default': AWS Error ACCESS_DENIED during HeadObject operation: No response body.
In addition: Warning messages:
1: replacing previous import ‘ggplot2::last_plot’ by ‘plotly::last_plot’ when loading ‘alienSpecies’ 
2: replacing previous import ‘data.table::dcast’ by ‘reshape2::dcast’ when loading ‘alienSpecies’ 
3: replacing previous import ‘data.table::melt’ by ‘reshape2::melt’ when loading ‘alienSpecies’ 
Execution halted
Error: Process completed with exit code 1.

see https://github.com/inbo/aspbo/actions/runs/9175867402

mvarewyck commented 4 months ago

Upload files processing still fails with this error:

@SanderDevisscher Should be fixed now.

SanderDevisscher commented 4 months ago

retrying https://github.com/inbo/aspbo/actions/runs/9188396008

TheJenne18 commented 4 months ago

@SanderDevisscher I've restarted the job, don't look like a memory problem

SanderDevisscher commented 4 months ago

retrying https://github.com/inbo/aspbo/actions/runs/9188396008

Failed again but with a different error:

Error in parse_aws_s3_response(r, Sig, verbose = verbose) : 
  Not Found (HTTP 404).
Calls: source ... save_object -> s3HTTP -> parse_aws_s3_response -> <Anonymous>
In addition: Warning messages:
1: replacing previous import ‘ggplot2::last_plot’ by ‘plotly::last_plot’ when loading ‘alienSpecies’ 
2: replacing previous import ‘data.table::dcast’ by ‘reshape2::dcast’ when loading ‘alienSpecies’ 
3: replacing previous import ‘data.table::melt’ by ‘reshape2::melt’ when loading ‘alienSpecies’ 
Execution halted
Error: Process completed with exit code 1.

mvarewyck commented 4 months ago

@SanderDevisscher I've restarted the job, don't look like a memory problem

@TheJenne18 We have tested locally with limited memory resources. Using 1GB it fails, using 2GB it fails after a while, using 3GB it doesn't fail. Can you please confirm at least 3GB is available for the workflow?

mvarewyck commented 4 months ago

Managed nests data for Vespa velutina seems to be missing from Active hotbeds & Management of Vespa velutina what goes wrong ?

@SanderDevisscher Sure about adding managed nests for Active hotbeds? I had to drop them since here This is a minor fix to implement.

the issue with GAM graphs not loading by default remains

clicking create report results in a timeout

I can reproduce this at exoten-uat.inbo.be. However, I have just built a docker image from scratch and there I can't reproduce. @TheJenne18 Can you copy me the sessionInfo() or the error logs? I have my docker image here docker pull registry.openanalytics.eu/inbo/alienspecies:1.0.0

mvarewyck commented 4 months ago

Managed nests data for Vespa velutina seems to be missing from Active hotbeds & Management of Vespa velutina what goes wrong ?

The managed nests data are there but only picked up at province level. Reported this issue at inbo/aspbo#170 Screenshot from 2024-05-29 12-28-32

mvarewyck commented 4 months ago

Dropdown for source @ Management of Vespa velutina should be replaced by management status (untreated & managed)

if possible add an option to filter category to Management map in case of Vespa velutina this should be nesttype

Default value should be "all nesttypes".

@SanderDevisscher Are these filters fine?

Screenshot from 2024-05-29 15-10-52

SanderDevisscher commented 4 months ago

Dropdown for source @ Management of Vespa velutina should be replaced by management status (untreated & managed)

if possible add an option to filter category to Management map in case of Vespa velutina this should be nesttype

Default value should be "all nesttypes".

@SanderDevisscher Are these filters fine?

yes

SanderDevisscher commented 4 months ago

Managed nests data for Vespa velutina seems to be missing from Active hotbeds & Management of Vespa velutina what goes wrong ?

@SanderDevisscher Sure about adding managed nests for Active hotbeds? I had to drop them since here This is a minor fix to implement.

Ok leave it as is for now. W'll combine it with other changes to the management page @jrhillae we need to take a look at the current state of the Vespa velutina management page !

jrhillae commented 4 months ago

@SanderDevisscher: Yes, it is important for the active hotbeds to :

Understand the API of waarnemingen.be to download most recent sightings of individuals
Only include sightings of the two last weeks within the map.

mvarewyck commented 4 months ago

@TheJenne18 I think the version on exoten-uat.inbo.be is not including recent changes. One month ago I have implemented a change to pick up the split translation files, but still the online version seems not to incorporate this change.

SanderDevisscher commented 4 months ago

@TheJenne18 implementing the changes required for https://github.com/inbo/alien-species-portal/issues/98 would reduce the number of questions like https://github.com/inbo/alien-species-portal/pull/74#issuecomment-2139242570 😉

SanderDevisscher commented 4 months ago

@TheJenne18 I think the version on exoten-uat.inbo.be is not including recent changes. One month ago I have implemented a change to pick up the split translation files, but still the online version seems not to incorporate this change.

@mvarewyck the previous deployment originated at https://github.com/inbo/alien-species-portal/commit/acc5d115b5698a2e62865018219aae8a81f1e5fb Currently a deployment is running originating from https://github.com/inbo/alien-species-portal/commit/6d6fb68cc3de68c8ff3ebcc1640ad52d304d951b, see https://github.com/inbo/alien-species-portal/actions/runs/9300534300

TheJenne18 commented 4 months ago

@mvarewyck

@TheJenne18 I think the version on exoten-uat.inbo.be is not including recent changes. One month ago I have implemented a change to pick up the split translation files, but still the online version seems not to incorporate this change.

I've just fixed it: https://github.com/inbo/alien-species-portal/pull/74/commits/679a8300fdb5ae2a1fcae37c704b854cbf8b3285

SanderDevisscher commented 4 months ago

@mvarewyck

@TheJenne18 I think the version on exoten-uat.inbo.be is not including recent changes. One month ago I have implemented a change to pick up the split translation files, but still the online version seems not to incorporate this change.

I've just fixed it: 679a830

ok redeploying

inbo / alien-species-portal

Fix v1.0.0 #74