search

Hirlam / obsmon

Tool for observation monitoring in the Harmonie-Arome NWP system
https://hirlam.github.io/obsmon/
MIT License
3 stars 8 forks source link

No plots produced, query fails #26

Closed khintz closed 2 years ago

khintz commented 2 years ago

I have installed obsmon v. 4.3.2 (e27e705) on a fresh server running ubuntu 22.04 with ansible.

Obsmon is running and I have some sqlite files holding both ecma and ccma, eg. on the path /obsmondata/dini/ecma/2022070512/ecma.db I run obsmon through a shiny-server. From these log files I see the error preformQuery: could not find function "partial". Full log file is below.

I guess that the error probably comes from src/sqlite/sqlite_wrappers.R but I don't know where the ´partial´ function is defined.

Any suggestions will be very much appreciated.

I had to update multiple packages for the obsmon installation to succeed:

-cpp11 == 0.2.7
+cpp11 == 0.4.2
-isoband == 0.2.4
+isoband == 0.2.5
-lobstr == 1.1.1
+lobstr == 1.1.2
-mapproj == 1.2.7
+mapproj == 1.2.8
-maps == 3.3.0
+maps == 3.4.0
-pryr == 0.1.4
+pryr == 0.1.5
-rlang == 0.4.10
+rlang == 1.0.3
-stringi == 1.7.3
+stringi == 1.7.6
-testthat == 3.0.2
+testthat == 3.1.4
-units == 0.7-1
+units == 0.8-0
-V8 == 3.4.0
+V8 == 4.2.0

These are the debian packages I installed before installing R-4.2.1, Shiny-server and Obsmon:

The config file looks as follows:

[general]
    configName = "UWC-West Obsmon" # Optional. If set, will appear in the GUI and logs.
    logLevel = "DEBUG"
    cacheDir = ".Rcache"
    sessionTimeout = 1800 # In seconds. Optional.
    appTimeout = 5 # In seconds. Optional.
    multiPlotsEnableInteractivity = true # The default value is false

[domain]
    nlon = 90
    nlat = 96
    lonc = 16.763011639
    latc = 63.489212956
    lon0 = 15.0
    lat0 = 63.0
    gsize = 25000
    lmrt = false

[[experiments]]
    displayName = "DINI"
    path = "/obsmondata/dini/"

[[experiments]]
    displayName = "IG"
    path = "/obsmondata/ig/"

[[multiPlots]]
    displayName = "DINI: ecma NObs for all obs but satem"
    experiment = "DINI"
    plotType = "Number of Observations"
    database = "ecma"
    startDate = -30
    excludeObs = ["satem"]
    [multiPlots.levels.aircraft]
        t = 100000
        u = [27500, 45000]
    [multiPlots.levels]
        synop = 0
        pilot = [1500, 4000, 8500, 45000]
    [multiPlots.excludeLevels]
      pilot = [8500, 100000]
    [multiPlots.excludeLevels.aircraft]
      v = [2500, 8500, 60000, 1500]
    [multiPlots.batchMode]
        parentDir = "batch_mode_plots"

[[multiPlots]]
    displayName = "IG"
    experiment = "IG"
    plotType = "Number of Observations"
    database = "ecma_sfc"
    startDate = -30
    [multiPlots.obs]
        synop=["t2m","rh2m", "snow"]
        aircraft=["t"]
    batchMode = true

[[multiPlots]]
    displayName = "A multiPlot with station choices"
    experiment = "IG"
    plotType = "Observation Usage"
    database = "ecma"
    startDate = "2022-05-19"
    endDate = "2022-05-19"
    cycles = "00"
    [multiPlots.stations]
        synop = ["01158", "01226"]
        aircraft = ["2BTQVXZA"]

[[multiPlots]]
    displayName = "A multiPlot with satem observations"
    experiment = "DINI"
    plotType = "First Guess and Analysis Departure"
    database = "ecma"
    startDate = "2022-05-19"
    nDays = 1 # To include only the startDate
    cycles = ["00", "09"] # Specifying multiple cycles
    [multiPlots.obs]
        synop=["z"]
    [[multiPlots.obs.satem]]
        sensor="amsua"
        satellite="metop1"
        channels=[1, 3, 5, 7]
    [[multiPlots.obs.satem]]
        sensor="iasi"
        satellite="metop2"
        channels=[38, 167, 236]

[[multiPlots]]
    displayName = "A multiPlot with more specific obs removed"
    experiment = "IG"
    plotType = "Observation Usage"
    database = "ecma"
    startDate = "2018-11-19"
    cycles = "00"
    [multiPlots.excludeObs]
        ship=["apd", "snow"]
        aircraft=["all"]
        [[multiPlots.excludeObs.satem]]
            sensor="amsua"
            satellite="metop1"
            channels=[1, 3, 5, 7]
        [[multiPlots.excludeObs.satem]]
            sensor="amsub"
            satellite="noaa16"

The shiny-server log file:

==================================================
   ____  _                               
  / __ \| |                              
 | |  | | |__  ___ _ __ ___   ___  _ __  
 | |  | | '_ \/ __| '_ ` _ \ / _ \| '_ \ 
 | |__| | |_) \__ \ | | | | | (_) | | | |
  \____/|_.__/|___/_| |_| |_|\___/|_| |_|

                  v4.3.2

Branch: Unknown
Tag: Unknown
Commit: Unknown
Commit date: Unknown
==================================================

Warning: replacing previous import 'lifecycle::last_warnings' by 'rlang::last_warnings' when loading 'tibble'
Warning: replacing previous import 'lifecycle::last_warnings' by 'rlang::last_warnings' when loading 'pillar'
Warning in createCacheDir(config, verbose = TRUE) :
  Cannot write to cacheDir /srv/checkout/.Rcache/obsmon_v4.3  !!!!!
 Please specify a valid cacheDir value under the
 "[general]" section in your config file.
 >> Caching will not work!!!!

DEBUG [2022-07-07 07:29:11] Temp dir for session: /tmp/RtmpFE9fBV

INFO [2022-07-07 07:29:11] Config name found: "UWC-West Obsmon"

DEBUG [2022-07-07 07:29:12] Config file contains user-defined multiPlots. Validating.
DEBUG [2022-07-07 07:29:12] Finished validation of user-defined multiPlots.

Listening on http://127.0.0.1:41723
INFO [2022-07-07 07:29:13] New session started. Session token: e2fe3e13e2aa772fccca35d9234314fd
DEBUG [2022-07-07 07:29:13] Error: invalid assignment for reference class field 'center_lonlat', should be from class "numeric" or a subclass (was class "logical")

DEBUG [2022-07-07 07:29:15] Getting ecma DTGs for DINI
DEBUG [2022-07-07 07:29:15] Done getting ecma DTGs for DINI

DEBUG [2022-07-07 07:29:17] Killing jobs with PID(s) 741713
ERROR [2022-07-07 07:29:21] preformQuery: could not find function "partial"
DEBUG [2022-07-07 07:29:21] preformQuery: Split 1 queries into 1 processes.
   > Total elapsed time: 0.009 sec. Avg time per query: 0.009 sec.
DEBUG [2022-07-07 07:29:22] Killing jobs with PID(s) 741716
DEBUG [2022-07-07 07:29:22] Killing jobs with PID(s) 741719
ewhelan commented 2 years ago

Hi @khintz

I think the partial function comes with purrr which is a dependency of dplyr - I think. On my PC (CentOS8, R==4.0) I can see both installed in /home/ewhelan/R/x86_64-redhat-linux-gnu-library/4.0. Can you explore your ubuntu installation in the same way?

Eoin

khintz commented 2 years ago

I just added require("purrr") to sqlite/sqlite_wrappers.R and then I could produce a plot. So for some reason, obsmon have not loaded the required packages.

I have both dplyr and purrr installed under /usr/local/lib/R/site-library/ (which is also what I expected, as I am only using the install script in obsmon)

ewhelan commented 2 years ago

Great! Progress. Don't know enough about R or Obsmon to point the finger of blame here.

khintz commented 2 years ago

The packages are also in .installer_local_R-libs/R-libs/dplyr/ (see .Rprofile). For now, I add the require('purrr'), but waiting to hear if @paulovcmedeiros has any suggestions for a real fix.

paulovcmedeiros commented 2 years ago

Hi!

Before I start looking into this: Have you changed anything else in the code? The start banner shows a lot of unknowns, which indicates that you are not carrying over the .git directory when copying the code to its final destination. This is in principle not a problem but then it gets a bit difficult for me to know if there are modified or missing files.

Another strange thing is that in your log I don't see the line saying Looking for R-libs in the following directories (listed in order of priority): and the subsequent list of directories. Can you double-check this?

paulovcmedeiros commented 2 years ago

Hi again,

Can you try reinstalling obsmon using the version in the devel branch? I've made some changes to the install script. Btw, take a look at the new "Advanced install options" section of the PDF doc file. It should now be easier to install selected versions of R-packages, or install the latest versions of all packages from CRAN.

Let me know how it goes. I won't merge this into master until the install works correctly.

khintz commented 2 years ago

Hi @paulovcmedeiros,

The only things I changed was changing version of many packages, as reported in the first comment in this issue, and I had to add require("purrr") in src/sqlite/sqlite_wrappers.R The repo is cloned so everything is brought over, also .git and .Rcache.

Now I tried using your devel branch, and it worked out-of-the-box... No changes to the code at all from my side. I have tried loading data from an experiment and it works as expected (after creating a config.toml file)

For reference, I put my ansible pipeline related to obsmon below, so you can see all the steps I did on an Ubuntu 22.04 VM.

---
- name: Write ansible log
  shell: date "+{{ ansible_log_date_format }} - DMI helper scripts" >> "{{ ansible_log_file }}"
  changed_when: False

#####################
# Include vars
#####################
- name: Include role environment variables
  include_vars:
    dir: "vars"
    extensions:
      - "yml"

#####################
# Install and configure required packages
#####################

- name: Update apt cache and install dependencies
  apt:
    name:
      - build-essential
      - git
      - dirmngr
      - gnupg
      - apt-transport-https
      - ca-certificates
      - software-properties-common
      - gdebi-core
      - libopenblas-base
      - libudunits2-dev
      - libcairo2-dev
      - libxt-dev
      - libgdal-dev
      - libgeos-dev
      - libv8-dev
      - libgeos++-dev
      - libproj-dev
      - sqlite3
      - libsqlite3-dev
      - libharfbuzz-dev
      - libfribidi-dev
    state: latest
    update_cache: yes

- name: Install CRAN devtools dependencies
  apt:
    name:
      - libxml2-dev
      - libssl-dev
      - libcurl4-openssl-dev
    state: latest
    update_cache: yes

- name: Add key for CRAN-R repository
  apt_key:
    url: https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc
    state: present

- name: Add CRAN-R repository
  apt_repository:
    repo: "deb https://cloud.r-project.org/bin/linux/{{ ansible_distribution|lower }} {{ ansible_distribution_release }}-cran40/"
    state: present
    update_cache: yes

- name: Install r-base
  apt:
    name:
      - r-base #=4.2.1-1.2204.0
    state: present
    update_cache: yes

- name: Install the necessary CRAN devtools package
  shell: R -e "install.packages('devtools', repos='http://cran.rstudio.com/')"
  args:
    warn: yes

- name: Install shiny
  shell: R -e "require(devtools); install_version('shiny', version = '1.7.1', repos = 'http://cran.us.r-project.org')"
  args:
    warn: yes

- name: Get and install shiny-server
  shell: |
    wget https://download3.rstudio.org/ubuntu-18.04/x86_64/shiny-server-1.5.18.987-amd64.deb
    gdebi --non-interactive shiny-server-1.5.18.987-amd64.deb
    ufw allow 3838
  args:
    warn: yes

- name: Install rmarkdown
  shell: R -e "require(devtools); install_version('rmarkdown', version = '2.14', repos = 'http://cran.us.r-project.org')"
  args:
    warn: yes

- name: Install tidyverse required by obsmon
  shell: R -e "require(devtools); install_version('tidyverse', version = '1.3.1', repos = 'http://cran.us.r-project.org')"
  args:
    warn: yes

#####################
# Replace Nginx role default configuration
#####################

- name: Default webpage content
  copy:
    src: "var/www/default/{{ item }}"
    dest: "/var/www/{{ hostname }}.{{ domain }}/"
    owner: "root"
    group: "root"
  with_items:
    - 'index.html'
    - 'uwc.png'

- name: Redirect obsmon to shiny by overwriting default nginx configuration
  copy:
    src: "etc/nginx/sites-enabled/{{ hostname }}.{{ domain }}"
    dest: "/etc/nginx/sites-enabled/{{ hostname }}.{{ domain }}"
    owner: "root"
    group: "root"

- name: Overwrite default shiny server sample apps location
  copy:
    src: "etc/shiny-server/shiny-server.conf"
    dest: "/etc/shiny-server/shiny-server.conf"
    owner: "root"
    group: "root"

- name: Reload Nginx configuration for obsmon
  service:
    name: "nginx"
    state: "reloaded"

- name: Reload shiny-server configuration for obsmon
  service:
    name: "shiny-server"
    state: "reloaded"

- name: Create data directories for obsmon
  shell: |
    mkdir -p /obsmondata/dini/
    mkdir -p /obsmondata/ig/
  args:
    warn: yes

#####################
# Fetch and install obsmon
#####################

- name: Git checkout obsmon
  git:
    repo: "https://github.com/Hirlam/obsmon.git"
    dest: /srv/checkout
    version: "devel"

- name: Make sure to start from a fresh obsmon installation
  shell: cd /srv/checkout/ && ./install clean all
  args:
    warn: yes

- name: Install obsmon frontend
  shell: cd /srv/checkout/ && ./install
  args:
    warn: yes
paulovcmedeiros commented 2 years ago

Hi @khintz ,

Great that the version on devel works! I will merge it into master and wrap up a new release soon then.

Thank you very much indeed for the ansible pipeline! This is very useful, I may even include it in a future release as a reference for those interested.