search

marinebon / mbon-dashboard-server

server software for MBON early alert dashboard using Docker
1 stars 2 forks source link

fknms rebuild #37

Closed 7yl4r closed 1 year ago

7yl4r commented 2 years ago

fknms dash has been acting strange. I am taking a look at it now and doing a rebuild to determine what steps are next.

I have manually exported the json for each of the 4 dashboards. And now I am doing a hard reset:

[tylarmurray@fknms-dashboard-01 mbon-dashboard-server]$ docker-compose down --volumes --rmi all
[tylarmurray@fknms-dashboard-01 mbon-dashboard-server]$ docker-compose up airflow-init
[...]
Sending build context to Docker daemon   31.1GB
Step 1/3 : FROM apache/airflow:2.1.2
 ---> 4900d7864343
Step 2/3 : USER root
 ---> Running in 91c46641c141
Removing intermediate container 91c46641c141
 ---> 16c146d07dd2
Step 3/3 : RUN apt-get update &&     apt-get install --yes --no-install-recommends wget build-essential &&     wget https://curl.se/download/curl-7.78.0.tar.gz &&     tar -xvf curl-7.78.0.tar.gz && 
cd curl-7.78.0 &&     ./configure --with-gnutls && make && make install
 ---> Running in 5b8ff1977950
+ apt-get update
Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
Get:2 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:3 http://repo.mysql.com/apt/debian buster InRelease [22.1 kB]
Get:4 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Err:3 http://repo.mysql.com/apt/debian buster InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 467B942D3A79BD29
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7906 kB]
Get:6 http://deb.debian.org/debian buster-updates/main amd64 Packages [8792 B]
Get:7 http://security.debian.org/debian-security buster/updates/main amd64 Packages [317 kB]
Reading package lists...
W: GPG error: http://repo.mysql.com/apt/debian buster InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 467B942D3A79BD29
E: The repository 'http://repo.mysql.com/apt/debian buster InRelease' is not signed.
The command '/bin/bash -o pipefail -e -u -x -c apt-get update &&     apt-get install --yes --no-install-recommends wget build-essential &&     wget https://curl.se/download/curl-7.78.0.tar.gz &&    
 tar -xvf curl-7.78.0.tar.gz && cd curl-7.78.0 &&     ./configure --with-gnutls && make && make install' returned a non-zero code: 100
ERROR: Service 'airflow-init' failed to build : Build failed

I think that there are some things in the folder that are slowing things down a lot. So I am going to blow away the directory and redownload the repo.

[tylarmurray@fknms-dashboard-01 ~]$ sudo rm -rf mbon-dashboard-server/
[tylarmurray@fknms-dashboard-01 ~]$ git clone https://github.com/marinebon/mbon-dashboard-server.git -b client-fknms
[tylarmurray@fknms-dashboard-01 mbon-dashboard-server]$ git submodule update --init --recursive --remote
Submodule 'mbon_data_uploader' (https://github.com/marinebon/mbon_data_uploader) registered for path 'mbon_data_uploader'
Cloning into '/home/tylarmurray/mbon-dashboard-server/mbon_data_uploader'...
Submodule path 'mbon_data_uploader': checked out '921343ace136954c37eda4bc09ae0bedd1331af6'

[tylarmurray@fknms-dashboard-01 mbon-dashboard-server]$ docker container ls -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

[tylarmurray@fknms-dashboard-01 mbon-dashboard-server]$ vi .env 
[tylarmurray@fknms-dashboard-01 mbon-dashboard-server]$ echo -e "\nAIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" >> .env
[tylarmurray@fknms-dashboard-01 mbon-dashboard-server]$ docker-compose up airflow-init
[...]
same error as above

Looks like this is the issue.

7yl4r commented 2 years ago

everything running good but there are 5 dashboard .json files and only 2 showing in the GUI. Here is a lead on why that might be:

[tylarmurray@fknms-dashboard-01 ~]$ docker logs grafana
[...]
Restart grafana after installing plugins . <service grafana-server restart>

t=2022-02-22T00:59:10+0000 lvl=info msg="Starting Grafana" logger=server version=6.7.3 commit=a04ef6cefc branch=HEAD compiled=2020-04-23T10:12:53+0000
t=2022-02-22T00:59:10+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini
t=2022-02-22T00:59:10+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini
t=2022-02-22T00:59:10+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.data=/var/lib/grafana"
t=2022-02-22T00:59:10+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.logs=/var/log/grafana"
t=2022-02-22T00:59:10+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.plugins=/var/lib/grafana/plugins"
t=2022-02-22T00:59:10+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.provisioning=/grafana_provisioning"
t=2022-02-22T00:59:10+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.log.mode=console"
t=2022-02-22T00:59:10+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana"
t=2022-02-22T00:59:10+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana"
t=2022-02-22T00:59:10+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
t=2022-02-22T00:59:10+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/grafana_provisioning"
t=2022-02-22T00:59:10+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_SECURITY_ADMIN_USER=imars_grafana_user"
t=2022-02-22T00:59:10+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_SECURITY_ADMIN_PASSWORD=*********"
t=2022-02-22T00:59:10+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_AUTH_ANONYMOUS_ENABLED=true"
t=2022-02-22T00:59:10+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana
t=2022-02-22T00:59:10+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana
t=2022-02-22T00:59:10+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana
t=2022-02-22T00:59:10+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins
t=2022-02-22T00:59:10+0000 lvl=info msg="Path Provisioning" logger=settings path=/grafana_provisioning
t=2022-02-22T00:59:10+0000 lvl=info msg="App mode production" logger=settings
t=2022-02-22T00:59:10+0000 lvl=info msg="Initializing SqlStore" logger=server
t=2022-02-22T00:59:10+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
t=2022-02-22T00:59:10+0000 lvl=info msg="Starting DB migration" logger=migrator
[...]
t=2022-02-22T00:59:11+0000 lvl=eror msg="failed to save dashboard" logger=provisioning.dashboard type=file name="a unique provider name" error="Alert validation error: Data source used by alert ru
le not found, alertName=High discharge into Florida Bay alert, datasource="
t=2022-02-22T00:59:11+0000 lvl=eror msg="failed to save dashboard" logger=provisioning.dashboard type=file name="a unique provider name" error="Alert validation error: Data source used by alert ru
le not found, alertName=High discharge into Florida Bay alert, datasource="
t=2022-02-22T00:59:11+0000 lvl=eror msg="the same 'uid' is used more than once" logger=provisioning.dashboard type=file name="a unique provider name" uid=qhXdKw7Gk provider="a unique provider name
"
t=2022-02-22T00:59:11+0000 lvl=eror msg="the same 'title' is used more than once" logger=provisioning.dashboard type=file name="a unique provider name" title=FL_keys_dash provider="a unique provid
er name"
7yl4r commented 1 year ago

I am seeing 3 dashboardds in the console now.

Looking at the files, only these 4 json files have a uid, and one uid is repeated.

[tylarmurray@fknms-dashboard-01 dashboards]$ grep -R uid ./
./home_dashboard.json:  "uid": "Eu4SuDnMk",
./FK_NMS_Early_Alert.json:  "uid": "qhXdKw7Gk",
./old_fk_dash.json:  "uid": "i9FG9ghnk",
./FK_NMS_Early_Alert_v1_12_21.json:  "uid": "qhXdKw7Gk",

I overwrote the old fk_nms one with the newer version and removed the dashboards without uids (d1e2273e8850929b1b4cd0835853bfcbf783c1e0).

[tylarmurray@fknms-dashboard-01 dashboards]$ mv ./FK_NMS_Early_Alert_v1_12_21.json ./FK_NMS_Early_Alert.json
7yl4r commented 1 year ago

The home & FKNMS dashboard is now working : http://fknms-dashboard.marine.usf.edu:3000/d/qhXdKw7Gk/fk-nms-early-alert?orgId=1 but showing a templating error. this might just be because the data has not yet generated. The details of those jobs are in the airflow interface: http://fknms-dashboard.marine.usf.edu:8888/login/?next=http%3A%2F%2Ffknms-dashboard.marine.usf.edu%3A8888%2Fhome

7yl4r commented 1 year ago

everything is broken: