Integrating summer students's works into our monitoring pipeline.

[novicecpp]

The main goal is integrate Ek-ong and Tan's works to crab as part of monitoring system and keep maintain it forever.

Resources:

• Ek-ong code: https://github.com/ekongat/crab-datapipeline/
• The dashboard: https://monit-grafana.cern.ch/d/w8jWTsC4k/crab-ss-ekong?orgId=11&from=now-30d&to=now
• Docs: TBD
• Ek-ong's final goal: https://codimd.web.cern.ch/n0zKtr4KTZq2gquFpoQJdA?both#
• Ticket dicussion with CMS MONIT: CMSMONIT-555

The plan:

• [ ] Move pipeline's cronjobs to our environment.
  • Move Ek-ong script to gitlab's crab3.
  • Run CMS MONIT docker container in TW machine.
  • Setup cronjobs entries via Puppet.
• [ ] Validate all panel created by Ek-ong in Grafana before move it in to production dashboard.
• [ ] Create monitoring.
  • to notify pipeline fail and measure running time.
• [ ] Docs

[mapellidario]

We continued our work on this topic, mainly some cleanup.

What we did:

• [x] create a doc page in mkdocs: https://cmscrab.docs.cern.ch/technical/crab-monitoring/crab-spark.html
• [x] open a PR to dmwm/CRABServer
• [x] change the workdir in crab-dev-tw04 so that we now use Ek-Ong's branch of dmwm/CRABServer, not his original repo.
• [x] puppet: create a crontab https://gitlab.cern.ch/ai/it-puppet-hostgroup-vocmsglidein/-/blob/crabdev_taskworker/code/manifests/crabtaskworker/crabspark.pp

What we still have to do.

• [x] (bug) set proper permissions of cron_daily.sh and run_spark.sh, the scripts need to be executable
• [x] (ops / tutorial for dario) create the new indexes in opensearch
  • dario created a couple of new indexes with better names, but still need to finish it
• [ ] (doc) add link to source of osearch.py by ceyhun
• [ ] (ops) ask cms monit team for the service account cmscrab to have permissions to write to cms hdfs
  • otherwise we can not use cmscrab keytab to process the condor raw metrics
• [x] (ops) add to puppet the cronjob for the daily script
  • dario started working on this, but he still has to finalize it
• [x] (org) move the code from ekongat/crab-datapipeline to dmwm/CRABServer/src/script/Monitor/crab-spark. Ek-Ong will need to open a PR to dmwm/CRABServer
• [ ] (ops) run the cronjob from a crab-prod-tw01 / tw02
• [x] (improve) the cronjob should send notification only if we have a failure!
  • logs are available at /var/log/crab/crabspark.log
• [ ] use crabspark keytab
  • I see that we do not have new data in some panels, i think this fails to write to hdfs the intermediate data...

Comments are welcome! Feel free to edit this comment as we progress with our work :)

[mapellidario]

We identified an issue with the data source that we use for raw condor metrics:

!hdfs dfs -ls /project/monitoring/archive/condor/raw/metric/2023/09/

drwxr-xr-x+  - monitops hdfs          0 2023-09-10 07:45 /project/monitoring/archive/condor/raw/metric/2023/09/01
drwxr-xr-x+  - monitops hdfs          0 2023-09-17 08:44 /project/monitoring/archive/condor/raw/metric/2023/09/02
drwxr-xr-x+  - monitops hdfs          0 2023-09-11 03:52 /project/monitoring/archive/condor/raw/metric/2023/09/03
drwxr-xr-x+  - monitops hdfs          0 2023-09-12 05:00 /project/monitoring/archive/condor/raw/metric/2023/09/04
drwxr-xr-x+  - monitops hdfs          0 2023-09-13 04:57 /project/monitoring/archive/condor/raw/metric/2023/09/05
drwxr-xr-x+  - monitops hdfs          0 2023-09-15 08:48 /project/monitoring/archive/condor/raw/metric/2023/09/06
drwxr-xr-x+  - monitops hdfs          0 2023-09-15 10:03 /project/monitoring/archive/condor/raw/metric/2023/09/07
drwxr-xr-x+  - monitops hdfs          0 2023-09-17 07:52 /project/monitoring/archive/condor/raw/metric/2023/09/08
drwxr-xr-x+  - monitops hdfs          0 2023-09-17 08:46 /project/monitoring/archive/condor/raw/metric/2023/09/09
drwxr-xr-x+  - monitops hdfs          0 2023-09-17 08:42 /project/monitoring/archive/condor/raw/metric/2023/09/10
drwxr-xr-x+  - monitops hdfs          0 2023-09-17 08:51 /project/monitoring/archive/condor/raw/metric/2023/09/11
drwxr-xr-x+  - monitops hdfs          0 2023-09-17 08:52 /project/monitoring/archive/condor/raw/metric/2023/09/12
drwxr-xr-x+  - monitops hdfs          0 2023-09-17 08:44 /project/monitoring/archive/condor/raw/metric/2023/09/13
drwxr-xr-x+  - monitops hdfs          0 2023-09-17 08:53 /project/monitoring/archive/condor/raw/metric/2023/09/14
drwxr-xr-x+  - monitops hdfs          0 2023-09-17 08:47 /project/monitoring/archive/condor/raw/metric/2023/09/15
drwxr-xr-x+  - monitops hdfs          0 2023-09-17 08:56 /project/monitoring/archive/condor/raw/metric/2023/09/16
drwxrwxr-x+  - monitops hdfs          0 2023-09-18 00:48 /project/monitoring/archive/condor/raw/metric/2023/09/17.tmp
drwxrwxr-x+  - monitops hdfs          0 2023-09-19 00:01 /project/monitoring/archive/condor/raw/metric/2023/09/18.tmp
drwxrwxr-x+  - monitops hdfs          0 2023-09-20 00:53 /project/monitoring/archive/condor/raw/metric/2023/09/19.tmp
drwxrwxr-x+  - monitops hdfs          0 2023-09-21 00:36 /project/monitoring/archive/condor/raw/metric/2023/09/20.tmp
drwxrwxr-x+  - monitops hdfs          0 2023-09-22 00:00 /project/monitoring/archive/condor/raw/metric/2023/09/21.tmp
drwxrwxr-x+  - monitops hdfs          0 2023-09-22 15:00 /project/monitoring/archive/condor/raw/metric/2023/09/22.tmp

We informed Nikodemas from CMS Monit team and we will follow up.

[mapellidario]

Moreover, we noticed that the data about rucio taperecall from cmsspark [1] does not match with rucio monitoring [2]

we will investigate

[1]

[2]

[mapellidario]

Request from Stefano, on mattermost private channel:

• [ ] "breakdown by datatier of data locked by CRAB for tape recall and data input "

[novicecpp]

@nikodemas kindly ping me today that our pipeline has been broken for a long time. I proposed to him to have a chat in the first week of April (including usage of cms1 instance), after Easter. Is it ok for you @mapellidario ?