HPC usage analytics

[penguian]

| by saw562@nci.org.au

---

How can we monitor usage of the HPC using the ACCESS system?

Things we might want to collect:

• Who is using the system? (CoE, BoM, CSIRO)
• What are people running? (UM version, global vs. LAM)
• What resources are being asked for / used?

How we might go about it:

• Log job information within Rose/UMUI when a job is submitted
• Use statistics from PBS by adding a tag to ACCESS jobs

---

Issue migrated from trac:93 at 2024-01-31 18:02:29 +1100

[penguian]

@scott.wales@bom.gov.au commented

---

Ideas:

http://www.cyberciti.biz/tips/howto-log-user-activity-using-process-accounting.html

Summarise what users connect to the system & what commands get run using psacct

---

logger command: write to syslog, e.g.

logger -t UMUI "${USER} ${PROJECT} submitted $RUNID"

creates a line in /var/log/messages like:

Apr  9 11:28:57 accessdev UMUI: saw562 w35 submitted vabcd

Querying the logs may require some tooling

[penguian]

@scott.wales@bom.gov.au commented

---

This looks interesting - open source log monitoring & analytics

http://docs.fluentd.org/articles/free-alternative-to-splunk-by-fluentd

[penguian]

@scott.wales@bom.gov.au commented

---

Demo analytics setup for UMUI - http://130.56.244.76/kibana. The UMUI POSTs a json file containing job info to an elasticsearch instance when a job is processed, Kibana then produces analytics from the data in elasticsearch

Requires following UMUI patch to send basis info (ATM this is only being done on a test instance, not on accessdev proper):

#!diff
Index: UM/um_nav_actions.tcl
===================================================================
--- UM/um_nav_actions.tcl   (revision 951)
+++ UM/um_nav_actions.tcl   (working copy)
@@ -110,10 +110,66 @@
     unset processing_in_progress

     set processing_done 1
-    
-    
+
+   # JSON document for elasticsearch
+   set json "{"
+   append json "\"user\":\"$::env(USER)\""
+   append json ",\"project\":\"$::env(PROJECT)\""
+   append json ",\"basis\":[ jsonify_basis ]"
+   append json ",\"timestamp\":\"[ clock format [ clock seconds ] -gmt true -format "%Y-%m-%dT%H:%M:%SZ" ]\""
+   append json "}"
+
+   # POST the JSON document to the elasticsearch server
+   # Just use CURL as I don't understand TCL's http library
+   set url "http://localhost:9200/umui/process"
+   exec -ignorestderr curl -XPOST $url -d @- << $json
+   
+   set testfile [open ~/tmp w]
+    puts $testfile $json
+       close $testfile
 }

+# Turn the basis file into json format
+# Result looks like {"key":"value","key":["a","b","c"]}
+proc jsonify_basis {} {
+   # Json doesn't allow for a trailing comma, so we'll play a trick here
+   set delim ""
+
+   set result "{"
+   set variableNames [ get_allVariableName ]
+   foreach key $variableNames {
+       # Is this variable an array?
+       if [ regexp {\(.*\)} $key ] {
+           # Strip array size
+           set key [ regsub {\(.*\)} $key ""]
+           set value [ jsonify_list [ get_variable_array $key ] ]
+       } else {
+           set value "\"[ get_variable_value $key ]\""
+       }
+
+       append result $delim "\"$key\": $value"
+       set delim ","
+   }
+   append result "}"
+
+   return $result
+}
+
+# Turn a TCL list into json
+# Result looks like '[ "a", "b", "c" ]'
+proc jsonify_list {a} {
+   set delim ""
+
+   set result "\["
+   foreach value $a {
+       append result $delim "\"$value\""
+       set delim ","
+   }
+   append result "\]"
+
+   return $result
+}
+
 # um_nav_submit
 #  Called when Submit button pressed
 # Comments

[penguian]

ibc599 commented

---

Can we set this up for Rose?

Looks like you are using https://stats.accessdev.nci.org.au/umui/submit_vn** for umui submission?

Can we add another url for rose?

[penguian]

@scott.wales@bom.gov.au commented

---

Certainly, I've added indexes called 'rose' and 'rose-test' that can be accessed from accessdev.

To add entries POST json documents to a subdirectory of the index e.g.,

curl -xPOST https://stats.accessdev.nci.org.au/rose-test/my_type -d '{
     "user":      "ibc599",
     "runid":     "abc00",
     "timestamp": "1900-01-01"
}'

The type subdirectories correspond to a database table - they should all have the same argument list. For the UMUI stuff I've put each UM version into a different type since the basis variable list changes between versions.

The documentation on the elasticsearch website is pretty good for getting a handle on how to use it for submitting and searching through documents - http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/getting-started.html

[penguian]

@scott.wales@bom.gov.au removed milestone (was Future)

[penguian]

@scott.wales@bom.gov.au commented

---

Milestone Future deleted

[penguian]

@martin.dix@anu.edu.au commented

---

I'd like to experiment with this to track which cylc versions are being used.

However trying

curl -XPOST https://stats.accessdev.nci.org.au/rose-test/my_type -d '{ ... '}

from the command line gave an authentication error.

Adding -u mrd599 prompted for a password and then failed with

The requested URL /rose-test/my_type was not found on this server.

[penguian]

@scott.wales@bom.gov.au commented

---

'rose-test' hadn't been set up in apache, fixed now

[penguian]

@martin.dix@anu.edu.au commented

---

Uploading works ok now.

However the timestamp isn't behaving. If I do

curl -XGET 'https://stats.accessdev.nci.org.au/rose-test/_search'

I get entries like

{"_index":"rose-test","_type":"my_type","_id":"TweMNn4BTI2gA2nvumqJkw","_score":1.0, "_source" : {
 "user" : "mrd599",
 "runid" : "test",
 "project" : "p66",
 "timestamp":"2015-02-27T03:10:48Z", 
 "hostname" : "accessdev.nci.org.au"

The timestamps look the same from the umui index

However they don't seem to be interpreted correctly. Using a cut-down example from one of the charts

curl -XGET 'https://stats.accessdev.nci.org.au/rose-test/_search?pretty' -d '{
  "facets": {
    "terms": {
      "terms": {
        "field": "timestamp"
      }
    }
  },
  "size": 0
}'

gives split up parts of the time string like

      "terms" : [ {
        "term" : "2015",
        "count" : 6
      }, {
        "term" : "02",
        "count" : 6
      }, {
        "term" : "30z",
        "count" : 3

rather than the proper timestamps I get using the umui index.

Is there some extra step to tell it that timestamp should be interpreted as a time?