Add resource usage reporting for associated allocations

alahiff commented 5 years ago

Will provide plots in Grafana in addition to an API for obtaining data.

API idea - to get usage data for a whole month (for the group you are in):

/prominence/v1/accounting/YYYY/MM

and to get a specific day:

/prominence/v1/accounting/YYYY/MM/DD

Response will be JSON, e.g.:

{
   "usage":[
      {
         "resource":"OpenStack-STFC",
         "cpuTime":10,
         "wallTime":11,
         "numberOfJobs":3
      },
      {
         "resource":"OpenStack-IC",
         "cpuTime":12,
         "wallTime":13,
         "numberOfJobs":4
      }
   ]
}

fcasson commented 5 years ago

number of users is also of interest

fcasson commented 5 years ago

Users may want to define custom periods if allocations do not map to calendar years / months

alahiff commented 4 years ago

Initial implementation in https://github.com/prominence-eosc/prominence/commit/8c73f25c6fd16f8148c3bd6cfc14b11655854c4f https://github.com/prominence-eosc/prominence/commit/aec7dabc579304df1167afe64a03e547329ad31b

Example queries:

/v1/accounting?start=2019-10-01&end=2019-11-01 to get usage for the current user only
/v1/accounting?start=2019-10-01&end=2019-11-01&by_group=true to get usage for the user's group
/v1/accounting?start=2019-10-01&end=2019-11-01&by_group=true&show_all_users=true to get usage for the user's group, showing individual usage by each user too

alahiff commented 4 years ago

Further updates in: https://github.com/prominence-eosc/prominence/commit/e4978f4f229033cd59e51a1afa67e7533f44b7cf https://github.com/prominence-eosc/prominence/commit/f173f4e35447bc725298058c516d289b835ee582 https://github.com/prominence-eosc/prominence/commit/c8ee7e67cfd812cc9a93938d56a3fa1287efd34e https://github.com/prominence-eosc/prominence/commit/ad091c39e4238c8d868b0410b147b243a5398382

Example default output:

{ 
   "usage":{ 
      "groups":{ 
      },
      "users":{ 
         "<username>":{ 
            "cpuTime":18.755324444444486,
            "numberOfJobs":245,
            "wallTime":31.673425716691547
         }
      }
   }
}

Example group output:

{ 
   "usage":{ 
      "groups":{ 
         "<groupname>":{ 
            "cpuTime":8378.71232888889,
            "numberOfJobs":265,
            "wallTime":12713.301708618137
         }
      },
      "users":{ 
      }
   }
}

Example group output showing individual users:

{ 
   "usage":{ 
      "groups":{ 
         "<groupname>":{ 
            "cpuTime":8378.71232888889,
            "numberOfJobs":265,
            "wallTime":12713.301708618137
         }
      },
      "users":{ 
         "<user1>":{ 
            "cpuTime":18.755324444444486,
            "numberOfJobs":245,
            "wallTime":31.673425716691547
         },
         "<user2>":{ 
            "cpuTime":415.7024377777778,
            "numberOfJobs":2,
            "wallTime":426.8041276974148
         },
         "<user3>":{ 
            "cpuTime":3027.5944122222227,
            "numberOfJobs":9,
            "wallTime":6949.955262517929
         },
         "<user4>":{ 
            "cpuTime":2.7777777777777186e-05,
            "numberOfJobs":1,
            "wallTime":384.0000411775377
         },
         "<user5>":{ 
            "cpuTime":4916.660126666667,
            "numberOfJobs":8,
            "wallTime":4920.868851508564
         }
      }
   }
}

(actual user names and group names replaced above).

alahiff commented 4 years ago

A usage command has been added to the CLI https://github.com/prominence-eosc/cli/commit/fe8eba77bb563dee7f4aa219e8738d1ffaa15f52

prominence-eosc / prominence

Add resource usage reporting for associated allocations #78