search

pitt-crc / wrappers

User focused command line wrappers around Slurm
https://crc-pages.pitt.edu/wrappers/
GNU General Public License v3.0
1 stars 1 forks source link

Take Rawusage reset date into account when determining start date for `sreport` calls #237

Closed Comeani closed 4 months ago

Comeani commented 4 months ago

Currently, the start date provided to sreport is just the earliest start date across a group's active allocations. This is not necessarily accurate as the usage value being compared against the limits is the usage since the last raw usage reset.

This PR take the rawusage reset date into consideration, by returning the max of the earliest start date found and the most recent raw usage reset date. We want the max because start dates found prior to the raw usage reset date are what's creating the possibility of > 100% usage in the crc-usage output.

I tested this works properly on the cluster, and that it does not break crc-sus.

Before:

+-----------------------------------------------------------------------------+
|              Resource Allocation Request Information for 'slee'             |
+---------+--------------------------------------------+----------------------+
|    ID   |                   TITLE                    |   EXPIRATION DATE    |
+---------+--------------------------------------------+----------------------+
|  33300  |             NSF MRI Investment             |      2028-04-01      |
+---------+--------------------------------------------+----------------------+
|         |                  CLUSTER                   |    SERVICE UNITS     |
|         |                    ----                    |         ----         |
|         |                    MPI                     |        424315        |
|         |                    GPU                     |         9958         |
|         |                                            |                      |
+---------+--------------------------------------------+----------------------+
|  35516  |    Resource Allocation Request for slee    |      2025-04-08      |
+---------+--------------------------------------------+----------------------+
|         |                  CLUSTER                   |    SERVICE UNITS     |
|         |                    ----                    |         ----         |
|         |                    MPI                     |        50000         |
|         |                    SMP                     |        50000         |
|         |                    GPU                     |        50000         |
|         |                    HTC                     |        150000        |
|         |                                            |                      |
+---------+--------------------------------------------+----------------------+
+-----------------------------------------------------------------------------+
|                     Summary of Usage Across All Clusters                    |
+-------+-------------------------+----------------------+--------------------+
|  MPI  |    TOTAL USED: 74172    |   AWARDED: 474315    |     % USED: 15     |
+-------+-------------------------+----------------------+--------------------+
|       |           USER          |         USED         |       % USED       |
|       |           ----          |         ----         |        ----        |
|       |          shf80          |        74172         |         15         |
|       |          erc117         |          0           |         <1         |
|       |                         |                      |                    |
+-------+-------------------------+----------------------+--------------------+
|  GPU  |      TOTAL USED: 0      |    AWARDED: 59958    |     % USED: 0      |
+-------+-------------------------+----------------------+--------------------+
|       |                         |                      |                    |
+-------+-------------------------+----------------------+--------------------+
|  SMP  |    TOTAL USED: 957293   |    AWARDED: 50000    |    % USED: 1914    | <----- !!!!
+-------+-------------------------+----------------------+--------------------+
|       |           USER          |         USED         |       % USED       |
|       |           ----          |         ----         |        ----        |
|       |          jih145         |        488305        |        976         |
|       |          shf80          |        468982        |        937         |
|       |          erc117         |          6           |         <1         |
|       |                         |                      |                    |
+-------+-------------------------+----------------------+--------------------+
|  HTC  |     TOTAL USED: 2590    |   AWARDED: 150000    |     % USED: 1      |
+-------+-------------------------+----------------------+--------------------+
|       |           USER          |         USED         |       % USED       |
|       |           ----          |         ----         |        ----        |
|       |          jih145         |         2588         |         1          |
|       |          erc117         |          1           |         <1         |
|       |                         |                      |                    |
+-------+-------------------------+----------------------+--------------------+

After:

+-----------------------------------------------------------------------------+
|              Resource Allocation Request Information for 'slee'             |
+---------+--------------------------------------------+----------------------+
|    ID   |                   TITLE                    |   EXPIRATION DATE    |
+---------+--------------------------------------------+----------------------+
|  33300  |             NSF MRI Investment             |      2028-04-01      |
+---------+--------------------------------------------+----------------------+
|         |                  CLUSTER                   |    SERVICE UNITS     |
|         |                    ----                    |         ----         |
|         |                    MPI                     |        424315        |
|         |                    GPU                     |         9958         |
|         |                                            |                      |
+---------+--------------------------------------------+----------------------+
|  35516  |    Resource Allocation Request for slee    |      2025-04-08      |
+---------+--------------------------------------------+----------------------+
|         |                  CLUSTER                   |    SERVICE UNITS     |
|         |                    ----                    |         ----         |
|         |                    MPI                     |        50000         |
|         |                    SMP                     |        50000         |
|         |                    GPU                     |        50000         |
|         |                    HTC                     |        150000        |
|         |                                            |                      |
+---------+--------------------------------------------+----------------------+
+-----------------------------------------------------------------------------+
|                     Summary of Usage Across All Clusters                    |
+-------+-------------------------+-----------------------+-------------------+
|  MPI  |      TOTAL USED: 0      |    AWARDED: 474315    |     % USED: 0     |
+-------+-------------------------+-----------------------+-------------------+
|       |                         |                       |                   |
+-------+-------------------------+-----------------------+-------------------+
|  GPU  |      TOTAL USED: 0      |     AWARDED: 59958    |     % USED: 0     |
+-------+-------------------------+-----------------------+-------------------+
|       |                         |                       |                   |
+-------+-------------------------+-----------------------+-------------------+
|  SMP  |    TOTAL USED: 49952    |     AWARDED: 50000    |     % USED: 99    |
+-------+-------------------------+-----------------------+-------------------+
|       |           USER          |          USED         |       % USED      |
|       |           ----          |          ----         |        ----       |
|       |          shf80          |         42077         |         84        |
|       |          jih145         |          7875         |         15        |
|       |                         |                       |                   |
+-------+-------------------------+-----------------------+-------------------+
|  HTC  |     TOTAL USED: 2588    |    AWARDED: 150000    |     % USED: 1     |
+-------+-------------------------+-----------------------+-------------------+
|       |           USER          |          USED         |       % USED      |
|       |           ----          |          ----         |        ----       |
|       |          jih145         |          2588         |         1         |
|       |                         |                       |                   |
+-------+-------------------------+-----------------------+-------------------+