search

IBM / CAST

CAST can enhance the system management of cluster-wide resources. It consists of the open source tools: cluster system management (CSM) and burst buffer.
Eclipse Public License 1.0
27 stars 34 forks source link

CSM BDS: findJobTimeRange.py is dependent on date+time format #995

Open thanh-lam opened 3 years ago

thanh-lam commented 3 years ago

Describe the bug The python script findJobTimeRange.py displays begin_time, history.end_time, and related info for an allocation. As described in #992, the date+time format is required to be updated to "yyyy-MM-ddThh:mm:ss.SSSZ". The problem is that findJobTimeRange.py defined this date format based on the old date string "yyyy-MM-dd hh:mm:ss.SSSZ". This caused a failure:

[csm04][/u/tlam/git/CAST/csm_big_data/python]> ./findJobTimeRange.py -a 259
# Found 1 matches for specified the job.
Traceback (most recent call last):
  File "./findJobTimeRange.py", line 149, in <module>
    sys.exit(main(sys.argv))
  File "./findJobTimeRange.py", line 121, in main
    start_time=datetime.strptime(tr_data["begin_time"], '%Y-%m-%d %H:%M:%S.%f')
  File "/usr/lib64/python3.6/_strptime.py", line 565, in _strptime_datetime
    tt, fraction = _strptime(data_string, format)
  File "/usr/lib64/python3.6/_strptime.py", line 362, in _strptime
    (data_string, format))
ValueError: time data '2021-02-12T16:17:28.393028' does not match format '%Y-%m-%d %H:%M:%S.%f'

To Reproduce Steps to reproduce the behavior:

  1. Go to /opt/ibm/csm/bigdata/python
  2. Run ./findJobTimeRange.py -a #The allocation ID should exist
  3. See error

Expected behavior Example of a successful run:

[csm04][/u/tlam/git/CAST/csm_big_data/python]> ./findJobTimeRange.py -a 259
# Found 1 matches for specified the job.allocation-id: 259
job-id: 1146 - 0
user-name: tlam 
user-id: 11745
begin-time: 2021-02-12.16:17:28:393 
end-time: 2021-02-12.16:17:48:421

Environment (please complete the following information):

Additional context Following two lines in findJobTimeRange.py are where date format is defined.

        date_format= '%Y-%m-%d %H:%M:%S.%f'
...
        start_time=datetime.strptime(tr_data["begin_time"], '%Y-%m-%d %H:%M:%S.%f')

Issue Source: Because the date format is "hard coded" in the script. It required to be changed every time the date+time format is changed.

thanh-lam commented 3 years ago

To fix the date+time format in the script, modify the two lines: Replace the blank between date and time with the letter T.

        date_format= '%Y-%m-%dT%H:%M:%S.%f'
...
        start_time=datetime.strptime(tr_data["begin_time"], '%Y-%m-%dT%H:%M:%S.%f')
thanh-lam commented 3 years ago

Found another script that also has similar date+time format hard coded: findJobKeys.py:

    date_format= '%Y-%m-%d %H:%M:%S.%f'

That needs to be modified similarly:

    date_format= '%Y-%m-%dT%H:%M:%S.%f'
thanh-lam commented 3 years ago

A 3rd script that uses old date+time format that needs to be corrected: findJobMetrics.py

from datetime import datetime
    date_format= '%Y-%m-%d %H:%M:%S.%f'
    start_time=datetime.strptime(allocation.get("begin_time"), date_format)
        end_time=datetime.strptime(allocation.get("history").get("end_time"), date_format)