NordicHPC / slurm2sql

Dump slurm accounting database to sqlite3 database for easy analysis
MIT License
10 stars 6 forks source link

Unable to convert memory-per-core figures (e.g. ReqMem = 8000Mc) #17

Open megatron-uk opened 3 weeks ago

megatron-uk commented 3 weeks ago

On my older Slurm system (sacct -V reports 20.11.8), when attempting to dump any amount of records, for example:

slurm2sql.py --history-days=7 -a slurm.sqlite3

The convert() code in slurm2sql.py does not seem to be handling memory-per-core values as recorded by Slurm:

Traceback (most recent call last):
  File "slurm2sql/slurm2sql.py", line 1204, in <module>
    exit(main(sys.argv[1:]))
         ^^^^^^^^^^^^^^^^^^
  File "slurm2sql/slurm2sql.py", line 688, in main
    errors = get_history(db, sacct_filter=sacct_filter,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "slurm2sql/slurm2sql.py", line 763, in get_history
    errors += slurm2sql(db, sacct_filter=new_filter, update=True, jobs_only=jobs_only,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "slurm2sql/slurm2sql.py", line 927, in slurm2sql
    processed_row = {k.strip('_'): (columns[k](row[k])
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "slurm2sql/slurm2sql.py", line 927, in <dictcomp>
    processed_row = {k.strip('_'): (columns[k](row[k])
                                    ^^^^^^^^^^^^^^^^^^
  File "slurm2sql/slurm2sql.py", line 147, in float_bytes
    return convert(x)
           ^^^^^^^^^^
ValueError: could not convert string to float: '2500Mc'

I can see the function slurmmem() does some parsing of these type of values for fields which are mapped to that function on line 927, but it doesn't appear to be getting called on this field (it's ReqMem in the above case).

megatron-uk commented 3 weeks ago

Thinking about this further, I think the issue stems from the fact that the defined mapper function for 'ReqMem' fields is float_bytes. If the mapper is changed to slurmmem() then the export of fields succeeds.

Is float_bytes() the default for ReqMem for a reason, or is this a bug?