CSM is reporting incorrect information from the job

[verolero86]

CSM seems to show incorrect values for several fields. This is using: ibm-csm-hcdiag-1.0.0-9460.noarch ibm-csm-api-1.0.0-9460.ppc64le ibm-csm-core-1.0.0-9460.ppc64le

$ /opt/ibm/csm/bin/csm_allocation_query_details -a 2272
---
allocation_id:                  2272
primary_job_id:                 19121
secondary_job_id:               0
num_nodes:                      2
compute_nodes:
  h41n01:
  h41n02:
num_transitions:                2
state_transitions:
  - history_time:            2018-05-03 17:06:46.655256
    state:                   running
  - history_time:            2018-05-03 17:06:30.105715
    state:                   to-running
ssd_file_system_name:
launch_node_name:               login1
user_flags:
system_flags:
ssd_min:                        0
ssd_max:                        0
num_processors:                 0
num_gpus:                       0
projected_memory:               0
state:                          running
type:                           jsm-cgroup-step
job_type:                       batch
user_name:                      vgv
user_id:                        11685
user_group_id:                  22401
**INCORRECT: user_script:                    1525381589.19121**
begin_time:                     2018-05-03 17:06:30.105715
account:
comment:
**INCORRECT: job_name:                       stf006accept**
job_submit_time:                2018-05-03 17:06:29
queue:                          batch
requeue:
time_limit:                     10800
wc_key:
isolated_cores:                 1
num_steps:      1

[mattaezell]

I've also observed job_type be batch when it should be interactive. Note: this may be an LSF issue (LSF calling the API with the wrong values).

[mew2057]

Two questions:

1. What were user_script and job_name originally supposed to be, as in was this even close.
2. What was actually stored in the database? SQL to follow: Deleted:

   SELECT allocation_id,user_script,job_name 
   FROM csm_allocation_history 
   WHERE allocation_id=2272;

   Active:

   SELECT allocation_id,user_script,job_name 
   FROM csm_allocation
   WHERE allocation_id=2272;

If the database has a match to user input then we're going to have to dig into the csm_allocation_query_details, if the database matches the output the problem is then further up the chain with either csm_allocation_create or LSF.

[jma562]

job_type: batch <=== job_type is filled in by CSM.

INCORRECT: job_name: stf006accept <=== this is the project name (bsub -P) filled in by LSF

INCORRECT: user_script: 1525381589.19121 <=== for bsub redirect job, it's the job file name for regular job submitted not by bsub < script, it's the real executable This is to workaround an issue in CSM to store long strings with special chars in job script.

[mattaezell]

Shouldn't the project name (bsub -P) be set for the CSM field account? We expect CSM job_name to be the value of bsub -J

[verolero86]

@mew2057 no, it was not even close, unfortunately. The user_script file name should be: batchscript_IMB_test_nbc_000002n_1ppn.sh and the job_name should be IMB_test_nbc_000002n_1ppn.

I checked the CSM DB and it shows the same incorrect values:

SELECT allocation_id,user_script,job_name FROM csm_allocation_history WHERE allocation_id=2272;
 2272 | 1525381589.19121 | stf006accept

[jma562]

For account, LSF passes in the bsub -G which eventually decides how fairshare and limits are applied.

-P in LSF is a tag (short description)

For the script name, if you submit a job using bsub < script to redirect the job script to bsub stdin, LSF does not know the script name, the content of the script is redirected to bsub stdin, therefore, LSF uses the LSF job file name.

[verolero86]

@jma562 what do you mean by LSF job file name? #BSUB -J? If so, that is still not the correct value.

[jma562]

@verolero86 LSF job file name is a shell script LSF construct and runs, it contains the executable name you specified on bsub command line, or the content of the script that you redirect to bsub stdin. The job file consists of .

In case of bsub < script, the script name is not known by LSF but the content of the script is redirected to bsub stdin. The script may contains too many lines or special chars that CSM DB cannot handle, therefore, for bsub redirect job, LSF passes the job file name to the csm create API.

[jma562]

For job_type, the initial thought is that "batch" means that this allocation is created for a job from a batch scheduling system. "Interactive" means that this allocation is created by hand of running csm command. Therefore in LSF, the job_type field is left default (BATCH).

bsub -J is tricky, if not specified, it's user script name or content of the script. We may end up in the same issue of user_script unless -J is enforced or auto filled for each job.

[fpizzano]

@verolero86 and @mattaezell are you OK tracking this problem under LSF ?

[verolero86]

@fpizzano yes, that is fine. should we open a new LSF PMR or is there an internal issue already being tracked?

[jma562]

@verolero86 Hi Veronica, please log an LSF PMR. Within IBM, there has been a ticket logged and the job name and job type issue have been fixed (available in next LSF release). The user_script has been "partially" fixed as it depends on enhancement for bsub -script.

[verolero86]

I've verified that job_name now matches the value passed to LSF. There is still one remaining issue where the user_script is not found that we need to sort through but that is not CSM related, so this issue can be closed. Thanks!