search

lanl / Pavilion

HPC testing harness
BSD 3-Clause "New" or "Revised" License
16 stars 12 forks source link

Better Report Generating #15

Open cadejager opened 8 years ago

cadejager commented 8 years ago

a) For running/queued jobs we need a way to query a Moab or Slurm jobid and find information about that job, e.g. test name b) As the number of results grows, generating reports is woefully slow. I believe this is due to the fact that reports are generated by parsing each directory each and every time. Is there a way to cache some data? Or should we consider the use of a more permanent database?

cadejager commented 8 years ago

Our current plan to fix this is to create a csv file every day containing all of the information for get_results so that pavilion does not need to walk the directory tree every time get_results is called.

I propose creating the following files under results/YEAR/MONTH/DAY:

trend_data.csv: jobid, Data Name, Data value, test name, args jobs.csv: jobid, path, test, args job_status.csv: jobid, state, path

The new version could default to the old treewalk if it does not find the files and the old version could still treewalk the new results.