[x] Have all tasks within the process_sfx DAG use the unified summary JSON file for eLog reporting.
[x] Updated stream_analysis to the single JSON.
[x] Updated merge to the single JSON.
find_peaks and index were already using it from previous PR.
Since solve was not previously reporting anything to the eLog it has not been updated. elog_display likewise has not been updated.
[x] Remove unnecessary task from index_run DAG.
[x] Fix multi-core bug (related to the other MPI issues) in stream_analysis for running from eLog.
[x] stream_analysis only updates the eLog summary for the most recent run with the specified tag. Therefore the report for each run contains the indexing results for that run, as well as the total indexing results for all runs up to that point with the same tag.
[x] Have first round jobs wait for JobScheduler jobs to finish to help with Airflow task processing/scheduling.
[x] Warning arising during plotting in merge is suppressed by setting xticks values before the labels.
[x] Fix apparent error in JSON handling during merge by only attempting read/write of summary file on rank 0.
Limitations
One core is sacrificial when using JobScheduler. In order for Airflow to not prematurely advance to the next task, the job it initially submitted must continue running.
It is only one core, but this should be the highest priority thing to fix!
Tasks run out of order may select the wrong summary file. Tasks such as stream_analysis or merge must be run as later tasks in a DAG or they are likely to select the wrong file.
There can be issues if the summary file is moved or deleted before the tasks finish. Or if there is no summary file before later tasks (such as merge) are called.
Functions created to help in stream_analysis and merge are non-general. Specifically, get_most_recent_run and get_most_recent_summary should be improved and moved to somewhere else in a refactor. They accomplish almost the exact same functionality but in a non-graceful and non-general way.
Future
Noticed an error in the index task related to the reading of HDF5 files. This doesn't cause the task to fail but should be investigated to make sure that there aren't underlying issues related to data handling.
HDF5-DIAG: Error detected in HDF5 (1.12.1) thread 0:
#000: H5Dio.c line 179 in H5Dread(): can't read data
Change Log
process_sfx
DAG use the unified summary JSON file for eLog reporting.stream_analysis
to the single JSON.merge
to the single JSON.find_peaks
andindex
were already using it from previous PR.solve
was not previously reporting anything to the eLog it has not been updated.elog_display
likewise has not been updated.index_run
DAG.stream_analysis
for running from eLog.stream_analysis
only updates the eLog summary for the most recent run with the specified tag. Therefore the report for each run contains the indexing results for that run, as well as the total indexing results for all runs up to that point with the same tag.JobScheduler
jobs to finish to help with Airflow task processing/scheduling.merge
is suppressed by setting xticks values before the labels.merge
by only attempting read/write of summary file on rank 0.Limitations
JobScheduler
. In order for Airflow to not prematurely advance to the next task, the job it initially submitted must continue running.stream_analysis
ormerge
must be run as later tasks in a DAG or they are likely to select the wrong file.merge
) are called.stream_analysis
andmerge
are non-general. Specifically,get_most_recent_run
andget_most_recent_summary
should be improved and moved to somewhere else in a refactor. They accomplish almost the exact same functionality but in a non-graceful and non-general way.Future
index
task related to the reading of HDF5 files. This doesn't cause the task to fail but should be investigated to make sure that there aren't underlying issues related to data handling.