[x] Have all tasks within the process_sfx DAG use the unified summary JSON file for eLog reporting.
[x] Updated stream_analysis to the single JSON.
[x] Updated merge to the single JSON.
find_peaks and index were already using it from previous PR.
Since solve was not previously reporting anything to the eLog it has not been updated. elog_display likewise has not been updated.
[x] Remove unnecessary task from index_run DAG.
[x] Fix multi-core bug (related to the other MPI issues) in stream_analysis for running from eLog.
[x] stream_analysis only updates the eLog summary for the most recent run with the specified tag. Therefore the report for each run contains the indexing results for that run, as well as the total indexing results for all runs up to that point with the same tag.
[x] Have first round jobs wait for JobScheduler jobs to finish to help with Airflow task processing/scheduling.
[x] Warning arising during plotting in merge is suppressed by setting xticks values before the labels.
[x] Fix apparent error in JSON handling during merge by only attempting read/write of summary file on rank 0.
Limitations
One core is sacrificial when using JobScheduler. In order for Airflow to not prematurely advance to the next task, the job it initially submitted must continue running.
It is only one core, but this should be the highest priority thing to fix!
Tasks run out of order may select the wrong summary file. Tasks such as stream_analysis or merge must be run as later tasks in a DAG or they are likely to select the wrong file.
There can be issues if the summary file is moved or deleted before the tasks finish. Or if there is no summary file before later tasks (such as merge) are called.
Functions created to help in stream_analysis and merge are non-general. Specifically, get_most_recent_run and get_most_recent_summary should be improved and moved to somewhere else in a refactor. They accomplish almost the exact same functionality but in a non-graceful and non-general way.
Future
Noticed an error in the index task related to the reading of HDF5 files. This doesn't cause the task to fail but should be investigated to make sure that there aren't underlying issues related to data handling.
HDF5-DIAG: Error detected in HDF5 (1.12.1) thread 0:
#000: H5Dio.c line 179 in H5Dread(): can't read data
Change Log
process_sfxDAG use the unified summary JSON file for eLog reporting.stream_analysisto the single JSON.mergeto the single JSON.find_peaksandindexwere already using it from previous PR.solvewas not previously reporting anything to the eLog it has not been updated.elog_displaylikewise has not been updated.index_runDAG.stream_analysisfor running from eLog.stream_analysisonly updates the eLog summary for the most recent run with the specified tag. Therefore the report for each run contains the indexing results for that run, as well as the total indexing results for all runs up to that point with the same tag.JobSchedulerjobs to finish to help with Airflow task processing/scheduling.mergeis suppressed by setting xticks values before the labels.mergeby only attempting read/write of summary file on rank 0.Limitations
JobScheduler. In order for Airflow to not prematurely advance to the next task, the job it initially submitted must continue running.stream_analysisormergemust be run as later tasks in a DAG or they are likely to select the wrong file.merge) are called.stream_analysisandmergeare non-general. Specifically,get_most_recent_runandget_most_recent_summaryshould be improved and moved to somewhere else in a refactor. They accomplish almost the exact same functionality but in a non-graceful and non-general way.Future
indextask related to the reading of HDF5 files. This doesn't cause the task to fail but should be investigated to make sure that there aren't underlying issues related to data handling.