Original Savannah ticket 93840 reported by None on Thu Apr 19 04:34:48 2012.
The merge SQL (in BlockLatency::mergeLogBlockLatency) used by the InfoStatesClean agent to archive rows from t_xfer_file_latency into t_log_file_latency in PHEDEX_4_1_0 is inefficient when large numbers of files need to be archived simultaneously.
In Testbed2, closing several large blocks that were left open for months triggered the archiving of 5 million rows simultaneously, which was still running after 2 days before I killed the agent.
Though it is unlikely that this will happen in Prod (block destinations are completed at a regular frequency, at a rate of about 100 block destinations/cycle, so the number of files that need to be archived on each cycle is of the order of 10000), it would be a good idea to protect the agent from something like this.
Note that:
5 million is already above the number of Prod transfers executed in one month
the FilePump agent in Testbed2 didn't have significant performance issues in updating the t_xfer_file_latency on every cycle, even when it contained 5 million rows
Original Savannah ticket 93840 reported by None on Thu Apr 19 04:34:48 2012.
The merge SQL (in BlockLatency::mergeLogBlockLatency) used by the InfoStatesClean agent to archive rows from t_xfer_file_latency into t_log_file_latency in PHEDEX_4_1_0 is inefficient when large numbers of files need to be archived simultaneously.
In Testbed2, closing several large blocks that were left open for months triggered the archiving of 5 million rows simultaneously, which was still running after 2 days before I killed the agent.
Though it is unlikely that this will happen in Prod (block destinations are completed at a regular frequency, at a rate of about 100 block destinations/cycle, so the number of files that need to be archived on each cycle is of the order of 10000), it would be a good idea to protect the agent from something like this.
Note that:
N.