semmerson
commented
9 years ago Daryl,
The modification times of the ".state" files are a red herring because the files are written when the LDM shuts down.
Each file identifies the last data-product that was successfully processed by the pqact(1) process associated with the similarly-named configuration-file (which is why pqact(1) processes and their configuration-files need to be one-to-one). pqact(1) uses this information to start processing where it left off (assuming the identified data-product is still in the product-queue).
Consequently, you shouldn't have encountered the problem you did unless something else was wrong.
Were "processed oldest product" messages being written to the LDM log file prior to this? This would violate the assumption at the end of the second paragraph.
Regards, Steve Emmerson
On Thu, May 7, 2015 at 6:43 AM, daryl herzmann notifications@github.com wrote:
With the current release, all pqact processes appear to write the same .state file and not a .state file that is accurate to that pqact process. For example, on a machine that has 9 pqact process, the written out .state file for each at shutdown time has the same timestamp!
Here's the problem. Yesterday, my system feeds NEXRAD2 from idd.unidata.ucar.edu and other feeds from idd.ssec.wisc.edu. The NEXRAD2 feed became very latent, with a feed latency at shutdown time of ~30 minutes. I have 7 pqact processes that only look at that feedtype, so their queue time should have been 30 minutes in the past. The IDS feedtype was current, so this resulted in .state files with the current timestamp . So when I started LDM back up, the 7 pqact processes ignored the incoming new NEXRAD2 data because the pqinsert times were within that recent 30 minute window. After some cussing, I realized what had happened, stopped LDM, removed the state files and started LDM again. Of course, data was missed during this period :(
Please modify pqact to write state files that represent where exactly that pqact process is within the LDM queue so that I don't loose data during restarts. Even in the normal case of a machine with multiple pqact processes, this would result in products being missed.
— Reply to this email directly or view it on GitHub https://github.com/Unidata/LDM/issues/36.
akrherz
With the current release, all pqact processes appear to write the same .state file and not a .state file that is accurate to that pqact process. For example, on a machine that has 9 pqact process, the written out .state file for each at shutdown time has the same timestamp!
Here's the problem. Yesterday, my system feeds NEXRAD2 from idd.unidata.ucar.edu and other feeds from idd.ssec.wisc.edu. The NEXRAD2 feed became very latent, with a feed latency at shutdown time of ~30 minutes. I have 7 pqact processes that only look at that feedtype, so their queue time should have been 30 minutes in the past. The IDS feedtype was current, so this resulted in .state files with the current timestamp . So when I started LDM back up, the 7 pqact processes ignored the incoming new NEXRAD2 data because the pqinsert times were within that recent 30 minute window. After some cussing, I realized what had happened, stopped LDM, removed the state files and started LDM again. Of course, data was missed during this period :(
Please modify pqact to write state files that represent where exactly that pqact process is within the LDM queue so that I don't loose data during restarts. Even in the normal case of a machine with multiple pqact processes, this would result in products being missed.