search

Unidata / LDM

The Unidata Local Data Manager (LDM) system includes network client and server programs designed for event-driven data distribution, and is the fundamental component of the Unidata Internet Data Distribution (IDD) system.
http://www.unidata.ucar.edu/software/ldm
Other
43 stars 27 forks source link

pqact state file resets when not given any work to do within LDM's lifecycle #73

Closed akrherz closed 3 years ago

akrherz commented 3 years ago

If LDM EXEC's pqact, but gives it no products to process, the state file written when LDM shuts pqact down has a "bad" timestamp. It would be nice if it retained the previous state's timestamp. I am using ldm-6.13.11.

Here's how I discovered this.

1) Have LDM run happily for many moons. 2) Restart LDM to apply a ldmd.conf update so to use a new upstream host. 3) After starting LDM, realize I typoed that hostname, stop LDM, fix the hostname, start LDM. 4) Observe pqact reprocess everything that was previously in the queue! The queue was holding 4000 seconds of data at the time, so it should not have retransferred

some log file examples to match the above from my development system

2) Starting from insertion-time 2020-09-08 01:46:32.160474 UTC 3) Starting from insertion-time 1969-12-31 23:59:59.9223372036854775807 UTC

and the state file looks like so:

# The following line contains the insertion-time of the last, successfully-
# processed data-product.  Do not modify it unless you know exactly what
# you're doing!
18446744073709551615.18446744073709551615

So my suggestion would be for pqact to retain the state from when it started and write that state back out in the case of a life-cycle of having no products processed.

Thanks

semmerson commented 3 years ago

Hi @akrherz,

This is fixed in the next release, which is undergoing testing. I hope to have it out by the end of this week.