search

Unidata / LDM

The Unidata Local Data Manager (LDM) system includes network client and server programs designed for event-driven data distribution, and is the fundamental component of the Unidata Internet Data Distribution (IDD) system.
http://www.unidata.ucar.edu/software/ldm
Other
43 stars 27 forks source link

LDM 6.13.16 not using full queue size as 6.13.13 did #101

Closed akrherz closed 2 years ago

akrherz commented 2 years ago

I updated idd-dc.agron.iastate.edu from LDM 6.13.13 to 6.13.16 today. I am finding that the new LDM is not filling the 40 GB queue like the old version of LDM did. Here's two plots from my monitoring:

product_count

byte_count

The product_count appears to be 285713, here's a couple of pqmon outputs

$ pqmon
20210917T233337.536034Z pqmon[35201]                pqmon.c:main:314                    NOTE  Starting Up (35201)
20210917T233337.536311Z pqmon[35201]                pqmon.c:main:355                    NOTE  nprods nfree  nempty      nbytes  maxprods  maxfree  minempty    maxext  age
20210917T233337.536356Z pqmon[35201]                pqmon.c:main:463                    NOTE  285710     4       0 26578196432    285713        5         0 9685010216 1499
20210917T233337.536377Z pqmon[35201]                pqmon.c:cleanup:77                  NOTE  Exiting
$ pqmon
20210917T234705.321938Z pqmon[36245]                pqmon.c:main:314                    NOTE  Starting Up (36245)
20210917T234705.322147Z pqmon[36245]                pqmon.c:main:355                    NOTE  nprods nfree  nempty      nbytes  maxprods  maxfree  minempty    maxext  age
20210917T234705.322196Z pqmon[36245]                pqmon.c:main:463                    NOTE  285711     3       0 27466102112    285713        5         0 9717777264 2175
20210917T234705.322213Z pqmon[36245]                pqmon.c:cleanup:77                  NOTE  Exiting

Downgrading back to 6.13.13 and max products is now 784312

akrherz commented 2 years ago

Found the changelog entry from 6.13.14

hehe

>>> 140 / 51
2.7450980392156863
>>> 784312 / 285713
2.745104352969588

well, then, that's the answer.

semmerson commented 2 years ago

@akrherz The maximum number of products that the queue can hold is guaranteed to be wrong if the "/queue/slots" registry parameter is "default". You should manually compute the number based on the feeds you intend to get for a virgin installation and then use "ldmadmin addmetrics" and "ldmadmin plotmetrics" to fine-tune that parameter and also "/queue/size".