search

Unidata / LDM

The Unidata Local Data Manager (LDM) system includes network client and server programs designed for event-driven data distribution, and is the fundamental component of the Unidata Internet Data Distribution (IDD) system.
http://www.unidata.ucar.edu/software/ldm
Other
43 stars 27 forks source link

Enhancement: Have multiple backup servers for a feed request #76

Closed sebenste closed 3 years ago

sebenste commented 3 years ago

Hello developers,

With LDM 6.13.12, you can request a feed with one primary, and one backup. Like so:

request IDS|DDPLUS "." server1.somewhere.com request IDS|DDPLUS (".") server2.somewhere.com

Or even a subset of a feed, like so:

request HDS "^[LM].[MN].98 KWNH" server1.somewhere.com request HDS "(^[LM].[MN].98 KWNH)" server2.somewhere.com

I would like to see up to 4 backups. With something like this:

request IDS|DDPLUS "." server1.somewhere.com request IDS|DDPLUS (".") server2.somewhere.com request IDS|DDPLUS ((".")) server3.somewhere.com request IDS|DDPLUS ((("."))) server4.somewhere.com request IDS|DDPLUS ((((".*")))) server5.somewhere.com

And similar using subsets:

request HDS|NEXRAD2NEXRAD3 "^[LM].[MN].98 KWNH" server1.somewhere.com request HDS|NEXRAD2|NEXRAD3 "(^[LM].[MN].98 KWNH)" server2.somewhere.com request HDS|NEXRAD2NEXRAD3 "((^[LM].[MN].98 KWNH))" server3.somewhere.com request HDS|NEXRAD2|NEXRAD3 "(((^[LM].[MN].98 KWNH)))" server4.somewhere.com request HDS|NEXRAD2NEXRAD3 "((((^[LM].[MN].98 KWNH))))" server5.somewhere.com

What do you think?

semmerson commented 3 years ago

Putting parentheses around the pattern field doesn't create a backup feed because the patterns are different. Instead, the parenthetical feed will also be in primary mode and will be a duplicate of the non-parenthetical feed in all respects.

Both the ".*" and subset requests in your example are supported now. They will each result in four permanently primary feeds. The resulting bandwidth utilization/requirement will be four times a single feed.

The LDM is 99.999% reliable. Consequently, having one primary and one alternate feed to independent upstream LDM-s will result in a failure rate of 1e-10. It's far more likely that the network or power would go out.

sebenste commented 3 years ago

Wait. So we are actually getting two full feeds then? I thought that it would take the header of the first one it got, and then requested the product from the fastest server, ie, the one from the header who got there first.

semmerson commented 3 years ago

Nope. See these activity diagrams on LDM behavior.

sebenste commented 3 years ago

Gotcha! OK, ticket closed.

semmerson commented 3 years ago

Just FYI, this "new" feature has been in the LDM for years. We didn't push it for the reasons given above: it uses much more bandwidth and doesn't accomplish much over a WAN.