ldmsend_nws.c

[dgaer]

Rewrite of ldmsend.c that will allow send retries if an error occurs during the file upload. Add an optional notification to verify that the product was added to the LDM queue on the remote host. Due to some network issues, several of our sites has product upload failures using ldmsend. As some critical products are only sent at specific times we needed a way to resend the product during periods when the network was saturated. Due to our firewall configurations we cannot request from our downstream servers so we could not do a pqinsert at the site and request from the downstream queue. Ldmsend works most of the time but was failing during periods of when the network was saturated. The new version of ldmsend allow us to resend and verify the upload, for example:

su - ldm /bin/bash dd if=/dev/urandom of=/tmp/testfile.1 bs=1024 count=8192 cd /tmp status=$(/usr/local/ldm/util/ldmsend_nws -vxnl- -h 216.38.81.25 -f EXP -o 3600 -r 5 -R 60 -T 25 -p '^testfile.*' testfile.1) echo $status

NOTE: For testing purposes the new version of ldmsend was compiled as ldmsend_nws and copied to the util directory.

If status variable equals PASS the file made is to the upstream LDM queue. If status equals anything else, the file did not upload and our scripts can be adjusted to send alert messages to the forecasters if any critical upload failed.

[semmerson]

ldmsend_nws(1) calls forn5(). The purpose of ldmsend(1) is to act as an upstream LDM. Why does ldmsend_nws(1) act as a downstream LDM?

[semmerson]

Do your firewalls allow TCP connections to port 80?

[dgaer]

Hi Steve,

With the exception of our public Web server, all access public to port 80 is blocked. Access to our LDM queue monitor on port 80 is limited to our NWS field offices.

Thanks, Doug

On 09/12/2016 06:40 PM, Steven Emmerson wrote:

Do your firewalls allow TCP connections to port 80?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Unidata/LDM/pull/42#issuecomment-246529597, or mute the thread https://github.com/notifications/unsubscribe-auth/AOHr03hzjazjntY8-jXaUwonyCxf3m7wks5qpeLygaJpZM4GD_tA.

// ****** // // Douglas Gaer // Information Technology Officer // Linux Engineer/Administrator/Programmer // // Email: Douglas.Gaer@noaa.gov // Phone: 1-682-703-3714 // // National Weather Service // SRH/STSD/Technology Infusion Branch // 819 Taylor Street, Room 10A06 // Fort Worth, TX 76102 // // http://www.srh.noaa.gov // ****** //

[dgaer]

Hi Steve,

The reason I added the NOTIFYME call was to verify the product was inserted into the upstream LDM send. For example, all the NWS coastal offices upload a forecaster generated wind grid that initiates an on demand wave model run. Due to various network issues and other system issues there are times when the LDM send does not successfully deliver the wind grid and we have no indication the wind grid was not delivered to the upstream queue. Due to various network and systems setting we cannot request the product from the downstream queue, leaving ldmsend as our only option. In ldmsend_nws, a NOTIFYME runs to verify the product made it into the upstream queue. If the product upload fails, the upload script will detect the upload failure and send an AWIPS WRK product to alert the forecasters their wind grid was not uploaded. At this point the forecasters can initiate another model run. The majority of our LDM send uploads are observation data. Given the frequency of the observation data, if one or two product uploads fail due to a network error, we may never notice. But there are a few programs where the every data upload has to work or signal a failure event.

Thanks, Doug

On 09/12/2016 06:36 PM, Steven Emmerson wrote:

|ldmsend_nws(1)| calls |forn5()|. The purpose of |ldmsend(1)| is to act as an upstream LDM. Why does |ldmsend_nws(1)| act as a downstream LDM?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Unidata/LDM/pull/42#issuecomment-246528957, or mute the thread https://github.com/notifications/unsubscribe-auth/AOHr04wT9AIMWWNYDR2N7SNsYni2y1rIks5qpeILgaJpZM4GD_tA.

// ****** // // Douglas Gaer // Information Technology Officer // Linux Engineer/Administrator/Programmer // // Email: Douglas.Gaer@noaa.gov // Phone: 1-682-703-3714 // // National Weather Service // SRH/STSD/Technology Infusion Branch // 819 Taylor Street, Room 10A06 // Fort Worth, TX 76102 // // http://www.srh.noaa.gov // ****** //

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}