tiffanycmeyer13
commented
2 years ago Is LDM running and do you (have you in the past) have access to data via Unidata's LDM feed?
Closed W3AXL closed 2 years ago
tiffanycmeyer13
commented
2 years ago Is LDM running and do you (have you in the past) have access to data via Unidata's LDM feed?
W3AXL
commented
2 years ago Well, you're most certainly on to something there:
20220426T030026.879548Z idd.unidata.ucar.edu[4756] error.c:err_log:237 NOTE Upstream LDM didn't reply to FEEDME request; RPC: Unable to receive; errno = Connection reset by peer
20220426T030026.879670Z idd.unidata.ucar.edu[4756] LdmConfFile.c:requester_exec:925 NOTE Sleeping 60 seconds before retrying...
20220426T030026.891974Z idd.unidata.ucar.edu[4750] error.c:err_log:237 NOTE Upstream LDM didn't reply to FEEDME request; RPC: Unable to receive; errno = Connection reset by peer
20220426T030026.892122Z idd.unidata.ucar.edu[4750] LdmConfFile.c:requester_exec:925 NOTE Sleeping 60 seconds before retrying...
20220426T030026.912362Z idd.unidata.ucar.edu[4749] error.c:err_log:237 NOTE Upstream LDM didn't reply to FEEDME request; RPC: Unable to receive; errno = Connection reset by peer
20220426T030026.912516Z idd.unidata.ucar.edu[4749] LdmConfFile.c:requester_exec:925 NOTE Sleeping 60 seconds before retrying...
20220426T030027.079166Z idd.unidata.ucar.edu[4752] error.c:err_log:237 NOTE Upstream LDM didn't reply to FEEDME request; RPC: Unable to receive; errno = Connection reset by peer
20220426T030027.079323Z idd.unidata.ucar.edu[4752] LdmConfFile.c:requester_exec:925 NOTE Sleeping 60 seconds before retrying...
20220426T030027.182731Z idd.unidata.ucar.edu[4760] error.c:err_log:237 NOTE Upstream LDM didn't reply to FEEDME request; RPC: Unable to receive; errno = Connection reset by peer
20220426T030027.182838Z idd.unidata.ucar.edu[4760] LdmConfFile.c:requester_exec:925 NOTE Sleeping 60 seconds before retrying...Are the LDM feeds restricted in any way? I was under the impression they were available for public use similar to the cloud-edex feed.
tiffanycmeyer13
commented
2 years ago Yes, the LDM feeds are restricted and not publicly available. They are however available to Universities and those on a .edu domain.
We have had a user write their own scripts to pull data from ftp sites and manually ingest them into EDEX, but not all datasets are available for this.
W3AXL
commented
2 years ago Thanks for the information. I'm curious how companies like AllisonHouse obtain their EDEX server data - are they doing as you said and collecting data manually from various public sources online?
My end goal was to host a local EDEX server to replace the cloud-edex public feed, to provide quicker data updates and more datasets without having to resort to a subscription service. I have a large server sitting idle most of the time that would be perfect for this application.
tiffanycmeyer13
commented
2 years ago Unidata has a special agreement with AllisonHouse, they provide us direct access to their NOAAPort ingest machine feeds, and we provide them with IDD access on a provisional basis. AllisonHouse also has several LDM feeds set up with other Universities to get data.
Are there specific datasets you are looking for?
W3AXL
commented
2 years ago Ah, understood. Thanks for all the helpful info!
I'm primarily interested in realtime data - NEXRAD, GOES, SPC products (outlooks, watches, warnings, etc), and surface observations (stations, lightning, etc). I typically don't ever use things like the models.
The only reasons I'm interested in running a local EDEX server are: a) because I've got a spare server with nothing better to run on it, b) I'd love to get faster load times for data in CAVE than I can get with the cloud-edex server, and c) because it seems like a fun thing to do.
WxmanJ
commented
2 years ago @W3AXL you can pull some of these data sets (GOES, NEXRAD) from NOAA Big Data. You just need to setup a script to manually ingest them either by dropping them in the manual/dropbox folder or run the qpidnotify command.
srcarter3
commented
2 years ago If you manually set up a process to collect/receive the data, use the notifyAWIPS2-unidata.py script found in /awips2/ldm/dev/ to trigger decoding and ingesting into EDEX.
Using the manual ingest point is extremely inefficient (processing wise) and should not be used in an automated fashion.
WxmanJ
commented
2 years ago @srcarter3 Kudos to you guys for setting up your own qpid notification script! :-)
Folling the EDEX install guide from http://unidata.github.io/awips2/install/install-edex/, I have an EDEX server set up for local use. I changed no config files and everything is as it comes installed via the installation script.
While I can connect to the server from CAVE, it appears that no data is actually being ingested or processed by the server. The output of edex qpid is below:
As you can see, no products are being processed. CPU usage is also relatively low, which would indicate no processors are running. Is there additional config required that isn't mentioned in the online installation guide?