Radar Decoder: EDEX Fails on NEXRAD2 Data

[NaderCHASER]

I'm working to incorporate Level 2 NEXRAD into my EDEX server. I seem to remember this being enabled by default in the past but EDEX appears to not support it anymore.

Any insight on how we might be able to get Level 2 NEXRAD processed? All files are arriving with the L2-BZIP2 header and are chunked.

Entire Stack Trace: ERROR 2018-09-24 17:59:04,547 9608 [Ingest.Radar-4] Ingest: EDEX: Ingest - Failed to ingest /awips2/data_store/radar/KMVX_20180924_1753 java.lang.ArrayIndexOutOfBoundsException: 460 at com.raytheon.edex.plugin.radar.level2.Level2BaseRadar.processRadarDataMessages(Level2BaseRadar.java:445) ~[com.raytheon.edex.plugin.radar.jar:na] at com.raytheon.edex.plugin.radar.level2.Level2BaseRadar.<init>(Level2BaseRadar.java:193) ~[com.raytheon.edex.plugin.radar.jar:na] at com.raytheon.edex.plugin.radar.RadarDecoder.decodeLevelTwoData(RadarDecoder.java:442) ~[com.raytheon.edex.plugin.radar.jar:na] at com.raytheon.edex.plugin.radar.RadarDecoder.decode(RadarDecoder.java:195) ~[com.raytheon.edex.plugin.radar.jar:na] at sun.reflect.GeneratedMethodAccessor230.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_101] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_101] at org.apache.camel.component.bean.MethodInfo.invoke(MethodInfo.java:458) ~[camel-core-2.18.3.jar:2.18.3] at org.apache.camel.component.bean.MethodInfo$1.doProceed(MethodInfo.java:289) ~[camel-core-2.18.3.jar:2.18.3] at org.apache.camel.component.bean.MethodInfo$1.proceed(MethodInfo.java:262) ~[camel-core-2.18.3.jar:2.18.3] at org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:178) ~[camel-core-2.18.3.jar:2.18.3] at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109) ~[camel-core-2.18.3.jar:2.18.3] at org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:69) ~[camel-core-2.18.3.jar:2.18.3] at com.raytheon.uf.edex.esb.camel.MessageProducer$2.process(MessageProducer.java:439) [com.raytheon.uf.edex.esb.camel.jar:na] at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:110) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.Pipeline.process(Pipeline.java:120) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.Pipeline.process(Pipeline.java:83) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109) ~[camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.Pipeline.process(Pipeline.java:63) [camel-core-2.18.3.jar:2.18.3] at com.raytheon.uf.edex.esb.camel.MessageProducer$2.process(MessageProducer.java:439) [com.raytheon.uf.edex.esb.camel.jar:na] at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:110) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:681) ~[camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:609) ~[camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:238) ~[camel-core-2.18.3.jar:2.18.3] at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109) ~[camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:221) ~[camel-core-2.18.3.jar:2.18.3] at com.raytheon.uf.edex.esb.camel.MessageProducer$2.process(MessageProducer.java:439) [com.raytheon.uf.edex.esb.camel.jar:na] at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:110) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.TryProcessor.process(TryProcessor.java:113) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.TryProcessor.process(TryProcessor.java:84) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109) ~[camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.TryProcessor.process(TryProcessor.java:68) [camel-core-2.18.3.jar:2.18.3] at com.raytheon.uf.edex.esb.camel.MessageProducer$2.process(MessageProducer.java:439) [com.raytheon.uf.edex.esb.camel.jar:na] at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:110) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.Pipeline.process(Pipeline.java:120) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.Pipeline.process(Pipeline.java:83) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109) ~[camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:116) [camel-core-2.18.3.jar:2.18.3] at com.raytheon.uf.edex.esb.camel.MessageProducer$2.process(MessageProducer.java:439) [com.raytheon.uf.edex.esb.camel.jar:na] at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:110) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:542) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.Pipeline.process(Pipeline.java:120) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.Pipeline.process(Pipeline.java:83) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97) [camel-core-2.18.3.jar:2.18.3] at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:112) [camel-jms-2.18.3.jar:2.18.3] at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:721) [spring-jms-4.2.9.RELEASE.jar:4.2.9.RELEASE] at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:681) [spring-jms-4.2.9.RELEASE.jar:4.2.9.RELEASE] at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:651) [spring-jms-4.2.9.RELEASE.jar:4.2.9.RELEASE] at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:313) [spring-jms-4.2.9.RELEASE.jar:4.2.9.RELEASE] at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:251) [spring-jms-4.2.9.RELEASE.jar:4.2.9.RELEASE] at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1164) [spring-jms-4.2.9.RELEASE.jar:4.2.9.RELEAS E] at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1156) [spring-jms-4.2.9.RELEASE.jar:4.2.9.RE LEASE] at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1053) [spring-jms-4.2.9.RELEASE.jar:4.2.9.RELEASE] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]

I'm executing FILE against all NEXRAD2 with only the -close flag. A second pqact.conf entry with ".*/E" added to the end of the regex executes FILE with both -close and -edex flags. The EDEX appears to be only running against the end of the L2 chunks, which I thought might mitigate this issue.

Note: 460 is always the index that causes the ArrayIndexOutOfBoundsException.

cc @jaredwsmith - He has replicated my findings.

[WxmanJ]

I have successfully gotten this to work, but it did require the installation of the ORPG software. The ORPG software converts the level2 data to level3 for display in D2D... in native level 2 resolution.

[NaderCHASER]

@WxmanJ Thanks for this. I've contacted the appropriate party and will look into this as a solution.

[NaderCHASER]

@WxmanJ Any suggestions on the use of lb_create and having separate Linear Buffers for each radar site? I'm running into an issue where read_ldm is giving me a Linear Buffer that doesn't really make much sense.

I'm starting ORPG with 'mrpg -p -v startup' and the HCI with 'hci &' and creating a Linear Buffer with lb_create. read_ldm is pointed at the separate Linear Buffer and appears to be running normally, despite the LB being completely "corrupt" in a sense.

cvt i data/KTLH.lb output:

Inventory of Linear Buffer: KTLH.lb MSG# LBMSGLEN LBMSGNUM PRODLEN VOLNUM ELEV VOL DATE TIME 1 003828 31239 1212961867 -1059254718 10681 1970/07/14 04:24:32 2 006880 28685 1212961867 1089098818 -11076 1970/07/14 04:24:32 3 006880 28685 1212961867 13522498 -5700 1970/07/14 04:24:32 4 006880 28685 1212961867 551114818 -68 1970/07/14 04:24:32 5 006880 28685 1212961867 -1865393598 5565 1970/07/14 04:24:32 6 006880 28685 1212961867 -792044478 11197 1970/07/14 04:24:32 7 006880 28685 1212961867 281370178 16573 1970/07/14 04:24:32 8 006880 28685 1212961867 -1060675518 22205 1970/07/14 04:24:32 9 006880 28685 1212961867 -1866898878 27837 1970/07/14 04:24:32 10 006880 28685 1212961867 1355441218 -32067 1970/07/14 04:24:32 11 006880 28685 1212961867 -525310398 -26691 1970/07/14 04:24:32 12 006880 28685 1212961867 1086810178 -21059 1970/07/14 04:24:32 13 006880 28685 1212961867 -1329698238 -15427 1970/07/14 04:24:32

[WxmanJ]

@NaderCHASER Quite honestly, I haven't run the ORPG in about 8+ months, so I may need a refresher on this myself. Are you successfully seeing the processed data in AWIPS II?

What kind of diagnostics are you seeing? Did you setup the comms on ORPG and does HCI show a connection to AWIPS either on HCI or the radar tools in AWIPS?

To my "limited" knowledge, you can only run it for a single radar site at a time. In the past, the way I set it up for multiple radars was to use Docker containers with an ORPG session running on each one. I know the ROC recently provided a new release within the past year that finally pushes ORPG from CentOS5 to CentOS7. I was still using CentOS5 to run it so I am not sure what may have changed. I'll see if I can run my old VM with ORPG this week.

This may be a good conversation to continue via email, if interested.

[NaderCHASER]

@WxmanJ I figured things out. I'll be doing a write-up which involves "pqinsert" instead of attaching the ORPG to EDEX via Comms, since I believe Unidata's EDEX has that functionality (RadarServer) taken out now.

[zmanvortex]

@NaderCHASER I have had an ORPG running for several years feeding Level 3 products to my custom display software. I just now am getting into AWIPS and can't figure out how to connect CAVE to my ORPG instance. Any help is appreciated.

[NaderCHASER]

@zmanvortex My ORPG writes to disk using my customized NBTCP which sends a PUB/SUB message to a Redis instance once the product has been written to disk. On the EDEX server I simply subscribe to those messages using a script that runs "pqinsert" for each message/product.

An alternative approach that doesn't use Redis and doesn't require NBTCP to be modified would be to use inotifywait to monitor directories for file creation. This can be difficult because you don't know for sure that the file has been completely written, only that it was created.

[WxmanJ]

I am connected directly to ORPG via CAVE/EDEX. If you are on the same network, ORPG will send the processed prods to your EDEX server (/tmp/sbn/radar) via the QPID message broker and process.

As far as actually connecting CAVE to the ORPG, make sure you have all the rcm plugins required to make that happen. Unidata may have stripped out some of these items (i.e. I use the NWS build, not Unidata), since they don't use them. Also of note, the Radar server does not start automatically when you start EDEX, you need to start it separately from /awips2/rcm/bin/start. That will start process, which uses port 8813 and this will need to be accessible for the ORPG to actually send data. Separately, you will also need a port enabled on your ORPG setup to connect (I set-up 5908 by default).

You can confirm you are connected to the ORPG session through the HCI indicating a comms link and by the radar config screen on CAVE. I am attach a few pictures to show my setup.

[zmanvortex]

Thanks! I am currently using nbtcp and have some custom software monitoring the nbtcp output directory, sending the products to a directory that can be accessed from the web, and inserting product data into a mysql database. I was looking at some (apparently older) manuals for CAVE which had some menu options under radar for what looked like connecting to an ORPG. I could not find these menu items in my recent install of CAVE and it looks like they removed it at some point. I was hoping CAVE could connect directly to the ORPG through the narrowband interface like nbtcp does. I was not planning on setting up my own EDEX server, but rather I was going to purchase the EDEX server subscription from AllisonHouse since that is where I get the Level II data to feed into my ORPG. Looks like I might give up on viewing my ORPG data on CAVE. I have created a web page that I am working on that utilizes data from my ORPG to display products. Still needs alot of work, but here is the url: http://zmanvortex-001-site2.atempurl.com/RadarViewer?proc=2 . It is in 3D so move the mouse around to see the results. Hold down the shift key with the left mouse to pan. Might take awhile to load the earth texture and county lines the first time you visit the page. Thanks for responding!

[WxmanJ]

@zmanvortex Very Cool!!

I am not sure it would work if you are using an EDEX server from AllisonHouse because it would need to connect to their QPID broker and the radar server would need to be running on their end -- it's all kind of intertwined.

I am pulling the radar data from NOAA Big Data project, through AWS S3 bucket.

[zmanvortex]

@WxmanJ thanks! I might try to set up my own EDEX server in the future. Here are some screenshots of the volume viewer function I am adding to my viewer page: