PPI Plots come as empty

[pandasambit15]

Hi! I was using the demo notebook for performing the HID on PPI data of C-Band. Strangely, the two panel plots fh arrays show empty data, while the actual array objects and radar fields have valid data. I tried plotting with simple pyplot.pcolormesh and wradlib.vis.plot_ppi for the same data files and everything seemed to be fine. Can you identify what is wrong with the approach? Note: My code was too big, so didn't insert here. Please let me know how I can share it with you.

@tjlang

[tjlang]

OK, based on your explanation, it appears that the FH field has been created OK, but the specific plotting approach in the notebook isn't working right? So the issue is with the plotting function defined in the notebook, not the actual FH processing? Is that correct?

[pandasambit15]

Yes Sir! You are right! The issue is with the plotting function!

One more thing I wanted to ask is regarding the HID algorithm itself! So, our radar is located in the south western India, which is a tropical station near the coast! It's a c-band dual pol radar. So after running the algorithm, I generally get only three dominant classes: Rain, Melting Hail and Aggregates. Some cases also show graupel, but those are pretty rare! The Melting Hail pixels always form a concentric circle around the PPI scan data at a specific range, which I think corresponds to the melting layer derived from the sounding data!

Does it have something to do with the membership functions, which might be tuned for US region and may not be applicable to Indian region?? Or is it a generalized approach?

Regards, Sambit

On Sat, Jul 18, 2020, 1:42 AM Timothy Lang notifications@github.com wrote:

OK, based on your explanation, it appears that the FH field has been created OK, but the specific plotting approach in the notebook isn't working right? So the issue is with the plotting function defined in the notebook, not the actual FH processing? Is that correct?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/CSU-Radarmet/CSU_RadarTools/issues/64#issuecomment-660316510, or unsubscribe https://github.com/notifications/unsubscribe-auth/APYEBNZUO6C2TXDBU4QLCBLR4CWDJANCNFSM4O6YXCYQ .

[bdolan44]

Hello, A couple of comments with respect to your HID questions. First of all, make sure that you are using the C-band option in the csu_fhc_summer function (band = 'C') to ensure it selects the MBFs for C-band. Secondly, temperature is weighted heavily in the fuzzy logic scoring, so the concentric circle is likely at your melting layer from the sounding data as you suspect (although it should also follow a minimum in rho_hv). Thirdly, it is difficult to say why you are only getting 3 classes. The C-band algorithm was developed using data from Darwin, Australia which sounds like it may be similar to your location (tropical / coastal), but it has also been tested in various other locations such as South America and the US, so it should be pretty generalized for summer convection. I would suggest looking carefully at the raw polarimetric data to see if your HID retrievals make sense, or if there is signal in the data which is not being caught by the algorithm or perhaps an issue with one of the variables which is driving the retrievals (a Zdr bias, noisy Kdp, anomalously low rho_hv are some of the common issues that can adversely affect HID retrievals). One thing you can change in the algorithm without changing the MBFs is the weighting functions, so if there is a problem with one of your variables you can weigh it less (or weigh the ones you trust more).

As for the plotting, are you getting any sort of error or is it just blank? Can you just post your lines of code which is creating the HID field, saving the HID field, and plotting the HID?

[pandasambit15]

Hello Brenda,

Thanks a lot for your response!

These are some of the points which I wanted to mention.

1. Regarding setting the band='C' for the csu_fhc_summer function, I had already done that while running the code.

2. Thanks for pointing out the temperature dependence of the algorithm, which i was suspecting from the results that were coming out from the algorithm. Additionally, is it possible to use any NWP model reanalysis sounding instead of the Wyoming Radiosonde data??? The reason why I ask this is because most of the time, in case of some extreme events like cyclones or flash floods, these radiosonde observations are not available. What would you suggest as an alternative for the sounding profile?

3. Regarding applying necessary weights to the polarimetric variables, can you please guide me how to implement it in the code? I want to do this for the RHOHV variable in the file, which is actually erroneous. There is some problem in the scale factor and offset applied by the radar product generation software that converts the files to netcdf format, which I don't have ay control over. This issue is being worked at currently, so, in the meantime I can just suppress the variables, as you suggested and use the algorithm.

4. One more experiment which I wanted to try is using only Z and ZDR for the HID algorithm, rather than the entire suite of polarimetric variables (which I guess is equivalent to setting the weights of the other variables equal to zero, except the ones for Z and ZDR). Let me know if this approach is right!

5. Regarding the plots, yes, the plots are actually blank, in spite of the variables having valid values in them. I am attaching my code here for your reference. Also, I am sharing a few sample files for your analysis. Please choose the raw_file (note below) for your testing.

Here is the link of the google drive folder containing the sample files:

https://drive.google.com/drive/folders/1wVz7P4bbO72K0QpDgoS47yqN9HdqnHtq?usp=sharing

Note:

1. There are two files: cfrad.CDWRTERLS_weather-DP_2017_11_30_05_03_47.nc http://cfrad.cdwrterls_weather-dp_2017_11_30_05_03_47.nc/ (raw_file) and v2_2017_11_30_05_03_47_highres_dprf_corrected.nc (proc_file). The raw_file is the raw ncfile from the radar, created through a Java Utility script, which I don't have any control over. The proc_file is the processed file where I have used wradlib VPR utility to create 3D CAPPI gridded file for DBZ and VEL. I want to use both kinds of files for my analysis. This proc_file I want to use for my analysis of reflectivity structure and rainfall characteristics.

2. The station coordinates are: (76.86, 8.53, 27.0) in (lon,lat,alt) format

Thanks and regards, Sambit

Sambit Kumar Panda Scientist/Engineer-SC, ASD/AOSG/EPSA, SAC(ISRO), Ahmedabad

On Mon, Jul 20, 2020 at 9:33 PM Brenda Dolan notifications@github.com wrote:

Hello, A couple of comments with respect to your HID questions. First of all, make sure that you are using the C-band option in the csu_fhc_summer function (band = 'C') to ensure it selects the MBFs for C-band. Secondly, temperature is weighted heavily in the fuzzy logic scoring, so the concentric circle is likely at your melting layer from the sounding data as you suspect (although it should also follow a minimum in rho_hv). Thirdly, it is difficult to say why you are only getting 3 classes. The C-band algorithm was developed using data from Darwin, Australia which sounds like it may be similar to your location (tropical / coastal), but it has also been tested in various other locations such as South America and the US, so it should be pretty generalized for summer convection. I would suggest looking carefully at the raw polarimetric data to see if your HID retrievals make sense, or if there is signal in the data which is not being caught by the algorithm or perhaps an issue with one of the variables which is driving the retrievals (a Zdr bias, noisy Kdp, anomalously low rho_hv are some of the common issues that can adversely affect HID retrievals). One thing you can change in the algorithm without changing the MBFs is the weighting functions, so if there is a problem with one of your variables you can weigh it less (or weigh the ones you trust more).

As for the plotting, are you getting any sort of error or is it just blank? Can you just post your lines of code which is creating the HID field, saving the HID field, and plotting the HID?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/CSU-Radarmet/CSU_RadarTools/issues/64#issuecomment-661131715, or unsubscribe https://github.com/notifications/unsubscribe-auth/APYEBNYQGL7X2K4EY2FNNQLR4RTF5ANCNFSM4O6YXCYQ .

[bdolan44]

2) With respect to other temperature sources, the temperature can come from wherever as long as you interpolate it to the radar coordinates (e.g. it should be the same size as the radar variables you are passing into the HID). So if you can do that with the reanalysis data, that should work just fine. 3) You can pass DEFAULT_WEIGHTS into csu_fhc_summer(DEFAULT_WEIGHTS =myweights), where myweights is a dictionary; these are the defaults as outlined in the code on line 21: DEFAULT_WEIGHTS = {'DZ': 1.5, 'DR': 0.8, 'KD': 1.0, 'RH': 0.8, 'LD': 0.5, 'T': 0.4} So change these based on how you feel about your data quality. NOTE: if you are using the method="hybrid", T and DZ weights do not matter as those scores are multiplied by the sum of the other variables multiplied by their weights. You can also select method='linear' which will add all the scores times their weights. This sometimes reduces the hard temperature boundaries but can be noisy, so you can play around with each and see what you like. 4) You can pass as many or as few variables to the HID as you want, so to do a Z, Zdr test (I would still include temperature) you would just do: scores = csu_fhc.csu_fhc_summer(dz=dz, zdr=dr, use_temp=True, band='C', T=radar_T) This is preferable to setting the weights to 0, and I think this would be a good test to understand your results. 5) I will try to take a look at your files when I have a moment. I do not see your code though?

[bdolan44]

--In trying to look and read your file (the "raw file"), I believe the problem is with pyart and your file. I could not get pyart to plot any of your variables, but I can see there is data there if I try pcolormesh as you have. From what I can tell, pyart does not calculate a correct 'x' value for the plotting, but where that comes from seems buried deep in the code and I don't know why all the x's are returned as 0's. So this might be a question for the doe-pyart community. --A second comment about the raw file is that the Z and Zdr values are integers which is unexpected. That could certainly be why you are only getting 3 categories from the HID. Particularly with Zdr you would not get the fidelity needed to resolve different hydrometers if it is not a floating value with several tenths of a dB precision.

[pandasambit15]

Hi Brenda,

Thanks a lot for your help.

I had actually attached my code csu_hid.py in my last email. Hope you found it.

Secondly, regarding the values of Z and ZDR, these are certainly not integers as I have been using them for quite some time with different utilities and packages, and also through basic netCDF4. There might be some reading issue, so can you please check that once again?

As you suggested, I have contacted the doe-pyart community and they have given me a quick fix for the plots being blank (it had something to do with data structuring and formatting). I will try to use that to create a proper radar object and then try out all those suggestions you have given for HID algorithm.

Once again, thank you so much for all the help and support, inspite of your busy schedule. I really appreciate it.

Regards, Sambit

On Sat, Jul 25, 2020, 3:11 AM Brenda Dolan notifications@github.com wrote:

--In trying to look and read your file (the "raw file"), I believe the problem is with pyart and your file. I could not get pyart to plot any of your variables, but I can see there is data there if I try pcolormesh as you have. From what I can tell, pyart does not calculate a correct 'x' value for the plotting, but where that comes from seems buried deep in the code and I don't know why all the x's are returned as 0's. So this might be a question for the doe-pyart community. --A second comment about the raw file is that the Z and Zdr values are integers which is unexpected. That could certainly be why you are only getting 3 categories from the HID. Particularly with Zdr you would not get the fidelity needed to resolve different hydrometers if it is not a floating value with several tenths of a dB precision.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/CSU-Radarmet/CSU_RadarTools/issues/64#issuecomment-663745845, or unsubscribe https://github.com/notifications/unsubscribe-auth/APYEBN5WRSTPY4OQ3SZQNFTR5H5ZDANCNFSM4O6YXCYQ .

[tjlang]

Closing.