Velocity not working

[Kiglirs]

Hello.

I'm trying to use the Velocity products, but it's impossible. Somehow my velocity products would always ressemble like any of these

And sometimes this appears. If i go on the previous scan, it goes back to something like the 1st screenshot

What also can happen is that when switching to velocity, nothing appears or those screenshots can appear, and when i try to switch some other product like reflectivity or spectrum width, nothing appears for any products

This is the error i get when i launch the program

D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\nlr_importdata.py:2069: RuntimeWarning: Mean of empty slice
  c_data = np.nanmean(np.reshape(c_data[:,i], (360, 2, data.shape[1], ratio)), axis=(1,3))
2024-03-05 21:38:29.137318: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2024-03-05 21:38:29.137399: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2024-03-05 21:38:31.310658: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found
2024-03-05 21:38:31.310754: W tensorflow/stream_executor/cuda/cuda_driver.cc:263] failed call to cuInit: UNKNOWN ERROR (303)
2024-03-05 21:38:31.313254: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: DESKTOP-XXXXXXX
2024-03-05 21:38:31.313383: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: DESKTOP-XXXXXXX
detected GPUs: []
2024-03-05 21:38:31.325918: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-05 21:38:31.746989: W tensorflow/core/util/tensor_slice_reader.cc:96] Could not open D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\dealiasing\unet_vda\models\dealias_sn16_csi9764.SavedModel: UNKNOWN: NewRandomAccessFile failed to Create/Open: D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\dealiasing\unet_vda\models\dealias_sn16_csi9764.SavedModel : AccÞs refusÚ.
; Input/output error
 get_data, panel 0
Traceback (most recent call last):
  File "D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\nlr_datasourcegeneral.py", line 796, in get_data
    self.perform_mono_PRF_dealiasing(j)
  File "D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\nlr_datasourcegeneral.py", line 853, in perform_mono_PRF_dealiasing
    self.data[j] = self.vda(self.data[j], self.nyquist_velocities_all_mps[self.crd.scans[j]], extra_dealias='extra' in self.gui.dealiasing_setting)
  File "D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\dealiasing\unet_vda\unet_vda.py", line 74, in __call__
    self.load_model()
  File "D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\dealiasing\unet_vda\unet_vda.py", line 49, in load_model
    self.vda.load_weights(gv.programdir+'/Python_files/dealiasing/unet_vda/models/dealias_sn16_csi9764.SavedModel')
  File "C:\Users\[USER]\miniconda3\envs\NLradar\lib\site-packages\keras\utils\traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "C:\Users\[USER]\miniconda3\envs\NLradar\lib\site-packages\tensorflow\python\training\py_checkpoint_reader.py", line 45, in error_translator
    raise errors_impl.OpError(None, None, error_message, errors_impl.UNKNOWN)
tensorflow.python.framework.errors_impl.OpError

And later, i can also have this error

update current
add len(self._bzip2_start_pos)-1
restore_previous_attrs not set_data
list index out of range get_data, panel 0
Traceback (most recent call last):
  File "D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\nlr_datasourcegeneral.py", line 777, in get_data
    self.source_classes[self.data_source()].get_data(j)
  File "D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\nlr_datasourcespecific.py", line 856, in get_data
    self.import_class().get_data(filepath, j)
  File "D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\nlr_importdata.py", line 2371, in get_data
    scan_index = self.dsg.scannumbers_all[i_p][scan][self.dsg.scannumbers_forduplicates[scan]]
IndexError: list index out of range

I really don't know what's causing those issues, what could i have done wrong compared to other installs i did where i had this bug but only once on first launch

I hope this issue report is good, it's one of my first time doing such thing

[Bram94]

It would be interesting to first try whether disabling dealiasing (press SHIFT+V) leads to normal velocity output, just with aliasing issues. If that's the case, then we at least know that the issue is with the use of tensorflow.

[Bram94]

Also, are you using the suggested Python (3.8) and Tensorflow (2.10) versions?

[Kiglirs]

Python and Tensorflow are in the good versions. I've used the Anaconda installation. I checked with conda list

And using SHIFT+V to disable dealiasing works indeed, showing normal velocity output. And in case of if it's related to CPU rendering, i do have a Vega 56 GPU, so i'm not able to use GPU rendering, while i think with the other setup i have a 3060 which i think was using CUDA.

Also on the problem of products not loading, it seems like it's happening when switching to another product from Radial Velocity product. The disabling of dealiasing doesn't seems to make any change on that issue. The code reports a table not found error but i don't know if it's related to the issue:

BufrTableError: Table not found: 'Tables\libdwd\datacat.table'
BufrTableError: Table not found: 'Tables\libdwd\operator.table'
BufrTableError: Table not found: 'Tables\libdwd\local_00085_00000\codeflags_020'
BufrTableError: Table not found: 'Tables\libdwd\datacat.table'
BufrTableError: Table not found: 'Tables\libdwd\operator.table'
BufrTableError: Table not found: 'Tables\libdwd\local_00085_00000\codeflags_012'

[Bram94]

So it seems to be indeed an issue of Tensorflow not working correctly.

Now I first have one suggestion to at least get dealiasing working partially: If you press ALT+V, you can select which dealiasing options to use. If you select the leftmost option it will only apply dual-PRF dealiasing, which doesn't use Tensorflow.

Regarding the issue with products not showing correctly: I wonder, does this only occur after first getting issues with the velocity product, or does it also occur without ever having shown the velocity product? If the former is true, then we know that the velocity error is the starting point of the trouble.

Regarding Tensorflow use: You are mentioning one setup that uses an NVidia GPU, which should be supported. Do you get the same problems there as with the AMD GPU? And importantly, do you have both CUDA and cuDNN installed?

[Kiglirs]

Hello!

Indeed using only dual-PRS dealiasing works.

For the products not showing correclty, this behaviour seems random and hard to reproduce the exact same way. I was able to see that it wasn't depending on the dealiasing issue. Although at one attempt when i should have not the reflectivity product loading after i switched from the velocity product, the product loaded. Just an idea, but i am thinking that it's just because i switch to different product while the product haven't finished loading that i'm getting this issue. Sometimes i have to wait for next download for the product to load. But on other times even waiting after switching doesn't work and the problem just perists

Sometimes i'm getting this error:

D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\nlr_importdata.py:2069: RuntimeWarning: Mean of empty slice
  c_data = np.nanmean(np.reshape(c_data[:,i], (360, 2, data.shape[1], ratio)), axis=(1,3))

I've also got a 500 error from the MF API while trying to test and switch fast

As for the GPU, i tried to do it again on my other computer, and i actually had the same issue too, i may just have disabled the dealiasing by accident maybe. I also don't actually have CUDA and cuDNN installed on the NVidia GPU, and always had the same issue when i tried again to use the dealiasing

[Bram94]

Regarding the products issue: One thing that might be important here is that MF provides 2 types of files, with velocity and spectrum width coming out of 1 type, and the other products coming out of the other. This can be important when looking at the latest available timestamp, since 1 file might come available before the other. Is the issue also present with older timestamps?

Regarding Tensorflow issues: I would like you to try whether forcing CPU use works instead. You can force this by uncommenting line 11 in the script Python_files/dealiasing/unet_vda/unet_vda.py:

[Bram94]

I've also got a 500 error from the MF API while trying to test and switch fast

This should be unrelated to what you do in the GUI, and be a downloading error that I think has to do with data for that radar not being available.

[Kiglirs]

Regarding the products issue: One thing that might be important here is that MF provides 2 types of files, with velocity and spectrum width coming out of 1 type, and the other products coming out of the other. This can be important when looking at the latest available timestamp, since 1 file might come available before the other. Is the issue also present with older timestamps?

The issue is indeed present with older timestamp

Regarding Tensorflow issues: I would like you to try whether forcing CPU use works instead. You can force this by uncommenting line 11 in the script Python_files/dealiasing/unet_vda/unet_vda.py

This didn't worked, still getting the problem with Tensorflow

[Bram94]

This didn't worked, still getting the problem with Tensorflow

Does it also still raise the tensorflow.python.framework.errors_impl.OpError? In that case the CPU/GPU choice at least doesn't seem the issue.

[Kiglirs]

Yes it does. Still the same error

[Bram94]

I now think the issue is with opening the file with the saved model, given the error in your first post is raised for the line "self.vda.load_weights(gv.programdir+'/Python_files/dealiasing/unet_vda/models/dealias_sn16_csi9764.SavedModel')". It earlier also says access denied for this line, but that's a message that I'm also getting somehow, so I doubt that that is related to your issue.

[Bram94]

It might be useful to do a test with this model outside of NLradar. Earlier I had created a public repo with my implementation of the Unet VDA: https://github.com/Bram94/unet-vda. It would be interesting if you could clone this, and then run the example.py script.

[Kiglirs]

I am also getting the same error as in NLradar.

2024-03-06 22:38:36.617815: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2024-03-06 22:38:36.617906: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2024-03-06 22:38:42.060005: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found
2024-03-06 22:38:42.060084: W tensorflow/stream_executor/cuda/cuda_driver.cc:263] failed call to cuInit: UNKNOWN ERROR (303)
2024-03-06 22:38:42.062913: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: DESKTOP-XXXXXXX
2024-03-06 22:38:42.063003: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: DESKTOP-XXXXXXX
[]
2024-03-06 22:38:42.089689: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-06 22:38:42.567834: W tensorflow/core/util/tensor_slice_reader.cc:96] Could not open models\dealias_sn16_csi9764.SavedModel: UNKNOWN: NewRandomAccessFile failed to Create/Open: models\dealias_sn16_csi9764.SavedModel : AccÞs refusÚ.
; Input/output error
Traceback (most recent call last):
  File "example.py", line 52, in <module>
    data_new = vda(data, vn, azis, da)
  File "D:\Users\[USER]\Desktop\Meteo\unet-vda\unet_vda.py", line 80, in __call__
    self.load_model()
  File "D:\Users\[USER]\Desktop\Meteo\unet-vda\unet_vda.py", line 48, in load_model
    self.vda.load_weights('models/dealias_sn16_csi9764.SavedModel')
  File "C:\Users\[USER]\miniconda3\envs\NLradar\lib\site-packages\keras\utils\traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "C:\Users\[USER]\miniconda3\envs\NLradar\lib\site-packages\tensorflow\python\training\py_checkpoint_reader.py", line 45, in error_translator
    raise errors_impl.OpError(None, None, error_message, errors_impl.UNKNOWN)
tensorflow.python.framework.errors_impl.OpError

Although i used the same conda environment as NLradar, i don't know if it have an impact on that

[Bram94]

So again trouble with loading the model, and unclear why. One thing that could maybe provide some info is printing the error message in C:\Users[USER]\miniconda3\envs\NLradar\lib\site-packages\tensorflow\python\training\py_checkpoint_reader.py, like this:

Edit: You can probably also put loading the model into a try-except clause and print the error, like this:

[Bram94]

Otherwise the only other thing I can suggest is trying a new install of anaconda (maybe not miniconda this time, though unsure whether that could be an issue) and setting up the NLradar environment again.

[Kiglirs]

Aight i tried to install and use Anaconda, and nothing changed

I tried to do the error printing and nothing was getting printed actually, just an empty string Also i don't know in which file i should put the try-except code. Same one as the one with the print?

[Bram94]

You can put the try-except around the line that loads the model:

But it sucks that nothing has worked so far...

[Bram94]

In fact, the try-except option should not give different information than the print-option that you tried already, I think. But looking into the error handling of Tensorflow I noticed the option to call "tf.debugging.disable_traceback_filtering()", which you can probably simply put at the start of the Python_files/dealiasing/unet_vda/unet_vda.py script (after importing tensorflow first of course). This should give the full error traceback, which hopefully provides more information.

If that doesn't provide more info, then testing with importing simpler testing models might maybe give a clue about what goes wrong, but I'm not sure how long you want to continue the effort of solving this issue?

[Kiglirs]

Alright, i tried that and this is the message i'm getting.

{'_node_def': None, '_op': None, '_message': '', '_error_code': 2, '_experimental_payloads': {}}

And i'm alright to continue trying to solve the issue. I'm not that skilled in that kind of python coding, but i do understand some stuff and i'd like to help you solving this issue!

[Bram94]

Thanks, and great that you're willing to continue on this!

So the above outcome shows the result of printing the error attributes. I'll look into whether the error code gives some relevant information (edit: No, it unfortunately just means Error Unknown). In the mean time it would be great if you tried the following:

But looking into the error handling of Tensorflow I noticed the option to call "tf.debugging.disable_traceback_filtering()", which you can probably simply put at the start of the Python_files/dealiasing/unet_vda/unet_vda.py script (after importing tensorflow first of course). This should give the full error traceback, which hopefully provides more information.

For this it is important to remove the try-except code that I asked you to add earlier, since otherwise you won't get to see the error traceback.

[Bram94]

Have you had a chance to give the above suggestion of calling tf.debugging.disable_traceback_filtering() a try @Kiglirs ?

[Bram94]

Well, someone else with the same problem just tried it out, and the info enabled me to find a workaround for this: https://twitter.com/BramvantVeen/status/1767317867622441460. Could you try whether this also works for you?

[Bram94]

I think I have solved this issue now. Can you pull (git pull) the updated code, and check whether it is indeed solved for you?

[Kiglirs]

Hello! Sorry for the lack of update, i was very busy with life stuff.

I have made a git pull, but i am now just unable to get to next scans. I can just download the 1st velocity product, and sometimes i can't even download the 1st product. So it's just stuck there

Here's the error message that keeps appearing when i try to get to next scan

's' get_data, panel 1
Traceback (most recent call last):
  File "D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\nlr_datasourcegeneral.py", line 776, in get_data
    self.source_classes[self.data_source()].get_data(j)
  File "D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\nlr_datasourcespecific.py", line 752, in get_data
    self.dsg.MeteoFrance_BUFR.get_data(filepaths[file_index], j)
  File "D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\nlr_importdata.py", line 2097, in get_data
    self.dsg.data[j], data_mask, self.dsg.scantimes[j] = self.read_data(filepath, product, scan, self.crd.apply_dealiasing[j])
  File "D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\nlr_importdata.py", line 2059, in read_data
    bounds = [offset, offset+scale*(2**self.product_nbits[product]-1)]
KeyError: 's'

[Bram94]

No worries about the delay, and nice that you have tested it now :) And that is a small bug related to a change I made yesterday... I just fixed that. So you can try it again...

[Kiglirs]

Tried again, and the dealiasing fix works!

[Kiglirs]

And also, about the products not appearing after the velocity products, i've found a "solution" to avoid the problem. Somehow when having multiple pannels, this bug is not present

[Bram94]

Tried again, and the dealiasing fix works!

Just remembered I forgot to say one thing, which is that by default Unet VDA is not applied anymore for Meteo-France radars (because their Nyquist velocity is sufficiently high that this type of dealiasing is almost never needed). This means however that just testing Meteo-France radars doesn't verify that the fix with load_weights of the Unet model works.

There are 2 ways to test whether this fix actuallyworks: (1) You can raise the max Nyquist velocity for which the Unet model is applied. For that you can press ALT+V, and then set this velocity to for example 150 kts. (2) You can download some German or US data, and display their velocities.

[Kiglirs]

I can confirm that the Unet VDA dealiasing works on the US radars and with the max Nyquist velocity at 150kts!

[Bram94]

Great!

[Kiglirs]

Thanks a lot for making this program and fixing it, you did an amazing job!

I hope we can talk more of it, i'm curious of what could be the next features of it!

[Bram94]

And also, about the products not appearing after the velocity products, i've found a "solution" to avoid the problem. Somehow when having multiple pannels, this bug is not present

But well, this remains a weird issue then, though nice that you have at least found a workaround. Maybe I'll come back to this later.

[Bram94]

Thanks a lot for making this program and fixing it, you did an amazing job!

I hope we can talk more of it, i'm curious of what could be the next features of it!

Thanks! :) I'm currently mainly working on fixing issues that users encounter, and on properly finishing the implementation of the French radars (which includes adding clutter-filtered versions of products). After that I'm planning to make a video that shows how I use the program, to give a good impression of its functionality. With the many keyboard shortcuts that there are, it takes a while to get up to speed with it, and with that video I hope to make that easier.

[Kiglirs]

Thanks! :) I'm currently mainly working on fixing issues that users encounter, and on properly finishing the implementation of the French radars (which includes adding clutter-filtered versions of products). After that I'm planning to make a video that shows how I use the program, to give a good impression of its functionality. With the many keyboard shortcuts that there are, it takes a while to get up to speed with it, and with that video I hope to make that easier.

I see yeah, that would be interesting for the French radars.

Any plan for making an installer script and/or launcher script? I do feel like the main blocking point to use NLradar for a lot of people is installing/update or launching the program

[Bram94]

Yes I want to make a Windows executable with a simple installer, I've done that in the past too. But I want to wait with that until at least the most urgent bugs are fixed, and this work on French radars is finished.

[Kiglirs]

I see!

Also any idea about adding layers to the map? So at least we can see cities? And also maybe implementing NWS alerts and maybe European vigilances?

[Bram94]

Cities will probably be added in the future, just don't know when yet. But with alerts/warning polygons I don't have such plans yet. My main focus with NLradar is not really operational use, but in the first place storm analysis/research. That's for example why I included the option to add cases to a list.

The fact that NLradar includes automatic download shows that I also use it a lot for real-time analysis. But my focus is then a bit more on following storm behaviour than on generating warnings. Since especially for the US there are already multiple apps that are more suited for this. For real-time use I envision NLradar more as an addition to that, instead of a replacement.

[Kiglirs]

I see. I'm quite glad there's now a software that is able to show the MF radar, but yeah i'm more interested about the real time, because i feel like it's something that lacks in EU, unlike the US. There's a lot of mistakes that happened because of the very restricted access to data that there were in the past, and i was quite disappointed by the problems with the data that went open to use last january, like the inability to have easy use of the radar products. One of my goal is to have a good access to such data for the public to have more awarness of the weather in Europe, kinda like in the US. I don't want to make stuff like replacing official warnings and do stuff like in the US (aka weather weenies lol), but my focus is the ability to have awarness of sudden severe events.

I wanted to maybe make a software or website oriented to something like this, but i still lack quite a good amount of knowledge and i was blocked by the issues of decoding the bufr files. A very optimistic project that i don't think i'd be able to do alone

[Bram94]

There should definitely be people that can develop something that really focuses on real-time use, and that has all additional information that is desired in such a case, like warning areas. I would also definitely like an app like Radarscope for MF radars, that can also be used on the phone for example. And if my decoder for MF data can help others achieve that, I will only encourage them to use my code.