Allow the user to use met data from other sites different than rda.ucar.edu

[kikocorreoso]

Right now a user only has one option to download met data within GIS4WRF. The only option needs an account. This is very limited.

The idea is to allow the user to download the met data the way they want and give them the responsability to provide the necessary information to GIS4WRF to 'discover' these datasets:

• dataset folder location
• dataset folder name (format)
• individual files name (format)
• ...

This is more flexible than adding datasets or new sites to GIS4WRF via code and gives the user the ability to download a bunch of files in an automated way instead of having to select one by one the potential possibilities from a dataset from rda.ucar.edu.

[dmey]

It would be good to have a chance to rethink the logic under Meteorological Data in Simulation > Data. The question here is how to handle cases where the data are not downloaded using GIS4WRF. At the moment, the data that are downloaded using the tool in Datasets > Met are stored in the gis4wrf directory under met, DATASET, PRODUCT, DATE_RANGE. By doing this we can then have a simple logic to display the grouped datasets under Meteorological Data in Simulation > Data.

If you are looking to import other datasets (out of the 5 we currently support) that you may already have on your system, you can just copy them user the met folder following the DATASET, PRODUCT, DATE_RANGE hierarchy -- GIS4WRF will pick them up and display them under the Available Datasets section.

If you are looking to import other datasets we do not currently support it gets a bit more tricky to handle. We would need to keep a simple UI that also gives more advanced users the option to import external datasets and set the appropriate UNGRIB.TBL.

@kikocorreoso Were you thinking of adding a button at the end of the Meteorological Data section in Simulation > Data -- similar to what there is currently for the geographical datasets (i.e. Add Dataset to List button) -- to import other datasets manually? But if we do this, we will also require to the user to specify date and UNGRIB.TBL. Or did you have something else in mind?

@letmaik Any additional thoughts?

[letmaik]

@dmey You mean METGRID.TBL, not UNGRIB.TBL. And this would not be the issue, but rather selecting the right VTable file, which then would have to be done by the user.

Ideally, GIS4WRF would automatically detect the met data provider from the GRIB metadata and then select the right VTable file according to http://www2.mmm.ucar.edu/wrf/users/download/free_data.html -- I assume these are the only datasets people use and we don't want to support custom VTable files for now. Due to some limitations in the GRIB metadata file reading this is currently not possible, however.

In summary, yes, this would involve what @kikocorreoso described plus selecting the right VTable file manually, and probably dealing with a different folder structure.

[dmey]

@letmaik apologies, I actually meant Vtables -- there are many more Vtables currently available than the few you reported (see https://github.com/NCAR/WPS/tree/master/ungrib/Variable_Tables). We need to decide whether we want to allow users to be responsible in setting Vtables manually for now, or if we prefer to hold off with this feature until we can read metadata from GRIB.

Additinally, if we want to support all Vtables we will need to support a few utilities as well. For example if one uses the ECMWF one for sigma coordinates, one needs to run the calc_ecmwf_p.exe program as well.

[kikocorreoso]

@dmey @letmaik thanks for your thoughts.

It is super cool to have a tool that does some of the hard job but letting the tool like this restricts flexibility.

Providing some docs to help a user to use its own downloaded datasets and to choose a VTable among the most usual ones would be super helpful (IMHO). In the 'Simulation > Data' tab, in the meteorological data area could be a new button that opens a popup window where the user can select a folder (with the structure, names,..., as defined in the docs) and a VTable. This would provide a lot of flexibility to the users and I think the change in the code wouldn't be very high.

Additinally, if we want to support all Vtables we will need to support a few utilities as well. For example if one uses the ECMWF one for sigma coordinates, one needs to run the calc_ecmwf_p.exe program as well.

I'm not asking to support all the potential situations. For instance, GRIB support in windows is far from perfect and some experienced users know how to the the hard stuff by themselves,... I see GIS4WRF a perfect tool for newbies and education and its Windows support is pretty cool for that.

Thanks.

[kikocorreoso]

Also, for me, right now, from a windows box behind a proxy, I'm getting cert errors ¿?

requests.exceptions.SSLError: HTTPSConnectionPool(host='rda.ucar.edu', port=443): Max retries exceeded with url: /apps/metadata/ds083.3 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:749)'),)) 
Traceback (most recent call last):
  File "D:\QGIS\apps\Python36\lib\site-packages\urllib3\connectionpool.py", line 595, in urlopen
    self._prepare_proxy(conn)
  File "D:\QGIS\apps\Python36\lib\site-packages\urllib3\connectionpool.py", line 816, in _prepare_proxy
    conn.connect()
  File "D:\QGIS\apps\Python36\lib\site-packages\urllib3\connection.py", line 326, in connect
    ssl_context=context)
  File "D:\QGIS\apps\Python36\lib\site-packages\urllib3\util\ssl_.py", line 329, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "D:\QGIS\apps\Python36\lib\ssl.py", line 401, in wrap_socket
    _context=self, _session=session)
  File "D:\QGIS\apps\Python36\lib\ssl.py", line 808, in __init__
    self.do_handshake()
  File "D:\QGIS\apps\Python36\lib\ssl.py", line 1061, in do_handshake
    self._sslobj.do_handshake()
  File "D:\QGIS\apps\Python36\lib\ssl.py", line 683, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:749)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\QGIS\apps\Python36\lib\site-packages\requests\adapters.py", line 440, in send
    timeout=timeout
  File "D:\QGIS\apps\Python36\lib\site-packages\urllib3\connectionpool.py", line 668, in urlopen
    **response_kw)
  File "D:\QGIS\apps\Python36\lib\site-packages\urllib3\connectionpool.py", line 668, in urlopen
    **response_kw)
  File "D:\QGIS\apps\Python36\lib\site-packages\urllib3\connectionpool.py", line 668, in urlopen
    **response_kw)
  File "D:\QGIS\apps\Python36\lib\site-packages\urllib3\connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "D:\QGIS\apps\Python36\lib\site-packages\urllib3\util\retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='rda.ucar.edu', port=443): Max retries exceeded with url: /apps/metadata/ds083.3 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:749)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/U406328/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gis4wrf\plugin\ui\widget_met.py", line 126, in on_dataset_changed
    self.products = get_met_products(dataset_name, auth)
  File "C:/Users/U406328/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gis4wrf\core\downloaders\met.py", line 22, in get_met_products
    response = requests_retry_session().get(f'https://rda.ucar.edu/apps/metadata/{dataset_name}', auth=auth)
  File "D:\QGIS\apps\Python36\lib\site-packages\requests\sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "D:\QGIS\apps\Python36\lib\site-packages\requests\sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "D:\QGIS\apps\Python36\lib\site-packages\requests\sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "D:\QGIS\apps\Python36\lib\site-packages\requests\adapters.py", line 506, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='rda.ucar.edu', port=443): Max retries exceeded with url: /apps/metadata/ds083.3 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:749)'),))

Python version: 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 08:06:12) [MSC v.1900 64 bit (AMD64)] 
QGIS version: 3.2.0-Bonn Bonn, bc43194061

I've tried using verify=False in the requests call with no luck.

I also tried curl to access some rda.ucar.edu urls to get metadata as in GIS4WRF with no luck. With curl and python I have no problems to access other sites. Maybe is a problem of my network configuration...

So, for my case, an only entry point to download met data is very limiting. I can't use GIS4WRF in my company computer :-(

[dmey]

@kikocorreoso @letmaik That's fine, it may need a couple of iterations but, in general, we can allow to import external met dataset and then display them as a new tree node for all imported custom datasets, and further group them by their name and date-time.

@kikocorreoso I have moved the issue your reported when trying to download met data to a separate issue. Please see #25.

[kikocorreoso]

@dmey thanks!!

[letmaik]

@kikocorreoso I put a PR up for this: #28. I would be happy if you could look at the screenshots and let me know whether this would work for you. I opted for the simplest and reasonably flexible option I could think of. It should cover most scenarios.

The only thing it doesn't do very well currently is that if you have a folder containing many subfolders (let's say 1000) and you only want to add a few subfolders (let's say 20) then you would have to add each subfolder individually or add the parent folder (which recursively adds all files in it) and remove unwanted files again with the "Remove selected" button. This is because the folder selection dialog doesn't support selecting multiple folders. I don't know if this case is common or not. If it is, then it would make sense to switch from the system-native folder dialog to a non-native dialog that supports this (wouldn't be very hard, just looks a bit ugly).