Running EISCAT example on python 3.7 fails

ted123zwh commented 2 years ago

The following code from the example "EISCAT from Madrigal with Marking Tools"

import datetime
import geospacelab.express.eiscat_dashboard as eiscat

dt_fr = datetime.datetime.strptime('20201122' + '1800', '%Y%m%d%H%M')
dt_to = datetime.datetime.strptime('20201122' + '2300', '%Y%m%d%H%M')

site = 'UHF'
antenna = 'UHF'
modulation = 'ant'
load_mode = 'AUTO'
dashboard = eiscat.EISCATDashboard(
    dt_fr, dt_to, site=site, antenna=antenna, modulation=modulation, load_mode='AUTO', data_file_type='madrigal-hdf5'
)
dashboard.quicklook()

n_e = dashboard.assign_variable('n_e')
print(n_e.value)

dt_fr_2 = datetime.datetime.strptime('20201209' + '2030', "%Y%m%d%H%M")
dt_to_2 = datetime.datetime.strptime('20201210' + '0130', "%Y%m%d%H%M")
dashboard.add_vertical_line(dt_fr_2, bottom_extend=0, top_extend=0.02, label='Line 1', label_position='top')

dashboard.add_shading(dt_fr_2, dt_to_2, bottom_extend=0, top_extend=0.02, label='Shading 1', label_position='top')

dt_fr_3 = datetime.datetime.strptime('20201210' + '0130', "%Y%m%d%H%M")
dt_to_3 = datetime.datetime.strptime('20201210' + '0430', "%Y%m%d%H%M")
dashboard.add_top_bar(dt_fr_3, dt_to_3, bottom=0., top=0.02, label='Top bar 1')

dashboard.save_figure()

dashboard.show()

Failed with the following message:

Traceback (most recent call last):

  File "D:\Python\Python\PycharmProject\solar_eclipse\untitled0.py", line 201, in <module>
    dt_fr, dt_to, site=site, antenna=antenna, modulation=modulation, load_mode='AUTO', data_file_type='madrigal-hdf5'

  File "D:\Python\Anaconda\lib\site-packages\geospacelab\express\eiscat_dashboard.py", line 25, in __init__
    ds_1.load_data(load_mode=kwargs['load_mode'])

  File "D:\Python\Anaconda\lib\site-packages\geospacelab\datahub\sources\madrigal\isr\eiscat\__init__.py", line 109, in load_data
    self.check_data_files(**kwargs)

  File "D:\Python\Anaconda\lib\site-packages\geospacelab\datahub\__dataset_base__.py", line 352, in check_data_files
    self.search_data_files(**kwargs)

  File "D:\Python\Anaconda\lib\site-packages\geospacelab\datahub\sources\madrigal\isr\eiscat\__init__.py", line 277, in search_data_files
    done = self.download_data()

  File "D:\Python\Anaconda\lib\site-packages\geospacelab\datahub\sources\madrigal\isr\eiscat\__init__.py", line 294, in download_data
    data_file_root_dir=self.data_root_dir)

  File "D:\Python\Anaconda\lib\site-packages\geospacelab\datahub\sources\madrigal\isr\eiscat\downloader.py", line 72, in __init__
    self.download_madrigal_files(download_pp=download_pp)

  File "D:\Python\Anaconda\lib\site-packages\geospacelab\datahub\sources\madrigal\isr\eiscat\downloader.py", line 212, in download_madrigal_files
    "hdf5"

  File "D:\Python\Anaconda\lib\site-packages\madrigalWeb\madrigalWeb.py", line 1769, in downloadFile
    urlFile = urllib2.urlopen(url, timeout=TIMEOUT)

  File "D:\Python\Anaconda\lib\urllib\request.py", line 222, in urlopen
    return opener.open(url, data, timeout)

  File "D:\Python\Anaconda\lib\urllib\request.py", line 531, in open
    response = meth(req, response)

  File "D:\Python\Anaconda\lib\urllib\request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)

  File "D:\Python\Anaconda\lib\urllib\request.py", line 569, in error
    return self._call_chain(*args)

  File "D:\Python\Anaconda\lib\urllib\request.py", line 503, in _call_chain
    result = func(*args)

  File "D:\Python\Anaconda\lib\urllib\request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

HTTPError: Internal Server Error

Running on Windows10 and python-3.7.10

JouleCai commented 2 years ago

Hi,

It looks like the EISCAT/Madrigal server does not respond to the request. The inputs 'dt_fr' and 'dt_to' in the example are not correct. But they should not be associated with the error reported above. Now the example has been updated with the correct dates (https://github.com/JouleCai/geospacelab/blob/master/examples/demo_eiscat_quicklook.py). Please try to check the following things for your issue:

First, check your network connection and the connection to the EISCAT/Madrigal database (http://cedar.openmadrigal.org/).

Second, GeospaceLab depends on the madrigalWeb Python API (http://madrigal.haystack.mit.edu/madrigal/madDownload.html) for downloading the madrigal hdf5 files. You can test madrigalWeb using the examples on the webpage (http://cedar.openmadrigal.org/docs/name/rt_python.html). See if the examples work or not.

Third, the package is most tested and used in Linux or MacOS. Hence, there might be some incompatible issues in Windows. However, we know it was successfully run on Windows 10.

Alternatively, you can change the keyword data_file_type = 'madrigal-hdf5' to data_file_type = 'eiscat-hdf5' or just simply remove it on line 23 in the newly updated example. The package will try to download another type of EISCAT files from EISCAT schdule page.

Please also upgrade GeospaceLab to the latest version >= 0.4.12.

ted123zwh commented 2 years ago

After removing line 23, the example program can run normally, but after modifying the input 'dt_fr' and 'dt_to' in the sample file, new problems are encountered.

import datetime
import geospacelab.express.eiscat_dashboard as eiscat

dt_fr = datetime.datetime.strptime('20201211' + '0400', '%Y%m%d%H%M')
dt_to = datetime.datetime.strptime('20201211' + '1100', '%Y%m%d%H%M')

site = 'UHF'
antenna = 'UHF'
modulation = '60'
load_mode = 'AUTO'
dashboard = eiscat.EISCATDashboard(
    dt_fr, dt_to, site=site, antenna=antenna, modulation=modulation, load_mode='AUTO',
    # data_file_type = "madrigal-hdf5"
)
dashboard.quicklook()

# dashboard.save_figure() # comment this if you need to run the following codes
# dashboard.show()   # comment this if you need to run the following codes.

"""
As the dashboard class (EISCATDashboard) is a inheritance of the classes Datahub and TSDashboard.
The variables can be retrieved in the same ways as shown in Example 1. 
"""
n_e = dashboard.assign_variable('n_e')
print(n_e.value)
print(n_e.error)

"""
Several marking tools (vertical lines, shadings, and top bars) can be added as the overlays 
on the top of the quicklook plot.
"""
# add vertical line
# dt_fr_2 = datetime.datetime.strptime('20201209' + '2030', "%Y%m%d%H%M")
# dt_to_2 = datetime.datetime.strptime('20201210' + '0130', "%Y%m%d%H%M")
# dashboard.add_vertical_line(dt_fr_2, bottom_extend=0, top_extend=0.02, label='Line 1', label_position='top')
# # add shading
# dashboard.add_shading(dt_fr_2, dt_to_2, bottom_extend=0, top_extend=0.02, label='Shading 1', label_position='top')
# # add top bar
# dt_fr_3 = datetime.datetime.strptime('20201210' + '0130', "%Y%m%d%H%M")
# dt_to_3 = datetime.datetime.strptime('20201210' + '0430', "%Y%m%d%H%M")
# dashboard.add_top_bar(dt_fr_3, dt_to_3, bottom=0., top=0.02, label='Top bar 1')

# save figure
dashboard.save_figure()
# show on screen
dashboard.show()

Failed with the folling message:

Create a new figure: Figure(1000x800).
Cannot find the requested data file in G:\geospace_data\Madrigal\EISCAT\analyzed\UHF\2020
INFO: No experiments available!
Cannot find the requested data file in G:\geospace_data\Madrigal\EISCAT\analyzed\UHF\2020
INFO: No experiments available!
Cannot find files from the online database!
Cannot find files from the online database!
Traceback (most recent call last):

  File "D:\Python\Python\PycharmProject\solar_eclipse\untitled0.py", line 211, in <module>
    dt_fr, dt_to, site=site, antenna=antenna, modulation=modulation, load_mode='AUTO',

  File "D:\Python\Anaconda\lib\site-packages\geospacelab\express\eiscat_dashboard.py", line 25, in __init__
    ds_1.load_data(load_mode=kwargs['load_mode'])

  File "D:\Python\Anaconda\lib\site-packages\geospacelab\datahub\sources\madrigal\isr\eiscat\__init__.py", line 127, in load_data
    self.calc_lat_lon()

  File "D:\Python\Anaconda\lib\site-packages\geospacelab\datahub\sources\madrigal\isr\eiscat\__init__.py", line 206, in calc_lat_lon
    az = np.tile(az, (1, range.shape[1]))  # make az, el, range in the same shape

AttributeError: 'NoneType' object has no attribute 'shape'

And the old 'HTTPError: Internal Server Error' also occurs when running other example files like 'https://github.com/JouleCai/geospacelab/blob/master/examples/demo_millstonehill_isr.py' failed with the message:

Create a new figure: Figure(1000x800).
Cannot find the requested data file in G:\geospace_data\Madrigal\MillstoneHill_ISR\2016\20160314
Searching data from the Madrigal database ...
Searching files from the experiment IS World Day Regional Westward Flow ...
Downloading  mlh160314n.006.hdf5 from the Madrigal database ...
http://millstonehill.haystack.mit.edu/

Traceback (most recent call last):

  File "D:\Python\Python\PycharmProject\solar_eclipse\untitled0.py", line 336, in <module>
    example_combined_datasets()

  File "D:\Python\Python\PycharmProject\solar_eclipse\untitled0.py", line 296, in example_combined_datasets
    datasource_contents=['madrigal', 'isr', 'millstonehill', 'vi'])

  File "D:\Python\Anaconda\lib\site-packages\geospacelab\datahub\__init__.py", line 193, in dock
    dataset = getattr(module, 'Dataset')(**kwargs)

  File "D:\Python\Anaconda\lib\site-packages\geospacelab\datahub\sources\madrigal\isr\millstonehill\vi\__init__.py", line 89, in __init__
    self.load_data()

  File "D:\Python\Anaconda\lib\site-packages\geospacelab\datahub\sources\madrigal\isr\millstonehill\vi\__init__.py", line 105, in load_data
    self.check_data_files(**kwargs)

  File "D:\Python\Anaconda\lib\site-packages\geospacelab\datahub\__dataset_base__.py", line 352, in check_data_files
    self.search_data_files(**kwargs)

  File "D:\Python\Anaconda\lib\site-packages\geospacelab\datahub\sources\madrigal\isr\millstonehill\vi\__init__.py", line 143, in search_data_files
    done = self.download_data()

  File "D:\Python\Anaconda\lib\site-packages\geospacelab\datahub\sources\madrigal\isr\millstonehill\vi\__init__.py", line 169, in download_data
    exp_name_pattern=self.exp_name_pattern)

  File "D:\Python\Anaconda\lib\site-packages\geospacelab\datahub\sources\madrigal\isr\millstonehill\downloader.py", line 95, in __init__
    self.download_madrigal_files()

  File "D:\Python\Anaconda\lib\site-packages\geospacelab\datahub\sources\madrigal\isr\millstonehill\downloader.py", line 182, in download_madrigal_files
    "hdf5"

  File "D:\Python\Anaconda\lib\site-packages\madrigalWeb\madrigalWeb.py", line 1769, in downloadFile
    urlFile = urllib2.urlopen(url, timeout=TIMEOUT)

  File "D:\Python\Anaconda\lib\urllib\request.py", line 222, in urlopen
    return opener.open(url, data, timeout)

  File "D:\Python\Anaconda\lib\urllib\request.py", line 531, in open
    response = meth(req, response)

  File "D:\Python\Anaconda\lib\urllib\request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)

  File "D:\Python\Anaconda\lib\urllib\request.py", line 569, in error
    return self._call_chain(*args)

  File "D:\Python\Anaconda\lib\urllib\request.py", line 503, in _call_chain
    result = func(*args)

  File "D:\Python\Anaconda\lib\urllib\request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

HTTPError: Internal Server Error

It looks like there is no problem with my network connenction and the connection to the eiscat/madrigal database. And the madrigalWeb examples worked after changing madrigalUrl from 'http://madrigal.haystack.mit.edu' to 'http://millstonehill.haystack.mit.edu/'.

JouleCai commented 2 years ago

Hi,

As seen from the EISCAT schedule webpage (https://portal.eiscat.se/schedule/?year=2020&month=12&A=on&TRO=on&UHF=on), there were three UHF experiments on the same day (2020-12-11). The data in those experiments are stored separately in three hdf5 files, and thus the file name are different (see https://madrigal.eiscat.se/madrigal/showExperiment/?experiment_list=20010420&show_plots). For this case, the inputs should be set carefully, e.g., the corresponding times for each experiment, pulse code (pulse_code), and modulation (modulation).

If you feel difficult to determine those inputs, alternatively, you can do the following steps:

Download manually the hdf5 file that stores the experiment you are intersted in from the madrigal database . The EISCAT hdf5 file can be also an output by your own GUISDAP analysis.
In the script, change load_mode = 'AUTO' to load_mode = 'dialog',
Run the program and select the hdf5 file downloaded, following the prompt message.

ted123zwh commented 2 years ago

Thanks, I found that the example code can work on Linux , so it might be an incompatibility issue on Windows.

JouleCai / geospacelab

Running EISCAT example on python 3.7 fails #7