Running EISCAT example on python 3.9 fails

[danielk333]

The following code from the readme:

import datetime
import geospacelab.express.eiscat_viewer as eiscat

dt_fr = datetime.datetime.strptime('20201209' + '1800', '%Y%m%d%H%M')
dt_to = datetime.datetime.strptime('20201210' + '0600', '%Y%m%d%H%M')

# check the eiscat-hdf5 filename from the EISCAT schedule page, e.g., "EISCAT_2020-12-10_beata_60@uhfa.hdf5"
site = 'UHF'
antenna = 'UHF'
modulation = '60'
load_mode = 'AUTO'
# The code will download and load the data automatically as long as the parameters above are set correctly.
viewer = eiscat.EISCATViewer(
      dt_fr, dt_to, site=site, antenna=antenna, modulation=modulation, load_mode='AUTO'
)
viewer.quicklook()
# viewer.save_figure() # comment this if you need to run the following codes
# viewer.show()   # comment this if you need to run the following codes.

"""
The viewer is an instance of the class EISCATViewer, which is a heritage of the class Datahub.
Thus, the variables can be retrieved in the same ways as shown in Example 1. 
"""
n_e = viewer.get_variable('n_e')

"""
Several marking tools (vertical lines, shadings, and top bars) can be added as the overlays 
on the top of the quicklook plot.
"""
# add vertical line
dt_fr_2 = datetime.datetime.strptime('20201209' + '2030', "%Y%m%d%H%M")
dt_to_2 = datetime.datetime.strptime('20201210' + '0130', "%Y%m%d%H%M")
viewer.add_vertical_line(dt_fr_2, bottom_extend=0, top_extend=0.02, label='Line 1', label_position='top')
# add shading
viewer.add_shading(dt_fr_2, dt_to_2, bottom_extend=0, top_extend=0.02, label='Shading 1', label_position='top')
# add top bar
dt_fr_3 = datetime.datetime.strptime('20201210' + '0130', "%Y%m%d%H%M")
dt_to_3 = datetime.datetime.strptime('20201210' + '0430', "%Y%m%d%H%M")
viewer.add_top_bar(dt_fr_3, dt_to_3, bottom=0., top=0.02, label='Top bar 1')

# save figure
viewer.save_figure()
# show on screen
viewer.show()

Failed with the following message:

Traceback (most recent call last):
  File "/home/danielk/python_scripts/geospacelab/eiscat_test.py", line 13, in <module>
    viewer = eiscat.EISCATViewer(
  File "/home/danielk/venvs/geospacelab/lib/python3.9/site-packages/geospacelab/express/eiscat_viewer.py", line 20, in __init__
    super().__init__(dt_fr=dt_fr, dt_to=dt_to, new_figure=new_figure)
  File "/home/danielk/venvs/geospacelab/lib/python3.9/site-packages/geospacelab/visualization/mpl/ts_viewer.py", line 127, in __init__
    super().__init__(visual='on', canvas=canvas, canvas_config=canvas_config, **kwargs)
  File "/home/danielk/venvs/geospacelab/lib/python3.9/site-packages/geospacelab/visualization/mpl/dashboards.py", line 50, in __init__
    super(Dashboard, self).__init__(figure=figure, figure_config=figure_config, **kwargs)
  File "/home/danielk/venvs/geospacelab/lib/python3.9/site-packages/geospacelab/datahub/__init__.py", line 71, in __init__
    super().__init__(**kwargs)
  File "/home/danielk/venvs/geospacelab/lib/python3.9/site-packages/geospacelab/visualization/mpl/_base.py", line 135, in __init__
    super().__init__(**kwargs)
TypeError: object.__init__() takes exactly one argument (the instance to initialize)

Running on manjaro arch variant and python-3.9.7

[JouleCai]

Hi, sorry for this issue. The scripts shown in README.md were old. New updates have been made. Now you will find the new examples in README.md and more in the ./examples folder. Please upgrade the package to v0.3.1 or later to run the codes.

[danielk333]

No worries and nice update! The omni example now works but the eiscat example still fails for me with:

Create a new figure: Figure(1000x800).
Load IGRF coefficients ...
Traceback (most recent call last):
  File "/home/danielk/venvs/geospacelab/lib/python3.9/site-packages/aacgmv2/wrapper.py", line 364, in convert_latlon_arr
    c_aacgmv2.set_datetime(dtime.year, dtime.month, dtime.day, dtime.hour,
RuntimeError: AACGM_v2_SetDateTime returned error code -2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/danielk/python_scripts/geospacelab/eiscat_test.py", line 11, in <module>
    dashboard = eiscat.EISCATDashboard(
  File "/home/danielk/venvs/geospacelab/lib/python3.9/site-packages/geospacelab/express/eiscat_dashboard.py", line 23, in __init__
    ds_1.load_data(load_mode=kwargs['load_mode'])
  File "/home/danielk/venvs/geospacelab/lib/python3.9/site-packages/geospacelab/datahub/sources/madrigal/eiscat/__init__.py", line 126, in load_data
    self.calc_lat_lon()
  File "/home/danielk/venvs/geospacelab/lib/python3.9/site-packages/geospacelab/datahub/sources/madrigal/eiscat/__init__.py", line 226, in calc_lat_lon
    cs_new = cs_new.to_AACGM()
  File "/home/danielk/venvs/geospacelab/lib/python3.9/site-packages/geospacelab/cs/_geo.py", line 28, in to_AACGM
    cs_new = cs_geoc.to_AACGM(append_mlt=append_mlt)
  File "/home/danielk/venvs/geospacelab/lib/python3.9/site-packages/geospacelab/cs/_geo.py", line 396, in to_AACGM
    lat[ind_dt], lon[ind_dt], r[ind_dt] = aacgm.convert_latlon_arr(in_lat=self.coords.lat[ind_dt],
  File "/home/danielk/venvs/geospacelab/lib/python3.9/site-packages/aacgmv2/wrapper.py", line 367, in convert_latlon_arr
    raise RuntimeError("cannot set time for {:}: {:}".format(dtime, err))
RuntimeError: cannot set time for 2020-12-09 13:05:42.503096: AACGM_v2_SetDateTime returned error code -2

[JouleCai]

Hi, this is an issue with the aacgmv2 package. I noticed that when the package is run with Pycharm or Vs code, the error will appear in the beginning. Spyder always works fine for this issue. And after I compile with Spyder several times, the codes work in Pycharm and VS code, too. My impression is that the C codes in the AACGMv2 are not compiled for some reason in Pycharm or vs code.

[JouleCai]

I consider using "try" when converting the az, el, range to aacgm coords in next version.

[danielk333]

I just ran this vanilla from the command-line in a python3 virtual environment, no conda, pycharm, vs-code or other IDE. When i try to use AACGMv2 from the interpreter it works just fine though

(geospacelab) danielk@IRF033-danielk ~/p/geospacelab> python
Python 3.9.7 (default, Aug 31 2021, 13:28:12) 
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import aacgmv2
>>> import datetime as dt
>>> import numpy as np
>>> np.set_printoptions(formatter={'float_kind': lambda x:'{:.4f}'.format(x)})
>>> dtime = dt.datetime(2013, 11, 3)
>>> np.array(aacgmv2.get_aacgm_coord(60, 15, 300, dtime))
array([57.4736, 93.6111, 1.4816])
>>> np.array2string(np.array(aacgmv2.convert_latlon_arr([90, -90], 0, 0, dtime, method_code="A2G"))).replace('\n', '')
'[[82.9686 -74.3390] [-84.6501 125.8476] [14.1246 12.8772]]'
>>> np.array(aacgmv2.convert_mlt([1.4822189, 12], dtime, m2a=True))
array([93.6203, -108.6130])
>>> 

also when i call it with keywords and "G2A" (as is done in _geo.py) it works too

>>> print(aacgmv2.convert_latlon_arr(in_lat=45, in_lon=12, height=0, dtime=dtime, method_code="G2A"))
(array([39.11720254]), array([86.55531691]), array([0.99941762]))

So there is something strange going on here :sweat_smile:

[JouleCai]

Right, accgmv2 works fine from the python console. Our students use Spyder and they never reported this issue. Actually, I often use Pycharm for coding. When I met this problem, I usually did the following things:

 1. Upgrade the aacgmv2 package.
 2. Run the codes in Spyder several times and restart the computer. In most cases, it solves the problem. Unkown reason.
 3. Reinstall gcc in the virtual environment using conda.

This is not friendly for the users. I try to update next time. Also, I have submitted this issue to github/aacgmv2. Hopefully, they have some suggestions to solve it.

[danielk333]

Awesome! Thanks! lets leave this open then until aburrell/aacgmv2#73 is resolved then

[JouleCai]

Hi,

Today I randomly found that installing "gcc" in the conda virtual environment can solve the problem:

conda install -c conda-forge gcc

The problem may be due to that some of the IDEs (like pycharm or vs code) cannot call the gcc compiler properly. There might be some associated options in the IDE preferences, which I'm not aware of.

[danielk333]

Hm, thats strange since i didnt use any IDE (just run a python script from the commandline) to produce this problem so it should not have any problem calling gcc.

As far as i can tell geospacelab does not do any extension or compiling itself so i did a pull and (im now on python 3.10) did a new test: Omni-example work Eiscat-example still fails with same error code

Just for fun i tried to print the input arguemnts in _geo.py and paste them into a file that does the exact same thing

import aacgmv2
import datetime as dt
import numpy as np
np.set_printoptions(formatter={'float_kind': lambda x: '{:.4f}'.format(x)})

in_dat = {'in_lat': np.array([69.30828641, 69.30334279, 69.29760738, 69.29220118, 69.28662164,
       69.28102827, 69.27505365, 69.26839548, 69.26310337, 69.25740369,
       69.25197769, 69.24588282, 69.23811373, 69.22787005, 69.21501042,
       69.20152363, 69.18630583, 69.16880848, 69.14947964, 69.128603  ,
       69.10523979, 69.08188808, 69.05619986, 69.02809153, 68.99838343,
       68.96623403, 68.93435919, 68.89898638, 68.86349659, 68.82568235,
       68.78809369, 68.74731084, 68.70681364, 68.66422558, 68.61859198,
       68.57396273, 68.52728645, 68.47962782, 68.43112162, 68.37954269,
       68.32730632, 68.28951278]), 'in_lon': np.array([19.16705994, 19.1649374 , 19.16247613, 19.16015734, 19.15776543,
       19.15536885, 19.15281029, 19.14996069, 19.14769701, 19.14526025,
       19.14294169, 19.14033872, 19.13702288, 19.13265454, 19.12717657,
       19.12143847, 19.11497254, 19.10754934, 19.09936314, 19.0905379 ,
       19.08068171, 19.07085166, 19.06006252, 19.04828619, 19.03587275,
       19.0224774 , 19.00923546, 18.99458559, 18.97993486, 18.96437676,
       18.94896454, 18.93230217, 18.91581748, 18.89854681, 18.88011482,
       18.86216168, 18.84346206, 18.82444962, 18.80518216, 18.78478562,
       18.76422438, 18.74940755]), 'height': np.array([ 64.90901924,  67.54535465,  70.60671404,  73.49508478,
        76.47882962,  79.47280341,  82.67400882,  86.24527084,
        89.0866981 ,  92.14982962,  95.06863301,  98.35045142,
       102.5387006 , 108.06946378, 115.02630643, 122.33885541,
       130.61020175, 140.14722286, 150.71576455, 162.17005418,
       175.03747935, 187.95037629, 202.21558424, 217.89747123,
       234.555193  , 252.67890027, 270.748363  , 290.91896147,
       311.28235361, 333.11973266, 354.97172794, 378.84558287,
       402.72400577, 428.02185662, 455.34354413, 482.28147039,
       510.68790776, 539.94090152, 569.97572769, 602.20641555,
       635.1603415 , 659.2014026 ]), 'dtime': dt.datetime(2020, 12, 9, 13, 5, 42, 503096), 'method_code': 'G2A'}

print(np.array2string(
    np.array(
        aacgmv2.convert_latlon_arr(**in_dat)
    )
).replace('\n', ''))

This files runs normally

(geospacelab) danielk@IRF033-danielk ~/p/geospacelab> python test_aacgmv2.py
[[66.6764 66.6763 66.6762 66.6761 66.6760 66.6759 66.6757 66.6756 66.6755  66.6755 66.6754 66.6753 66.6752 66.6751 66.6749 66.6748 66.6748 66.6747  66.6747 66.6748 66.6750 66.6752 66.6756 66.6762 66.6769 66.6778 66.6789  66.6803 66.6819 66.6839 66.6860 66.6887 66.6916 66.6949 66.6988 66.7030  66.7077 66.7129 66.7185 66.7250 66.7321 66.7375] [100.5244 100.5245 100.5245 100.5245 100.5245 100.5246 100.5246 100.5246  100.5247 100.5247 100.5247 100.5248 100.5248 100.5249 100.5250 100.5251  100.5252 100.5254 100.5256 100.5258 100.5260 100.5263 100.5266 100.5270  100.5275 100.5280 100.5285 100.5292 100.5299 100.5307 100.5316 100.5326  100.5336 100.5348 100.5361 100.5375 100.5391 100.5407 100.5425 100.5445  100.5467 100.5483] [1.0083 1.0088 1.0092 1.0097 1.0102 1.0106 1.0111 1.0117 1.0121 1.0126  1.0131 1.0136 1.0143 1.0151 1.0162 1.0174 1.0187 1.0202 1.0218 1.0236  1.0256 1.0277 1.0299 1.0324 1.0350 1.0378 1.0407 1.0438 1.0470 1.0505  1.0539 1.0576 1.0614 1.0654 1.0697 1.0739 1.0783 1.0829 1.0877 1.0927  1.0979 1.1017]]

only difference between this and the geospacelab is that i used datetime.datetime rather than real_datetime but that should not matter. So it somehow related to HOW geospacelab calls aacgmv2 or how aacgmv2 handles external calls.

Looking at the C-code implementation in aacgm the error code 2 cited is:

    case 2: /* COEF Path not set */
  fprintf(stderr,
  "* AACGM-v2 ERROR: AACGM_v2_DAT_PREFIX path not set *\n"
  "*                                                                        *\n"
  "* You must set the environment variable AACGM_v2_DAT_PREFIX to the       *\n"
  "\n");
    break;
  }
  fprintf(stderr,
  "**************************************************************************"
  "\n\n");

but it does not print that, but in the AACGM_v2_LoadCoef the -2 error is defined as not being able to load coefficients. So it seems the path might be set but it cant load it anyway.

So probably its something with the compiled C-code not finding the AACGM data files when being called from an external library.

[danielk333]

This function might lead somewhere https://github.com/aburrell/aacgmv2/blob/main/aacgmv2/wrapper.py#L110 and these defines https://github.com/aburrell/aacgmv2/blob/main/aacgmv2/__init__.py#L46 but using the set function in _geo.py does nothing. Well thats enough investigating for me!

[JouleCai]

Hi,

This looks like a new error, which is different from the top. Have you tried to upgrade or reinstall the package aacgmv2?

[danielk333]

Yep, still same error

(geospacelab) danielk@IRF033-danielk ~/g/geospacelab  master
> pip uninstall aacgmv2
Found existing installation: aacgmv2 2.6.2
Uninstalling aacgmv2-2.6.2:
  Would remove:
    /home/danielk/venvs/geospacelab/bin/aacgmv2
    /home/danielk/venvs/geospacelab/lib/python3.10/site-packages/aacgmv2
    /home/danielk/venvs/geospacelab/lib/python3.10/site-packages/aacgmv2-2.6.2-py3.10.egg-info
Proceed (Y/n)? 
  Successfully uninstalled aacgmv2-2.6.2
(geospacelab) danielk@IRF033-danielk ~/g/geospacelab  master
> pip install aacgmv2
Collecting aacgmv2
  Using cached aacgmv2-2.6.2.tar.gz (1.6 MB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: numpy in /home/danielk/venvs/geospacelab/lib/python3.10/site-packages (from aacgmv2) (1.22.3)
Using legacy 'setup.py install' for aacgmv2, since package 'wheel' is not installed.
Installing collected packages: aacgmv2
  Running setup.py install for aacgmv2 ... done
Successfully installed aacgmv2-2.6.2
(geospacelab) danielk@IRF033-danielk ~/g/geospacelab  master
> python ~/python_scripts/geospacelab/eiscat_test.py 
Create a new figure: Figure(1000x800).
Load IGRF coefficients ...
Traceback (most recent call last):
  File "/home/danielk/venvs/geospacelab/lib/python3.10/site-packages/aacgmv2/wrapper.py", line 364, in convert_latlon_arr
    c_aacgmv2.set_datetime(dtime.year, dtime.month, dtime.day, dtime.hour,
RuntimeError: AACGM_v2_SetDateTime returned error code -2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/danielk/python_scripts/geospacelab/eiscat_test.py", line 11, in <module>
    dashboard = eiscat.EISCATDashboard(
  File "/home/danielk/venvs/geospacelab/lib/python3.10/site-packages/geospacelab/express/eiscat_dashboard.py", line 25, in __init__
    ds_1.load_data(load_mode=kwargs['load_mode'])
  File "/home/danielk/venvs/geospacelab/lib/python3.10/site-packages/geospacelab/datahub/sources/madrigal/eiscat/__init__.py", line 126, in load_data
    self.calc_lat_lon()
  File "/home/danielk/venvs/geospacelab/lib/python3.10/site-packages/geospacelab/datahub/sources/madrigal/eiscat/__init__.py", line 226, in calc_lat_lon
    cs_new = cs_new.to_AACGM()
  File "/home/danielk/venvs/geospacelab/lib/python3.10/site-packages/geospacelab/cs/_geo.py", line 28, in to_AACGM
    cs_new = cs_geoc.to_AACGM(append_mlt=append_mlt)
  File "/home/danielk/venvs/geospacelab/lib/python3.10/site-packages/geospacelab/cs/_geo.py", line 397, in to_AACGM
    lat[ind_dt], lon[ind_dt], r[ind_dt] = aacgm.convert_latlon_arr(in_lat=self.coords.lat[ind_dt],
  File "/home/danielk/venvs/geospacelab/lib/python3.10/site-packages/aacgmv2/wrapper.py", line 367, in convert_latlon_arr
    raise RuntimeError("cannot set time for {:}: {:}".format(dtime, err))
RuntimeError: cannot set time for 2020-12-09 13:05:42.503096: AACGM_v2_SetDateTime returned error code -2
(geospacelab) danielk@IRF033-danielk ~/g/geospacelab  master

im semi-confident this is something to do with the way the C-code is implemented in aacgmv2 so that it breaks when being called from within another package

[JouleCai]

I agree with you. Very strange. So far, I have no idea how to explain it. I will have another test next.

[JouleCai]

Now I find the problem that is associated with cython. The package should be installed before installing aacgmv2. I don't know the exact reason. It seems that when aacgmv2 is installed, its c files are built into a .so library, and probably cython plays a role in building the c source files. I have added cython in the setup.py file.

So, please create a new virtual environment and install the latest version of geospacelab (>=0.4.10). If it doesn't work, please try to install cython when creating the conda virtual environment:

conda create -n test python=3.9 cython

I have tested both ways on my ubuntu computer. Hopefully, it works for you for this time.

Best regards, Lei

[danielk333]

Nice that you found a fix for the conda environment! I tried it in just native virtual environment (I dont have conda) and i still get the same error (i tried to install cython both as a system package but also locally) so i guess its just something weird with the setup. If I wanna use geospacelab ill just run it in a ubuntu docker with conda and it'll work :+1: