search

spedas / pyspedas

Python-based Space Physics Environment Data Analysis Software
https://pyspedas.readthedocs.io/
MIT License
149 stars 58 forks source link

mth5 import test fails on github #846

Closed jameswilburlewis closed 4 months ago

jameswilburlewis commented 5 months ago

It's expecting to see an exception, but evidently it's not being raised. This is with the mth5 package in the environment.

Starting MTH5 import test at Fri May 3 06:19:24 UTC 2024
F
======================================================================
FAIL: test_pyspedas_mth5_import_error (__main__.TestImportErrorHandling.test_pyspedas_mth5_import_error)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/pyspedas/pyspedas/pyspedas/mth5/tests/test_mth5_import.py", line 27, in test_pyspedas_mth5_import_error
    with self.assertRaises(ImportError):
AssertionError: ImportError not raised

It might be something to do with this:

self.logging_patch = patch('pyspedas.logging.error', new_callable=MagicMock)

logging isn't in the pyspedas namespace in the master branch -- I had to change that in a few other places.

xandrd commented 5 months ago

Hi @jameswilburlewis,

I thought pyspedas.logging was the main way to output the log reports to console.

Also, I think I should redesign the test to be more robust. I had similar issue but it disappeared after. This test checks if external mth5 module is not present and the ImportError error should appear. Also, external mth5 module should not be a part of pyspedas requirement. I will take a look how the test can be performed.

jameswilburlewis commented 5 months ago

We just "import logging" wherever it's needed in pyspedas or pytplot, rather than bringing it into the pyspedas namespace.

I've taken mth5 back out of the requirements.txt file -- it's now being explicitly installed when github starts the test script.

jameswilburlewis commented 5 months ago

The import test passed this time, but now another test is failing:

======================================================================
ERROR: test03_load_fdsn_nowarnings (__main__.TestMTH5LoadFDSN.test03_load_fdsn_nowarnings)
Test with and without nowarning flag
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/pyspedas/pyspedas/pyspedas/mth5/tests/test_load_fdsn.py", line 139, in test03_load_fdsn_nowarnings
    load_fdsn(network="4P", station="REU49", trange=[date_start, date_end], nowarnings=True)
  File "/home/runner/work/pyspedas/pyspedas/pyspedas/mth5/load_fdsn.py", line 140, in load_fdsn
    mth5_path = fdsn_object.make_mth5_from_fdsn_client(request_df, interact=False, path=mth5dir)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/mth5/clients/fdsn.py", line 467, in make_mth5_from_fdsn_client
    inv, streams = self.get_inventory_from_df(df, self.client)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/mth5/clients/fdsn.py", line 661, in get_inventory_from_df
    cha_inv = client.get_stations(
              ^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/obspy/clients/fdsn/client.py", line 756, in get_stations
    inventory = read_inventory(data_stream, format='STATIONXML')
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/obspy/core/util/decorator.py", line 297, in _map_example_filename
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/obspy/core/inventory/inventory.py", line 91, in read_inventory
    return _generic_reader(path_or_file_object, _read, format=format,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/obspy/core/util/base.py", line 624, in _generic_reader
    generic = callback_func(pathname_or_url, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/obspy/core/util/decorator.py", line 142, in uncompress_file
    return func(filename, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/obspy/core/inventory/inventory.py", line 100, in _read
    inventory, format = _read_from_plugin('inventory', filename, format=format,
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/obspy/core/util/base.py", line 423, in _read_from_plugin
    list_obj = read_format(filename, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/obspy/io/stationxml/core.py", line 152, in _read_stationxml
    root = etree.parse(path_or_file_object).getroot()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/lxml/etree.pyx", line 3570, in lxml.etree.parse
  File "src/lxml/parser.pxi", line 1969, in lxml.etree._parseDocument
  File "src/lxml/parser.pxi", line 1989, in lxml.etree._parseMemoryDocument
  File "src/lxml/parser.pxi", line 1876, in lxml.etree._parseDoc
  File "src/lxml/parser.pxi", line 1164, in lxml.etree._BaseParser._parseDoc
  File "src/lxml/parser.pxi", line 633, in lxml.etree._ParserContext._handleParseResultDoc
  File "src/lxml/parser.pxi", line 743, in lxml.etree._handleParseResult
  File "src/lxml/parser.pxi", line 672, in lxml.etree._raiseParseError
  File "<string>", line 1
lxml.etree.XMLSyntaxError: Start tag expected, '<' not found, line 1, column 1
----------------------------------------------------------------------
Ran 23 tests in 71.[733](https://github.com/spedas/pyspedas/actions/runs/8979872004/job/24662592681#step:8:733)s
FAILED (errors=1, skipped=2)
xandrd commented 4 months ago

Hm... I was not able to preproduce this error so far. I will keep investigating.

xandrd commented 4 months ago

I looked into github actions is it seems that all tests have passed.

The tests reply on the internet connection to the fdsn servers. Perhaps it is not ideal, but this is the only way to test that the code works as intended. Maybe during the "failed" run the fdsn servers were not available, or provided "bad" data.

If the tests have passed perhaps we can close this issue.

jameswilburlewis commented 4 months ago

Yep, I guess it was a transient error -- it's getting past the MTH5 tests now (failing in MMS instead) so we can close this issue.