Closed rhroberts closed 3 years ago
Great, thank you @rhroberts ! I'll get to it within a few days. I've been meaning to issue a new release, so this will be a good motivator.
Thanks, @emiliom! Much appreciated!
Note: Probably should add from . import noaa
to ulmo/__init__.py
. I'll look closer.
@rhroberts sorry for taking so long to review this PR. I'm finally working on it.
I tested your PR locally by running the goes
tests. All but one of the tests succeeded. The one that fails is test_parse_dcp_message_timestamp
. The final error message is:
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='dcs1.noaa.gov', port=443):
Max retries exceeded with url:
/Account/FieldTestData?addr=C5149430&hours=12
(Caused by SSLError(SSLError("bad handshake: SysCallError(32, 'EPIPE')",),))
and the stack trace looks something like this:
test/noaa_goes_test.py:32:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ulmo/noaa/goes/core.py:112: in get_data
messages = _fetch_url(params)
ulmo/noaa/goes/core.py:137: in _fetch_url
r = requests.post(dcs_url, params=params, timeout=60)
/home/mayorga/.local/lib/python3.6/site-packages/requests/api.py:119: in post
return request('post', url, data=data, json=json, **kwargs)
/home/mayorga/.local/lib/python3.6/site-packages/requests/api.py:61: in request
return session.request(method=method, url=url, **kwargs)
/home/mayorga/.local/lib/python3.6/site-packages/requests/sessions.py:530: in request
resp = self.send(prep, **send_kwargs)
/home/mayorga/.local/lib/python3.6/site-packages/requests/sessions.py:643: in send
r = adapter.send(request, **kwargs)
Examining noaa_goes_test.py
, it looks like test_parse_dcp_message_timestamp
would try to loop through the 3 list elements in message_test_sets
, failing on the first one. I tried to see if the other two might work by commenting out first the first element, then both the first and second elements, but the same error type occurred.
Are you able to run that test successfully?
Note: As you can see this was run with the Python 3.6 conda environment created by conda_environment.yml
. I've been reworking the environment creation workflow to, among other things, remove dependencies no longer used and remove old version pinning that were causing the Python version to be locked at 3.6. Let me know what Python version you used when running your tests.
Thanks for checking it out @emiliom! I seem to recall running into this test failure awhile back, but I wasn't able to replicate it when I tested today using the conda_environment.yml
with Python 3.6.13. I'll dig into it more on Monday though.
Output of pip list
for reference:
Package Version Location
----------------------------- ------------------- ----------------------
alabaster 0.7.12
appdirs 1.4.4
attrs 20.3.0
Babel 2.9.0
beautifulsoup4 4.9.3
brotlipy 0.7.0
certifi 2020.12.5
cffi 1.14.5
chardet 4.0.0
colorama 0.4.4
coverage 5.5
coveralls 3.0.1
cryptography 3.4.6
docopt 0.6.2
docutils 0.16
flake8 3.9.0
freezegun 1.1.0
future 0.18.2
geojson 2.5.0
html5lib 0.9999999
httpretty 0.8.10
idna 2.10
imagesize 1.2.0
importlib-metadata 3.7.3
iniconfig 1.1.1
isodate 0.6.0
Jinja2 2.11.3
lxml 4.6.2
MarkupSafe 1.1.1
mccabe 0.6.1
mock 4.0.3
more-itertools 8.7.0
numexpr 2.7.3
numpy 1.19.5
numpydoc 1.1.0
packaging 20.9
pandas 1.0.5
pip 21.0.1
pluggy 0.13.1
py 1.10.0
pycodestyle 2.6.0
pycparser 2.20
pyflakes 2.2.0
Pygments 2.8.1
pyOpenSSL 20.0.1
pyparsing 2.4.7
PySocks 1.7.1
pytest 6.2.2
pytest-cov 2.11.1
pytest-runner 5.3.0
python-dateutil 2.8.1
pytz 2021.1
PyYAML 5.4.1
requests 2.25.1
setuptools 49.6.0.post20210108
six 1.15.0
snowballstemmer 2.1.0
soupsieve 2.0.1
Sphinx 3.5.2
sphinxcontrib-applehelp 1.0.2
sphinxcontrib-devhelp 1.0.2
sphinxcontrib-htmlhelp 1.0.3
sphinxcontrib-jsmath 1.0.1
sphinxcontrib-qthelp 1.0.3
sphinxcontrib-serializinghtml 1.1.4
suds-jurko 0.6
tables 3.6.1
toml 0.10.2
typing-extensions 3.7.4.3
ulmo 0.8.7.dev0 /home/ubuntu/Code/ulmo
urllib3 1.26.4
webencodings 0.5.1
wheel 0.36.2
zipp 3.4.1
Hey @emiliom, I'm still not able to replicate this test failure on my machine. Based on this issue, it looks like it might be an issue with the versions of HTTPretty
and pyOpenSSL
being used. The HTTPretty
dependency is from the tox.ini
, not from conda_environment.yml
, so it might be that we have different versions installed when running the tests? Could you let me know what versions you are using? Thanks!
Thanks. I'm going to try running the tests using the new dependency setup I've implemented, using requirements.txt
and requirements-dev.txt
, where I've removed the pinning of httpretty to a very old version. I'll report back. Regardless, I'm glad to hear you're able to successfully run the tests.
Since the tests work for you, and the test set up for ulmo is already improved, I'll just merge your PR and re-run the goes tests after merging. I'll keep you posted via #178. Thanks again!
Awesome, thanks Emilio!
Finally getting around to finishing this :) I picked up where Paul left off since he is no longer with TWDB.
usgs.eddn
module has been removednoaa.goes
module and associated unit tests have been added to replace it173 can be closed in favor of this PR. Thanks!