search

dtcenter / METplotpy

https://metplotpy.readthedocs.io/en/latest/index.html
Apache License 2.0
12 stars 7 forks source link

Ensure METplotpy installs on NOAA machines #143

Closed jprestop closed 1 year ago

jprestop commented 3 years ago

Ensure that METplotpy installs on the following NOAA machines:

Describe the Task

Julie suggests a working session with Minna and Hank to guide installation and troubleshooting on the above mentioned machines.

Time Estimate

1 to 3 days of work

Sub-Issues

Consider breaking the task down into sub-issues.

Relevant Deadlines

After each beta release in order to have everything ready to go for the official release. After the first successful installation, guidance from Minna and Hank may not be necessary.

Funding Source

2793541

Define the Metadata

Assignee

Labels

Projects and Milestone

Define Related Issue(s)

Consider the impact to the other METplus components.

Task Checklist

See the METplus Workflow for details.

jprestop commented 3 years ago

From Logan from the [20210823 METplus User's Telecon](20210823 METplus User's Telecon):

Could set up a virtual python environment on Orion, but he’ll be working more on that in the coming weeks. A higher priority is getting METplotpy/calcpy setup and running on Hera. This won’t be on the radar for WCOSS2 until 2022.

jprestop commented 3 years ago

I was unable to begin installing on Hera due to lack of permission to create the top level directory in /contrib.  I submitted a helpdesk ticket for the directory to be created.  Once it is created I will try to install on hera.

In testing on Orion, I received the following message:

Orion-devel-1[89] jprestop$ python /apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/performance_diagram.py /apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/minimal_performance_diagram.yaml Traceback (most recent call last):   File "/apps/intel-2020.2/intel-2020.2/intelpython3/lib/python3.7/site-packages/matplotlib/backends/backend_webagg.py", line 26, in     import tornado ModuleNotFoundError: No module named 'tornado' During handling of the above exception, another exception occurred: Traceback (most recent call last):   File "/apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/performance_diagram.py", line 8, in     import matplotlib.pyplot as plt   File "/apps/intel-2020.2/intel-2020.2/intelpython3/lib/python3.7/site-packages/matplotlib/pyplot.py", line 2349, in     switch_backend(rcParams["backend"])   File "/apps/intel-2020.2/intel-2020.2/intelpython3/lib/python3.7/site-packages/matplotlib/pyplot.py", line 221, in switch_backend     backend_mod = importlib.import_module(backend_name)   File "/apps/intel-2020.2/intel-2020.2/intelpython3/lib/python3.7/importlib/init.py", line 127, in import_module     return _bootstrap._gcd_import(name[level:], package, level)   File "/apps/intel-2020.2/intel-2020.2/intelpython3/lib/python3.7/site-packages/matplotlib/backends/backend_webagg.py", line 28, in     raise RuntimeError("The WebAgg backend requires Tornado.") RuntimeError: The WebAgg backend requires Tornado.

@bikegeek @fisherhucar Is "tornado" a Python package that I should request be installed on Orion?

LoganDawson-NOAA commented 3 years ago

@jprestop it appears we're on the same page with testing METplotpy since the latest beta release on Orion.

There's a potential alternative to requesting to have the "tornado" package installed with the intelpython/2020.2 version of python you're using. There's a python/3.7.5 installation available on Orion as well that doesn't produce the Tornado error from the matplotlib backend script. Rahul suggested that I try this other python installation because Orion sys admins may not be fully supporting that Intel python installation.

Because you have the METplus codes set up to run with intelpython/2020.2, I force my environment to switch python versions by loading python last: module use /apps/contrib/modulefiles module load metcalcpy/1.1.0-beta2 module load metplotpy/1.1.0-beta2 module load python/3.7.5

When I run the same command as you have in your comment above, I get the same error you mentioned in the METcalcpy issue:

python /apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/performance_diagram.py /apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/minimal_performance_diagram.yaml Traceback (most recent call last): File "/apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/performance_diagram.py", line 16, in import metcalcpy.util.utils as calc_util File "/apps/contrib/MET/METcalcpy/METcalcpy-1.1.0-beta2/metcalcpy/util/utils.py", line 19, in from statsmodels.tsa.arima.model import ARIMA ModuleNotFoundError: No module named 'statsmodels'

However, when I run the default test of the line plot, I get a error on plotly not being found:

python /apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/line/line.py /apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/line/custom_line.yaml Traceback (most recent call last): File "/apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/line/line.py", line 17, in import plotly.graph_objects as go ModuleNotFoundError: No module named 'plotly'

Again, these errors are returned when I test METplotpy with python/3.7.5 (instead of intelpython/2020.2) which gets us past the tornado error from matplotlib.

jprestop commented 3 years ago

Hi @LoganDawson-NOAA. Thanks for the update on things.

There's a potential alternative to requesting to have the "tornado" package installed with the intelpython/2020.2 version of python you're using. There's a python/3.7.5 installation available on Orion as well that doesn't produce the Tornado error from the matplotlib backend script. Rahul suggested that I try this other python installation because Orion sys admins may not be fully supporting that Intel python installation.

It turns out we can actually use the intelpython/2020.2 version and can add packages to our own user area by running:

python3 -m pip install --user <package name>

For example:

python3 -m pip install --user statsmodels

Using this, I made some forward progress with METplotpy (and METcalcpy) but am still stuck. Here's an update:

I ran the following:

module load contrib
module load intel/2020.2
module load intelpython3/2020.2
module use /apps/contrib/modulefiles
module load metplotpy/1.1.0-beta2
python -m pip install --user tornado
python -m pip install --user statsmodels
python -m pip install --user plotly

Then, I ran this to test:

Orion-dtn-2[43] jprestop$ python /apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/performance_diagram.py /apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/minimal_performance_diagram.yaml

But, I got the following error:

Error
Traceback (most recent call last):
  File "/apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/performance_diagram.py", line 447, in <module>
    main()
  File "/apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/performance_diagram.py", line 440, in main
    PerformanceDiagram(docs)
  File "/apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/performance_diagram.py", line 53, in __init__
    super().__init__(parameters, default_conf_filename)
  File "/apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/base_plot.py", line 44, in __init__
    with open(os.path.join(location, default_conf_filename), 'r') as stream:
FileNotFoundError: [Errno 2] No such file or directory: '/home/config/performance_diagram_defaults.yaml'

It can’t seem to find performance_diagram_defaults.yaml. I found it here:

/apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/config/performance_diagram_defaults.yaml

I added the path to that file to the PATH and PYTHONPATH variables:

export PATH=/apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/config:${PATH}
export PYTHONPATH=/apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/config:${PYTHONPATH}

But that did not work, and I got the same error. It seems it is looking in /home/config/, so I am wondering if there is an environment variable for the first part of the path (/apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/) that is expected to be set. @bikegeek or @fisherhucar Is there an environment variable I need to set for the plotting config files?

@LoganDawson-NOAA Minna, Hank, and I are planning to have a meeting to work through these issues, hopefully, sometime next week, however, since I found out the ability to run pip install, I thought I'd try to make some forward progress on my own.

LoganDawson-NOAA commented 3 years ago

Hey @jprestop, thanks as well for your follow-up. It's good to see us moving past additional errors. Progress is progress.

Just following along here, it looks like METPLOTPY_BASE is the environment variable that is likely causing your latest error. I didn't define that in my environment at all when doing my testing. So I imagine it's not being set currently, which would then cause METplotpy to look for the config file under your current working directory.

Sounds good on you, Minna, and Hank working on this over the next couple weeks. Just let me know if you need any other testing from me.

jprestop commented 3 years ago

Hi @LoganDawson-NOAA. Thank you so much for the link showing me that METPLOTPY_BASE is what I wanted to set. You were exactly correct and that did help. I also realized that in addition to setting that I needed to run "module load metcalcpy/1.1.0-beta2" because METplotpy was expecting METcalcpy to be loaded.

@bikegeek @fisherhucar When I ran:

python /apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/performance_diagram.py /apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/minimal_performance_diagram.yaml

I received an error:

FileNotFoundError: [Errno 2] File b'../performance_diagram/plot_20200317_151252.data' does not exist: b'../performance_diagram/plot_20200317_151252.data'

I noticed that when I ran the command line from this directory "/apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/config" the command worked great!

So, I went into the file "/apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/config/performance_diagram_defaults.yaml" and changed

stat_input: ../performance_diagram/plot_20200317_151252.data

to be:

stat_input: /apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/plot_20200317_151252.data

which allowed me to run the command from another directory instead of only from "/apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/config".

Users will not have permission to write to directories in the installation location (to either change config values or to run from the installed area and have output placed in the current directory as "plot_filename: ./performance_diagram_default.png" indicates). How did you envision this would work and how should we handle this? Should I modify this and other yaml files (if so, which ones) with the full path when I install on machines? Should users copy these yaml files to their own directories to modify?

We're getting close! Thanks again for the help @LoganDawson-NOAA!

LoganDawson-NOAA commented 3 years ago

Hi @jprestop,

Now that you made that change to the "/apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/config/performance_diagram_defaults.yaml" file, I tried to run the same command to see if I could get a plot from my working directory (/work/noaa/ovp/ldawson). I get the following error: 

python /apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/performance_diagram.py /apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/minimal_performance_diagram.yaml
Traceback (most recent call last):
  File "/apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/performance_diagram.py", line 447, in <module>
    main()
  File "/apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/performance_diagram.py", line 440, in main
    PerformanceDiagram(docs)
  File "/apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/performance_diagram.py", line 77, in __init__
    self.figure = self._create_figure()
  File "/apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/performance_diagram.py", line 266, in _create_figure
    self.write_output_file()
  File "/apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/performance_diagram.py", line 388, in write_output_file
    os.remove(output_file)
OSError: [Errno 30] Read-only file system: '/apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/plot_20200317_151252.points1'

I previously saw the error where METplotpy couldn't find the data file, so I know your change to the default config file isn't the problem.

I also got the same error when I copied the custom_performance_diagram.yaml file into my working directory and changed the stat_input path to the full path. In the test with custom_performance_diagram.yaml, I left the plot_filename unchanged so the image would be saved in my own working directory:

stat_input:  /apps/contrib/MET/METplotpy/METplotpy-1.1.0-beta2/metplotpy/plots/performance_diagram/plot_20200317_151252.data
plot_filename: ./performance_diagram_custom.png

I haven't dug too far to confirm this, but in my tests, it looks like the METplotpy code is attempting to write at least one intermediary file (ending in ".points1") in the same location where the test data exists. And of course I don't (and shouldn't) have write access to that directory.

Now getting to your questions here...

How did you envision this would work and how should we handle this? Should I modify this and other yaml files (if so, which ones) with the full path when I install on machines? Should users copy these yaml files to their own directories to modify?

@bikegeek @fisherhucar, if I'm misinterpreting the error I've listed above, just let me know. I can also follow up again later in the week after we have our weekly EMC Verification System planning meeting to make sure any change to the output directory for the intermediary files is a good long-term solution.

jprestop commented 3 years ago

@LoganDawson-NOAA Thank you for this valuable information and feedback. I'll have to defer to @bikegeek and @fisherhucar, but I agree that it is essential that the METplotpy code needs to be changed to write any intermediary files to 1) the users' current working directory or 2) a user-specified directory. I'll create an issue for it now. @bikegeek and @fisherhucar, please let us know if we have a misunderstanding of the way things are working.

jprestop commented 3 years ago

@LoganDawson-NOAA I created the following ticket for this issue: Write any intermediary files to a user-specified directory #153. I think it would be best to have the user specific a directory instead of writing to the current directory. It is more flexible and transparent. Please feel free to comment on the issue.

LoganDawson-NOAA commented 3 years ago

@jprestop thanks for creating the new issue. I agree that writing to a specified directory is the best way to move forward. I subscribed to the new issue and will be happy to test when @bikegeek and @fisherhucar have an update ready

jprestop commented 2 years ago

@fisherhucar @TatianaBurek @bikegeek @TaraJensen @LoganDawson-NOAA I have installed METplotpy and METcalcpy on Hera. To use, I load the following:

module load intel/18.0.5.274
module load anaconda/latest
module use /contrib/METplotpy/modulefiles
module load metplotpy/1.1.0-beta3
module use /contrib/METcalcpy/modulefiles
module load metcalcpy/1.1.0-beta3

When I try to run:

python /contrib/METplotpy/METplotpy-1.1.0-beta3/metplotpy/plots/performance_diagram/performance_diagram.py /contrib/METplotpy/METplotpy-1.1.0-beta3/metplotpy/plots/performance_diagram/minimal_performance_diagram.yaml

I get the following:

Traceback (most recent call last):
  File "/contrib/METplotpy/METplotpy-1.1.0-beta3/metplotpy/plots/performance_diagram/performance_diagram.py", line 16, in <module>
    import metcalcpy.util.utils as calc_util
  File "/contrib/METcalcpy/METcalcpy-1.1.0-beta3/metcalcpy/util/utils.py", line 19, in <module>
    from statsmodels.tsa.arima.model import ARIMA
ModuleNotFoundError: No module named 'statsmodels.tsa.arima'

I know that on Orion we could run, for example "python -m pip install --user statsmodels" to make "statsmodels" available to us, however we are using a module load of "intelpython3/2020.2" on Orion as opposed to the anaconda version we are using on Hera, where we do a module load of "anaconda/latest". I went ahead and tried to run the pip install even though I suspected it wouldn't work. Here is the result:

[METplotpy]$ python -m pip install --user statsmodels
Requirement already satisfied: statsmodels in /contrib/anaconda/anaconda3/latest/lib/python3.7/site-packages (0.9.0)

So, then I tried the following, more specifically (but it gave an error):

[METplotpy]$ python -m pip install --user statsmodels.tsa.arima.model
Collecting statsmodels.tsa.arima.model
  ERROR: Could not find a version that satisfies the requirement statsmodels.tsa.arima.model (from versions: none)
ERROR: No matching distribution found for statsmodels.tsa.arima.model

Can anyone tell me if this is something I should write to the Hera helpdesk to try to get installed? Or, is there really some other issue that I am not understanding? Thanks for any guidance you can give. Also, for reference, on the command line for python, as expected:

[METplotpy]$ python
Python 3.7.3 (default, Mar 27 2019, 22:11:17) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import metcalcpy.util.utils as calc_util
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/contrib/METcalcpy/METcalcpy-1.1.0-beta3/metcalcpy/util/utils.py", line 19, in <module>
    from statsmodels.tsa.arima.model import ARIMA
ModuleNotFoundError: No module named 'statsmodels.tsa.arima'
>>> import statsmodels
>>> import statsmodels.tsa
>>> import statsmodels.tsa.arima
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'statsmodels.tsa.arima'
jprestop commented 2 years ago

@fisherhucar @TatianaBurek @bikegeek @TaraJensen @LoganDawson-NOAA

And, since we can't seem to test out the same functionality on Hera as we are experiencing on Orion, where we are further along, I will schedule a working session where perhaps Logan could share his screen to demonstrate the problem. Logan, to bring you up to speed, we had an internal meeting today and discussed having a working session to demonstrate the problem so that Tatiana (and Hank and Minna) could see where the problem with the intermediary files lies.

TatianaBurek commented 2 years ago

@fisherhucar @bikegeek @jprestop From Julie's comment: Requirement already satisfied: statsmodels in /contrib/anaconda/anaconda3/latest/lib/python3.7/site-packages (0.9.0) Does it mean that the version of statsmodels that is installed is 0.9.0? The min required version that we support is statsmodels 0.11.1: https://github.com/dtcenter/METcalcpy/blob/main_v1.0/docs/Users_Guide/installation.rst I don't know if v0.9.0 contains arima functionality or not

bikegeek commented 2 years ago

It looks like ARIMA is supported in the v0.9.0 of statsmodel: https://www.statsmodels.org/stable/api.html

ARIMA https://www.statsmodels.org/stable/generated/statsmodels.tsa.arima.model.ARIMA.html#statsmodels.tsa.arima.model.ARIMA (endog[, exog, order, seasonal_order, …])

Minna Win Pronouns: she/her National Center for Atmospheric Research Developmental Testbed Center Phone: 303-497-8423 Fax: 303-497-8401

On Thu, Nov 4, 2021 at 9:10 AM Tatiana Burek @.***> wrote:

@fisherhucar https://github.com/fisherhucar @bikegeek https://github.com/bikegeek @jprestop https://github.com/jprestop From Julie's comment: Requirement already satisfied: statsmodels in /contrib/anaconda/anaconda3/latest/lib/python3.7/site-packages (0.9.0) Does it mean that the version of statsmodels that is installed is 0.9.0? The min required version that we support is statsmodels 0.11.1:

https://github.com/dtcenter/METcalcpy/blob/main_v1.0/docs/Users_Guide/installation.rst I don't know if v0.9.0 contains arima functionality or not

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dtcenter/METplotpy/issues/143#issuecomment-961138326, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4UJHSNLZ6YFXB6JQP7WMDUKKV73ANCNFSM5CO5JX4A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

TatianaBurek commented 2 years ago

The link https://www.statsmodels.org/stable/api.html points to latest stable version v0.13.0 and not to v0.9.0

bikegeek commented 2 years ago

oops, here's a correct link: https://www.statsmodels.org/0.9.0/generated/statsmodels.tsa.arima_model.ARIMA.html#statsmodels.tsa.arima_model.ARIMA

Minna Win Pronouns: she/her National Center for Atmospheric Research Developmental Testbed Center Phone: 303-497-8423 Fax: 303-497-8401

On Thu, Nov 4, 2021 at 9:43 AM Tatiana Burek @.***> wrote:

The link https://www.statsmodels.org/stable/api.html points to latest stable version v0.13.0 and not to v0.9.0

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dtcenter/METplotpy/issues/143#issuecomment-961169339, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4UJHSYWPMDCSGITDU4P2LUKKZ4ZANCNFSM5CO5JX4A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

TatianaBurek commented 2 years ago

Here is the difference: v0.9.0: statsmodels.tsa.arima_model.ARIMA v0.11.1: statsmodels.tsa.arima.model.ARIMA Is it possible to update statsmodels's version to at least v0.11.1?

jprestop commented 2 years ago

Thank you for finding this information. How would I attempt to update the version? If there some variation of this command that I could try with some additional argument to update the version?

python -m pip install --user statsmodels

bikegeek commented 2 years ago

does python -m pip install --user statsmodels=0.11 work?

Minna Win Pronouns: she/her National Center for Atmospheric Research Developmental Testbed Center Phone: 303-497-8423 Fax: 303-497-8401

On Thu, Nov 4, 2021 at 10:38 AM jprestop @.***> wrote:

Thank you for finding this information. How would I attempt to update the version? If there some variation of this command that I could try with some additional argument to update the version?

python -m pip install --user statsmodels

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dtcenter/METplotpy/issues/143#issuecomment-961218793, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4UJHW3HMUPOJYVIUTJZE3UKLAJTANCNFSM5CO5JX4A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

jprestop commented 2 years ago

@bikegeek I tried that (see below). Unfortunately, it looks like I'll need to request this from the Hera helpdesk. However, before I do that, could you please provide a list of Python packages and their versions for METplotpy and METcalcpy so that I can attempt to take care of everything at once?

[METplotpy]$ python -m pip install --user statsmodels=0.11 ERROR: Invalid requirement: 'statsmodels=0.11' = is not a valid operator. Did you mean == ?

[METplotpy]$ python -m pip install --user statsmodels==0.11 Collecting statsmodels==0.11 WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f494040ce80>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /packages/4d/a6/685d3ed1e1de53d2c19f8dd76ac03cbd144302a9ab59a743308d69aba3c9/statsmodels-0.11.0-cp37-cp37m-manylinux1_x86_64.whl ... ERROR: Could not install packages due to an EnvironmentError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/4d/a6/685d3ed1e1de53d2c19f8dd76ac03cbd144302a9ab59a743308d69aba3c9/statsmodels-0.11.0-cp37-cp37m-manylinux1_x86_64.whl (Caused by NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f49403a9ef0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

TatianaBurek commented 2 years ago

This is not going to help but ... we need statsmodels==0.11.1 I usually use this file for the version's check: https://github.com/dtcenter/METcalcpy/blob/main_v1.0/docs/Users_Guide/installation.rst @bikegeek is any other files except this one?

bikegeek commented 2 years ago

Here you go:

For METplotpy: https://metplotpy.readthedocs.io/en/develop/Users_Guide/installation.html

and for METcalcpy: https://metcalcpy.readthedocs.io/en/develop/Users_Guide/installation.html

Minna Win Pronouns: she/her National Center for Atmospheric Research Developmental Testbed Center Phone: 303-497-8423 Fax: 303-497-8401

On Thu, Nov 4, 2021 at 11:05 AM jprestop @.***> wrote:

I tried that (see below). Unfortunately, it looks like I'll need to request this from the Hera helpdesk. However, before I do that, could you please provide a list of Python packages and their versions for METplotpy and METcalcpy so that I can attempt to take care of everything at once?

[METplotpy]$ python -m pip install --user statsmodels=0.11 ERROR: Invalid requirement: 'statsmodels=0.11' = is not a valid operator. Did you mean == ? @bikegeek https://github.com/bikegeek [METplotpy]$ python -m pip install --user statsmodels==0.11 Collecting statsmodels==0.11 WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f494040ce80>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /packages/4d/a6/685d3ed1e1de53d2c19f8dd76ac03cbd144302a9ab59a743308d69aba3c9/statsmodels-0.11.0-cp37-cp37m-manylinux1_x86_64.whl ... ERROR: Could not install packages due to an EnvironmentError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/4d/a6/685d3ed1e1de53d2c19f8dd76ac03cbd144302a9ab59a743308d69aba3c9/statsmodels-0.11.0-cp37-cp37m-manylinux1_x86_64.whl (Caused by NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f49403a9ef0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dtcenter/METplotpy/issues/143#issuecomment-961241236, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4UJHQHVTNDXESSSPLHZB3UKLDN7ANCNFSM5CO5JX4A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

jprestop commented 2 years ago

@bikegeek Fantastic! Thank you so much for these lists. I'll follow up with Hera helpdesk and will update here once I've heard back and hopefully have been able to make further progress. I appreciate your help very much!

bikegeek commented 2 years ago

@Tatiana Burek @.***> , yes that is the correct file. It is what the ReadTheDocs documentation is using. I also use that when I'm building new conda environments (like right now, because I updated my OS).

Minna Win Pronouns: she/her National Center for Atmospheric Research Developmental Testbed Center Phone: 303-497-8423 Fax: 303-497-8401

On Thu, Nov 4, 2021 at 11:10 AM Tatiana Burek @.***> wrote:

This is not going to help but ... we need statsmodels==0.11.1 I usually use this file for the version's check:

https://github.com/dtcenter/METcalcpy/blob/main_v1.0/docs/Users_Guide/installation.rst @bikegeek https://github.com/bikegeek is any other files except this one?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dtcenter/METplotpy/issues/143#issuecomment-961245831, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4UJHRYVDCAW2F3HAFG4CLUKLEBTANCNFSM5CO5JX4A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

bikegeek commented 2 years ago

No problem @Julie Prestopnik @.***>, let me know if you have any other questions.

Minna Win Pronouns: she/her National Center for Atmospheric Research Developmental Testbed Center Phone: 303-497-8423 Fax: 303-497-8401

On Thu, Nov 4, 2021 at 11:15 AM jprestop @.***> wrote:

@bikegeek https://github.com/bikegeek Fantastic! Thank you so much for these lists. I'll follow up with Hera helpdesk and will update here once I've heard back and hopefully have been able to make further progress. I appreciate your help very much!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dtcenter/METplotpy/issues/143#issuecomment-961249669, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4UJHXXMAJLRPXLXFKD7BDUKLESPANCNFSM5CO5JX4A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

jprestop commented 2 years ago

@bikegeek Actually it looks like the file @TatianaBurek linked to is old (still has bootstrapped and pingouin listed. I think if she replaced the "main_v1.0" in the link with "develop" that would bring her to the updated list.

jprestop commented 2 years ago

@bikegeek Can you please tell me what version of lxml is needed? Hera currently has version 4.2.5. Is that sufficient?

Also, under the "Currently doesn’t have" section below for the packages that do not have package versions listed, could you please tell me what versions you are using?

I am very glad we're doing this all at once since we have a lot of requests. Thanks again for your help.

Currently doesn’t have: cmocean eofs imutils 0.5.3 imagio python-kaleido 0.2.1 pymysql

Need updated packages: cartopy 0.18.0 (currently has 1.17.0) matplotlib 3.3.0 (currently has 3.1.0) netcdf4 1.5.1.2 (currently has 1.4.2) numpy 1.17.0 (currently has 1.16.4) pandas 1.0.1 (currently has 0.23.4) psutil 5.7.2 (currently has 5.4.7) pytest 5.2.1 (currently has 2.8.0) pyyaml 5.3.1 (currently has 3.13) scikit-image 0.16.2 (currently has 0.14.0) scikit-learn 0.23.2 (currently has 0.19.2) scipy 1.5.1 (currently has 1.3.0) statsmodels 0.11.1 (currently has 0.9.0) xarray 0.16.2 (currently has 0.11.3)

We’re good on: plotly 4.9.0 (currently has 4.9.0)

bikegeek commented 2 years ago

@jprestop, here are the versions that you'll probably need to request:

cmocean 2.0 eofs 1.3.1 or above (1.4.0 installed on 'seneca') imutils 0.5.3 imageio 2.9.0 (this is the latest version available via anaconda) python-kaleido 0.2.1 pymysql 0.9.3 (currently in use by METdatadb, but Venita thinks perhaps its best to use the latest version available, 1.0.2) lxml 4.5.2 (currently in use by Venita)

*I think the version for cartopy (in comment above) that is currently installed should be 0.17.0 (1.17.0), as this isn't a 1.x released version yet.

jprestop commented 2 years ago

@bikegeek

This is great info! Thank you so much!

jprestop commented 2 years ago

@bikegeek I just now saw the additional comment:

I think the version for cartopy (in comment above) that is currently installed should be 0.17.0 (1.17.0), as this isn't a 1.x released version yet.

Since https://metplotpy.readthedocs.io/en/develop/Users_Guide/installation.html lists 1.18.0 would it be ok to get that version still, or is there a reason not to? I just didn't want to have to ask them again soon for this later version if we need it for the 1.1.0 release.

bikegeek commented 2 years ago

@jprestop, you will definitely need 0.18 version of cartopy, this fixes a major issue with crossing the International Dateline (visible in the METplus wrapper CyclonePlotter for instance), so we should request this version for METplotpy.

TatianaBurek commented 2 years ago

I was thinking the same about statsmodels. If it is not too late, can we install the version 0.12.2 instead of 0.11.1

jprestop commented 2 years ago

@TatianaBurek @bikegeek Thanks! I requested version 0.12.2 of statsmodels and version 0.18 of cartopy.

bikegeek commented 2 years ago

Thanks for following up @jprestop. I hope those versions will be easy to acquire and install. Those versions are crucial for the correct behavior of the METplus analysis tools.

hankenstein2 commented 2 years ago

Done for several NOAA machines, waiting on WCOSS admins for next round. I think it is useful to keep these particular issues going so we can review what we have asked for.

hankenstein2 commented 1 year ago

I think this issue is not longer useful/relevant. Since we opened it we have morphed our tracking of tasks elsewhere and discussed and tracked in meetings. If we need to re-open for the next round of releases we can just start a new issue.