search

NCAR / SMYLE-analysis

MIT License
3 stars 0 forks source link

environment.yml has conflicts #1

Closed sgyeager closed 3 years ago

sgyeager commented 3 years ago

conda env update -f envs/environment.yml has been running for about 10 minutes now and still hasn't solved it. Looks like we introduced some incompatibilities by modifying https://github.com/marbl-ecosys/HiRes-CESM-analysis/blob/master/environments/environment.yaml

andersy005 commented 3 years ago

I will take a look at this after my 3pm meeting. Could you post here the output message you are currently getting?

sgyeager commented 3 years ago

env_errors.txt

andersy005 commented 3 years ago

@sgyeager, these are the culprit lines/packages (I believe :)) 

- bokeh=1.4
- dask==2.14
- distributed==2.14

Let's unpin these versions (i.e. remove the version requirements) and see if conda is able to create the environment

- bokeh
- dask
- distributed

Let me know how it goes...

sgyeager commented 3 years ago

Got another "UnsatisfiableError: The following specifications were found to be incompatible with each other:" followed by a very lengthy list.

andersy005 commented 3 years ago

It turns out cf_units should be cf-units instead. With the following changes conda is able to create an environment. Here's the git diff:

diff --git a/glade/u/home/yeager/analysis/python/CESM2-SMYLE/SMYLE-analysis/envs/environment.yml b/environment.yml
index 8631743..a348d98 100644
--- a/glade/u/home/yeager/analysis/python/CESM2-SMYLE/SMYLE-analysis/envs/environment.yml
+++ b/environment.yml
@@ -3,16 +3,16 @@ channels:
     - conda-forge
     - nodefaults
 dependencies:
-    - bokeh
+    - bokeh==1.4
     - bottleneck
     - cartopy
     - climpred
-    - cf_units
+    - cf-units
     - cftime
     - cmocean
-    - dask
+    - dask==2.14
     - dask-jobqueue
-    - distributed
+    - distributed==2.14
     - dask-labextension
     - eofs
     - esmpy
sgyeager commented 3 years ago

OK. Got the smyle-analysis environment successfully built, but when I use it in JupyterHub, I get the following error after client when I click on the dashboard URL:

tornado.application - ERROR - Uncaught exception GET /status/ws (::1) HTTPServerRequest(protocol='http', host='jupyterhub.ucar.edu', method='GET', uri='/status/ws', version='HTTP/1.1', remote_ip='::1') Traceback (most recent call last): File "/glade/work/yeager/miniconda3/envs/smyle-analysis/lib/python3.8/site-packages/tornado/websocket.py", line 954, in _accept_connection open_result = handler.open(*handler.open_args, *handler.open_kwargs) File "/glade/work/yeager/miniconda3/envs/smyle-analysis/lib/python3.8/site-packages/tornado/web.py", line 3173, in wrapper return method(self, args, **kwargs) File "/glade/work/yeager/miniconda3/envs/smyle-analysis/lib/python3.8/site-packages/bokeh/server/views/ws.py", line 128, in open raise ProtocolError("Subprotocol header is not 'bokeh'") bokeh.protocol.exceptions.ProtocolError: Subprotocol header is not 'bokeh'

andersy005 commented 3 years ago

The issue you are running into has to do with some version mismatch between your environment and the default environment used by the JupyterHub. You need to pin the following package versions in environment.yml:

- bokeh=1.4
- dask==2.14
- distributed==2.14
sgyeager commented 3 years ago

Final solution: Had to add lines death-timeout: 60 to my ~/.config/dask/ncar-jobqueue.yaml file. Also modified notebook so that cluster call has that spec: cluster = NCARCluster(cores=ncores, death_timout=60, processes=ncores, memory=nmem, project=proj, walltime='3:00:00')

Now, things are working.