nebari-dev / nebari

🪴 Nebari - your open source data science platform
https://nebari.dev
BSD 3-Clause "New" or "Revised" License
281 stars 93 forks source link

Benchmark installation of large environment.yaml files on nfs #33

Closed costrouc closed 4 years ago

costrouc commented 4 years ago

Take this environment for example. @dharhas mentioned that with the previous nfs share this took over 45 minutes

name: default
channels:
  - bokeh  # required for jupyterlab bokeh extension
  - pyviz  # required for spatialpandas
  - conda-forge
dependencies:
  - python=3.7
  - ipython > 7
  - pip
  # jupyterhub/jupyterlab
  - nb_conda_kernels  # jupyterhub
  - jupyter-server-proxy  # jupyterhub
  - ipywidgets  # jupyterhub
  - jupyterlab < 2.0  # jupyterhub
  - jupyter_client < 6.0  # jupyterhub
  - jupyter_console  # jupyterhub
  - jupyterhub  # jupyterhub
  - nodejs  # jupyterhub
  - nbconvert
  - jupyter-archive  # jupyterhub
  # dask
  - dask  # repo
  - distributed  # repo
  # - dask-gateway
  - dask-kubernetes  # repo
  - kubernetes  # repo
  - fastparquet  # repo
  - python-blosc
  - cytoolz
  - python-snappy
  - msgpack-python
  - bokeh < 2
  # numerical
  - numba
  - numpy  # repo
  - pandas  # epo
  - scikit-learn  # repo
  - statsmodels
  # aws tools
  - s3fs  # repo
  # - awscli
  - boto3  #  repo
  # geo tools
  - geopandas >= 0.6.0  # repo
  # - fiona
  # - gdal
  # - rasterio
  - datashader
  - haversine
  - googlemaps  # repo
  - python-geohash # required by polygon-geohasher
  - spatialpandas >= 0.3.5
  - geopy # optional geopandas dependency for geocoding
  - cartopy  # repo
  # - gmaps  # jupyterhub
  # viz tools
  - panel  # repo
  - holoviews < 1.13  # repo
  - hvplot
  - geoviews < 1.7  # repo
  - param
  - python-graphviz # required for Dask visualization
  - matplotlib
  - seaborn
  # - plotly
  # - selenium  # required for saving bokeh images
  # - phantomjs # required for saving bokeh images
  - descartes # required for plotting polygons in geopandas
  # - mapclassify # optional geopandas dependency
  - jupyter_bokeh  # jupyterhub
  # testing, docs, linting
  - pytest  # repo
  - hypothesis  # repo
  - importnb
  - nbval
  - flake8
  - sphinx  # repo
  - pytest-cov
  - black  # repo
  - isort  # repo
  # VS Code linters
  - pydocstyle
  - pycodestyle
  - bandit
  - mypy
  - yapf
  # data ingestion
  - xlrd
  # other pandas dependencies
  - bottleneck
  - cython
  - numexpr
  - openpyxl
  - PyTables
  - XLsxWriter
  - xlwt
  - BeautifulSoup4  # repo
  - html5lib
  - lxml
  # Google Cloud
  - google-cloud-storage
  - gcsfs
  # additional packages
  - requests
  - pre-commit 
  - python-geohash  #repo
  - setuptools # required by polygon-geohasher
  - shapely   #  repo, required by polygon-geohasher
  - prefect # for automating workflows
  - tqdm  # for progress bars
  - watermark  # IPython magic to print important information
  - cssselect
  - h5py
  - python-dotenv
  - pyct  #  repo
  - toolz  # repo
  - loguru
  - boto
  - kartothek
  - vcrpy
  - pip:
    - smartystreets_python_sdk  # repo
    - polygon-geohasher  # repo
    - python-docx  # repo
    - dask_labextension  # jupyterhub
    - aws-logging-handlers  # repo, save logs to s3
    - prospector[with_pyroma]  # VS Code linters
    - pylama  # VS Code linters
    - pytest-vcr  # repo
tylerpotts commented 4 years ago

Using conda store which doesn't use NFS for conda installs. Not needed