search

pydata / parallel-tutorial

Parallel computing in Python tutorial materials
300 stars 111 forks source link

Spark package #11

Open mrocklin opened 7 years ago

mrocklin commented 7 years ago

Our two choices for pyspark on anaconda.org are either the conda-forge or quasiben channel.

it looks like conda-forge is 2.7 only?

(parallel) mrocklin@carbon:~$ conda install -c conda-forge pyspark
Fetching package metadata ...........
Solving package specifications: .

UnsatisfiableError: The following specifications were found to be in conflict:
  - pyspark -> python 2.7*
  - python 3.6*
Use "conda info <package>" to see the dependencies for each package.

While the quasiben package lacks support for python 3.6

(parallel) mrocklin@carbon:~$ conda install -c quasiben spark
Fetching package metadata ...........
Solving package specifications: .

UnsatisfiableError: The following specifications were found to be in conflict:
  - python 3.6*
  - spark -> py4j ==0.10.1 -> python 3.5* -> openssl 1.0.1*
  - spark -> py4j ==0.10.1 -> python 3.5* -> xz 5.0.5
Use "conda info <package>" to see the dependencies for each package.

Do we use quasiben and force 3.5? Do we ask @quasiben to update his package to 3.6? Do we ask conda-forge people to update the pyspark package?

mrocklin commented 7 years ago

Sorry, it looks like conda-forge does support python 3.5