pip install spark-testing-base throws a warning (failed building wheel for spark-testing-base)

[luck02]

I haven't gotten deep enough to see what actual impact this has.

(venv) ➜  pySparkTesting pip install -r requirements.txt
Collecting pyspark==2.2.0.post0 (from -r requirements.txt (line 1))
Requirement already satisfied: py4j==0.10.4 in ./venv/lib/python3.5/site-packages (from pyspark==2.2.0.post0->-r requirements.txt (line 1))
Installing collected packages: pyspark
Successfully installed pyspark-2.2.0
(venv) ➜  pySparkTesting pip install spark-testing-base
Collecting spark-testing-base
  Downloading spark-testing-base-0.7.1.tar.gz
Collecting unittest2 (from spark-testing-base)
  Downloading unittest2-1.1.0-py2.py3-none-any.whl (96kB)
    100% |████████████████████████████████| 102kB 1.8MB/s 
Collecting findspark (from spark-testing-base)
  Downloading findspark-1.1.0-py2.py3-none-any.whl
Collecting pytest (from spark-testing-base)
  Downloading pytest-3.2.3-py2.py3-none-any.whl (187kB)
    100% |████████████████████████████████| 194kB 3.7MB/s 
Collecting traceback2 (from unittest2->spark-testing-base)
  Downloading traceback2-1.4.0-py2.py3-none-any.whl
Collecting argparse (from unittest2->spark-testing-base)
  Downloading argparse-1.4.0-py2.py3-none-any.whl
Collecting six>=1.4 (from unittest2->spark-testing-base)
  Downloading six-1.11.0-py2.py3-none-any.whl
Requirement already satisfied: setuptools in ./venv/lib/python3.5/site-packages (from pytest->spark-testing-base)
Collecting py>=1.4.33 (from pytest->spark-testing-base)
  Downloading py-1.4.34-py2.py3-none-any.whl (84kB)
    100% |████████████████████████████████| 92kB 3.2MB/s 
Collecting linecache2 (from traceback2->unittest2->spark-testing-base)
  Downloading linecache2-1.0.0-py2.py3-none-any.whl
Building wheels for collected packages: spark-testing-base
  Running setup.py bdist_wheel for spark-testing-base ... error
  Complete output from command /home/gary/dev/pySparkTesting/venv/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-6xvw0qyk/spark-testing-base/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmp85uvtkbnpip-wheel- --python-tag cp35:
  /usr/lib/python3.5/distutils/dist.py:261: UserWarning: Unknown distribution option: 'test_requires'
    warnings.warn(msg)
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for spark-testing-base
  Running setup.py clean for spark-testing-base
Failed to build spark-testing-base
Installing collected packages: linecache2, traceback2, argparse, six, unittest2, findspark, py, pytest, spark-testing-base
  Running setup.py install for spark-testing-base ... done
Successfully installed argparse-1.4.0 findspark-1.1.0 linecache2-1.0.0 py-1.4.34 pytest-3.2.3 six-1.11.0 spark-testing-base-0.7.1 traceback2-1.4.0 unittest2-1.1.0

This happened on first attempting to install via command above. I uninstalled, pip install wheel and then re-install and no warning.

[holdenk]

So were you able to install other packages at the time? We could add wheel to the required list I suppose but I'm not sure that it makes sense.

[luck02]

I'm sorry @holdenk I'm a bit of a python newbie. I've really only written python code for spark so my knowledge around python environments is nothing special. I'm on a fresh-ish install of ubuntu 17.04 and using python 3.5.3 in a locally installed virtual environment. I mostly posted in case others had experienced. I'm functional now,

I was able to install mulitple other packages. I did run pip install wheel immediately after and then my install was fine.

I'm just going to close it so others have something to search for if they run into it. Maybe just a python newb problem.

Thanks!

[luck02]

I'm just poking at a sample pyspark 'how to test your code' library: https://github.com/luck02/PysparkTestingExample

it's not done, and probably won't be for awhile. The hope is to provide a starting point for how to build datapipelines with spark, to start with at my company, but I'm keeping this aspect in my personal repo in the hopes that it helps anyone else with getting going.

[holdenk]

That's awesome! If you're interested in contributing this to spark-testing-base as an example I'd be happy to make you a collaborator on the project. Otherwise we can just add a link in the README file :)

[luck02]

I'll totally put it here as examples. give me another couple of weeks to get feedback from better pythonista's than me. We have at least a couple really competent python devs at my shop (Unbounce)

[luck02]

Actually, I don't need to wait. Take a look and if it seems valuable (https://github.com/luck02/PysparkTestingExample) I'll start moving it over soon-ish. In an ideal world I'd like to drive out a robust set of patterns and practices for test-driven pyspark code and then move onto the same set of patterns and practices in Scala. Ultimately I'd also like to start showing patterns around how to validate existing data as I don't think we have a sane option around full integration tests at this stage. IE at least not a clear pattern, but if one does emerge I'd like to cover that as well.

[luck02]

@holdenk did you want me to move those example tests over? I realize they're kinda bare right now I was planning on moving more examples over as we introduced testing to our actual code. Thanks!

[holdenk]

Moving example tests over would be rad :)

[luck02]

Ok, I'll clone this and start integrating them somewhere in the next week or so.