Install Delta with JAR

[MrPowers]

@dennyglee - Great question. Here's my understanding of the Python dependency management situation:

• pip installs stuff globally, which makes updating versions & switching between software environments hard
• The Python community built up abstractions that allow you to pip install stuff into different virtual environments, so stuff wasn't installed globally, and it was easier to switch between different software environments on a given machine. pip installing to a venv was a big improvement, but required manual work from the user.
• The Python web community built up two version managers to make the Python environment building process better. pipenv was gaining steam, but there were some community issues, and the project lost a bit of momentum. poetry filled the void and is now beloved by lots of Python web developers. It's also good for some PySpark project, see quinn for example. I can specify the dependencies in the pyproject.toml file and a fully deterministic build in the poetry.lock file. Most programming languages have fully deterministic builds out of the box, for Python this is relatively new. poetry is nice because you can clone the quinn project, create an environment on your machine, and have exactly all the dependency versions that I have on my end.
• While the Python web community was building pipenv / poetry, Anaconda was building conda to give Python data programmers a great user experience. conda is the best best if you want to build a complicated software environment for data projects. Let's say you want the latest version of Tensorflow (from a branch) and a bunch of other dependencies. From what I understand, you won't be able to build a "latest versions of everything" environment easily with poetry. So conda dominates in the Python data world.

From what I've seen, for web projects, something like poetry is definitely the best. It has deterministic, reproducible environments.

For Python data projects, conda is the best. I can send you this yaml file, you can run conda env create -f envs/pyspark-322-delta-121.yml and you can create a virtual environment that's roughly equivalent to what I have. You'll definitely have PySpark 3.2.2 & Delta 1.2.1 in that virtual environment, but there is no guarantee that conda resolves the other dependencies that aren't pinned to a specific version the same on your end.

Also note that conda and pip aren't mutually exclusive. You can pip install into a conda environment. That's what's being done in this environment file:

name: pyspark-322-delta-121
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.9
  - pyspark=3.2.2
  - pip
  - pip:
    - delta-spark==1.2.1

We're pip installing delta-spark via conda because delta-spark is only published to PyPi and isn't published to conda.

Sorry for the rambling response, haha. Feel free to ask more questions. I am still learning about this.

[dennyglee]

Not at all - this is extremely informative. I'm wondering if we should create an issue to add delta-lake to conda-forge then, eh?!

[bjornjorgensen]

@dennyglee +1

https://github.com/jupyter/docker-stacks/issues/1746 We are waiting for a new release. One of the things that are holding me back from using delta is that it doesn't follow spark releases. CC mathbunnyru Bidek56