Deephaven Community Core

Deephaven Community Core is a real-time, time-series, column-oriented analytics engine with relational database features. Queries can seamlessly operate upon both historical and real-time data. Deephaven includes an intuitive user experience and visualization tools. It can ingest data from a variety of sources, apply computation and analysis algorithms to that data, and build rich queries, dashboards, and representations with the results.

Deephaven Community Core is the open version of Deephaven Enterprise, which functions as the data backbone for prominent hedge funds, banks, and financial exchanges.

This README is intended to provide a high-level overview of the installation and use of Deephaven Community Core. For more detailed guides on the topics presented below, see our Community documentation.

Supported Languages

Deephaven's client APIs use gRPC, protobuf, Apache Arrow Flight, and Barrage to handle ticking data. Users who wish to build their own client APIs can use these tools to do so.

The following list contains documentation links for installation instructions and more:

• Python
  • Run from Docker
  • pip-installed
• Groovy
  • Run from Docker
• Python client
• Java client
• JS client
• Go client
• R client

Install and run Deephaven

The Deephaven server can be installed and instantiated from Docker, from Python, or from source code.

From Docker

This is the easiest way to get started with Deephaven. For complete instructions, see our quickstart for Docker. The table below shows installation dependencies.

The quickest way to install and run Deephaven from Docker is with a single Docker command:

Python without Docker Compose

# Python
docker run --rm --name deephaven -p 10000:10000 ghcr.io/deephaven/server:latest

Groovy without Docker Compose

# Groovy
docker run --rm name deephaven -p 10000:10000 ghcr.io/deephaven/server-slim:latest

Users who wish to customize their deployment should use Docker Compose. Deephaven offers a multitude of pre-made docker-compose.yml files to choose from. To get started, all that's required is to download a file, pull the images, and start the server.

Python with Docker Compose

The base Python docker-compose.yml file can be found here.

mkdir deephaven-deployment
cd deephaven-deployment

curl -O https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/base/docker-compose.yml

docker compose pull
docker compose up

Groovy with Docker Compose

The base Groovy docker-compose.yml file can be found here.

mkdir deephaven-deployment
cd deephaven-deployment

curl -O https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/groovy/docker-compose.yml

docker compose pull
docker compose up

pip-installed Deephaven

Users who wish to use Python but not Docker should use pip-installed Deephaven. For users with Windows operating systems, WSL is not required to use Deephaven this way.

pip install --upgrade pip setuptools wheel
pip install deephaven-server deephaven-ipywidgets

Then, from Python:

from deephaven_server import Server
s = Server(port=10000, jvm_args=["-Xmx4g"]).start()

The input arguments to Server specify to bind to the Deephaven server on port 10000 and to allocate 4GB of memory to the server JVM.

Built from source

Users who wish to modify source code and contribute to the project should build Deephaven from source. For complete instructions, see How to build Deephaven from source.

Building and running Deephaven requires a few software packages.

You can check if these packages are installed and functioning by running:

git version
java -version
docker version
docker compose version
docker run hello-world

NOTE: Internally, the Java build process will use Gradle Auto Provisioning to download and use the appropriate Java version for building and testing.

NOTE: On Windows, all commands must be run inside a WSL 2 terminal.

Python

A Python virtual environment is highly recommended for building Deephaven from source. Additionally, the wheel is installed with pip and built with Gradle.

git clone https://github.com/deephaven/deephaven-core.git
cd deephaven-core
python3 -m venv /tmp/my-dh-venv
source /tmp/my-dh-venv/bin/activate
./gradlew py-server:assemble
pip install "py/server/build/wheel/deephaven_core-<version>-py3-non-any.whl[autocomplete]
./gradlew server-jetty-app:run

Groovy

The Groovy server is built with Gradle. -Pgroovy builds the Groovy server instead of Python.

git clone https://github.com/deephaven/deephaven-core.git
cd deephaven-core
./gradlew server-jetty-app:run -Pgroovy

Debugging

You can debug the server by adding the -Pdebug flag, and then attaching a debugger to port 5005. This can be used in conjunction with other flags. For example, if you wanted to debug a server and startup with Groovy:

./gradlew server-jetty-app:run -Pgroovy -Pdebug

Get the authentication key

Deephaven, by default, uses pre-shared key authentication to authenticate against unauthorized access.

Deephaven run from Docker

The pre-shared key is printed to the Docker logs when the server is started. Set your own key with the configuration parameter -Dauthentication.psk=<YourKey>. For users running Deephaven via Docker, this is set in the environment section of a docker-compose.yml file, or as a space-separated configuration parameter at the end of the docker run command.

To find the pre-shared key in the Docker logs:

docker compose logs -f | grep "access through pre-shared key"

Deephaven run from Python

When a Deephaven server is started from Python, executing Deephaven queries from Python does not require the key. However, if you wish to connect to the IDE via your web browser, you will need the pre-shared key. You will not be able to get the pre-shared key unless you set it yourself. To set the pre-shared key, add "-Dauthentication.psk=<YourKey>" as an additional JVM parameter to the server. The following example sets the key to MyPreSharedKey:

from deephaven_server import Server
s = Server(port=10000, jvm_args=["-Xmx4g", "-Dauthentication.psk=MyPreSharedKey"]).start()

Client APIs

Clients that attempt to connect to a server using pre-shared key authentication will need to supply the key to complete the connection. The key is the same for a client connection as it is for connecting directly to the server. For instance, in the above example, the key for a client connection would also be MyPreSharedKey.

Connect to the server

The Deephaven UI is accessible from a web browser. For a server running locally on port 10000, it can be connected to via https://localhost:10000/ide. For a server running remotely on port 10000, it can be connected to via https://<hostname>:10000/ide. If using authentication, enter credentials to gain access to the IDE. For information on supported browsers, see here.

First query

From the Deephaven IDE, you can perform your first query.

The scripts below create two small tables: one for employees and one for departments. They are joined on the DeptID column to show the name of the department where each employee works.

Python

from deephaven import new_table
from deephaven.column import string_col, int_col
from deephaven.constants import NULL_INT

left = new_table([
        string_col("LastName", ["Rafferty", "Jones", "Steiner", "Robins", "Smith", "Rogers"]),
        int_col("DeptID", [31, 33, 33, 34, 34, NULL_INT]),
        string_col("Telephone", ["(347) 555-0123", "(917) 555-0198", "(212) 555-0167", "(952) 555-0110", None, None])
    ])

right = new_table([
        int_col("DeptID", [31, 33, 34, 35]),
        string_col("DeptName", ["Sales", "Engineering", "Clerical", "Marketing"]),
        string_col("Telephone", ["(646) 555-0134", "(646) 555-0178", "(646) 555-0159", "(212) 555-0111"])
    ])

t = left.join(right, "DeptID", "DeptName, DeptTelephone=Telephone")

Groovy

left = newTable(
        string_col("LastName", "Rafferty", "Jones", "Steiner", "Robins", "Smith", "Rogers"),
        int_col("DeptID", 31, 33, 33, 34, 34, NULL_INT),
        string_col("Telephone", "(347) 555-0123", "(917) 555-0198", "(212) 555-0167", "(952) 555-0110", null, null)
    )

right = newTable(
        intCol("DeptID", 31, 33, 34, 35),
        stringCol("DeptName", "Sales", "Engineering", "Clerical", "Marketing"),
        stringCol("Telephone", "(646) 555-0134", "(646) 555-0178", "(646) 555-0159", "(212) 555-0111")
    )

t = left.join(right, "DeptID", "DeptName, DeptTelephone=Telephone")

Resources

• Help!
• Deephaven Community Slack
• Discussions
• Deephaven Community Core docs
• Java API docs
• Python API docs

Contributing

See CONTRIBUTING for full instructions on how to contribute to this project.

Code Of Conduct

This project has adopted the Contributor Covenant Code of Conduct. For more information see the Code of Conduct or contact opencode@deephaven.io with any additional questions or comments.

License

Copyright © 2016-2023 Deephaven Data Labs and Patent Pending. All rights reserved.

Provided under the Deephaven Community License.