jefflester / minitrino

A tool that makes it easy to run modular Trino environments locally.
Apache License 2.0
32 stars 3 forks source link

STARBURST_LIC_PATH property for Minipresto configuration is not used #23

Closed joechu1 closed 3 years ago

joechu1 commented 4 years ago

It doesn't look like the STARBURST_LIC_PATH property in the Minipresto configuration is used when provisioning a cluster. More importantly, once the STARBURST_LIC_PATH property is set and the volume is uncommented in the docker-compose.yml, Minipresto will return an error when provisioning:

[i]  Creating presto       ...
[i]  Creating presto       ... error
[i]  ERROR: for presto  Cannot create container for service presto: create .: volume name is too short, names should be at least two alphanumeric characters
[i]  Starting ranger-admin ... done
[i]  ERROR: for presto  Cannot create container for service presto: create .: volume name is too short, names should be at least two alphanumeric characters
[i]  Encountered errors while bringing up the project.
[w]  Rolling back provisioned resources due to errors encountered while provisioning the environment.
[e]  Failed to execute shell command:
[e]  COMPOSE_PROJECT_NAME="minipresto" STARBURST_VER="338-e.0" ELASTICSEARCH_VER="7.6.2" MYSQL_VER="5" OPEN_LDAP_VER="1.3.0" POSTGRES_VER="11"
     POSTGRES_HIVE_HMS_VER="11" POSTGRES_EVENT_LOGGER_VER="11" POSTGRES_RANGER_VER="12.2" SEP_RANGER_VER="2.0.24" KIBANA_VER="ver" MEMSQL_VER="ver"
     MONGODB_VER="ver" SQL_SERVER_VER="ver" S3_ENDPOINT="s3.us-west-2.amazonaws.com" S3_ACCESS_KEY="*****"
     S3_SECRET_KEY="*****" AWS_REGION="us-west-2" SNOWFLAKE_DIST_CONNECT_URL="" SNOWFLAKE_DIST_CONNECT_USER=""
     SNOWFLAKE_DIST_CONNECT_PASSWORD="" SNOWFLAKE_DIST_WAREHOUSE="" SNOWFLAKE_DIST_DB="" SNOWFLAKE_DIST_STAGE_SCHEMA="" SNOWFLAKE_JDBC_CONNECT_URL=""
     SNOWFLAKE_JDBC_CONNECT_USER="" SNOWFLAKE_JDBC_CONNECT_PASSWORD="" SNOWFLAKE_JDBC_WAREHOUSE="" SNOWFLAKE_JDBC_DB="" SNOWFLAKE_JDBC_STAGE_SCHEMA="" \
[e]  docker-compose -f /Users/joechu/git/minipresto/lib/docker-compose.yml \
[e]  -f /Users/joechu/git/minipresto/lib/modules/security/system-ranger/system-ranger.yml \
[e]  -f /Users/joechu/git/minipresto/lib/modules/security/ldap/ldap.yml \
[e]  up -d
[e]  Exit code: 1

So for some reason, STARBURST_LIC_PATH is not set as an environmental variable. Manually setting an environmental variable export STARBURST_LIC_PATH= or using the flag minipresto provision --env STARBURST_LIC_PATH= works as a workaround.

Minipresto version - commit 419edb49866c604e6df5dca1e6d7b44aa6d84444

minipresto.cfg

[CLI]
LIB_PATH=/Users/joechu/git/minipresto/lib
TEXT_EDITOR=/usr/bin/vim

[DOCKER]
DOCKER_HOST=unix:///var/run/docker.sock

[PRESTO]
STARBURST_LIC_PATH=/Users/joechu/Documents/Licenses/signed_trial.license
CONFIG=
    query.max-memory-per-node=8GB
    query.max-total-memory-per-node=8GB
JVM_CONFIG=
    -Xmx16G

[MODULES]
S3_ENDPOINT=s3.us-west-2.amazonaws.com
S3_ACCESS_KEY=*****
S3_SECRET_KEY=*****
AWS_REGION=us-west-2

SNOWFLAKE_DIST_CONNECT_URL=
SNOWFLAKE_DIST_CONNECT_USER=
SNOWFLAKE_DIST_CONNECT_PASSWORD=
SNOWFLAKE_DIST_WAREHOUSE=
SNOWFLAKE_DIST_DB=
SNOWFLAKE_DIST_STAGE_SCHEMA=

SNOWFLAKE_JDBC_CONNECT_URL=
SNOWFLAKE_JDBC_CONNECT_USER=
SNOWFLAKE_JDBC_CONNECT_PASSWORD=
SNOWFLAKE_JDBC_WAREHOUSE=
SNOWFLAKE_JDBC_DB=
SNOWFLAKE_JDBC_STAGE_SCHEMA=

docker-compose.yml

version: "3.7"
services:

  presto:
    build:
      context: .
      args:
        STARBURST_VER: "${STARBURST_VER}"
      labels:
        - "com.starburst.tests=minipresto"
        - "com.starburst.tests.module=presto"
    image: "minipresto/presto:${STARBURST_VER}"
    container_name: "presto"
    hostname: "presto.minipresto.starburstdata.com"
    labels:
      - "com.starburst.tests=minipresto"
      - "com.starburst.tests.module=presto"
    # Uncomment this to enable the volume mount. The variable should point to a
    # valid SEP license.
    volumes:
      - "${STARBURST_LIC_PATH}:/usr/lib/presto/etc/starburstdata.license:ro"
    ports:
      - "8080:8080"
jefflester commented 3 years ago

@joechu1 sorry for the long delay here. I've been working on a sizable update and just got it merged––this should now work as expected with the latest changes.

Manually setting an environmental variable export STARBURST_LIC_PATH=

That method is currently unsupported. I could add logic to consume already-set shell variables, but we'd need to decide if a shell environment variable should override an existing config if already set in minipresto.cfg. The only way to pass variables to Minipresto right now is in minipresto.cfg and the --env option.

Setting STARBURST_LIC_PATH in minipresto.cfg or passing as an --env option should now work as long as that volume mount is uncommented in your docker-compose.yml file.

I'm going to close this, but please let me know if you still experience issues.