Can't load Athena JDBC plugin

chharish commented 5 years ago

I'm trying to connect Athena JDBC plugin to Superset

I get an error ERROR: {"error": "Connection failed!\n\nThe error message returned was:\nCan't load plugin: sqlalchemy.dialects:awsathena.jdbc"}

Same with rest plugin: ERROR: {"error": "Connection failed!\n\nThe error message returned was:\nCan't load plugin: sqlalchemy.dialects:awsathena.rest"}

Am I missing something? should I load the plugin manually?

chharish commented 5 years ago

I found that there seems to be a solution for this #13 but I couldn't understand what files to edit and where to add the pip install

chharish commented 5 years ago

I tried building a new image to add/install the database-dependencies but the build process hangs at step 34

Step 20/38 : RUN mkdir -p /home/superset/config
 ---> Using cache
 ---> cfdb0a76b552
Step 21/38 : RUN pip install --upgrade setuptools pip
 ---> Using cache
 ---> 221a78fe1607
Step 22/38 : RUN pip install -r requirements.txt
 ---> Using cache
 ---> 56b24524b96e
Step 23/38 : RUN pip install -r requirements-dev.txt
 ---> Using cache
 ---> 2177092eaa44
Step 24/38 : RUN pip install -e .
 ---> Using cache
 ---> e6f4df4ae9bf
Step 25/38 : COPY docker-files/database-dependencies.txt .
 ---> Using cache
 ---> 46d40b80992b
Step 26/38 : RUN pip install -r database-dependencies.txt
 ---> Using cache
 ---> 5dbca9ff5f5f
Step 27/38 : ENV PATH=${SUPERSET_HOME}/superset/bin:$PATH     PYTHONPATH=${SUPERSET_HOME}superset/:${SUPERSET_HOME}config/:$PYTHONPATH
 ---> Using cache
 ---> e286118133e2
Step 28/38 : COPY script/docker-entrypoint.sh /usr/local/bin/
 ---> Using cache
 ---> bbd99b9a1b42
Step 29/38 : RUN chmod +x /usr/local/bin/docker-entrypoint.sh
 ---> Using cache
 ---> 7b7f0ec8fbc6
Step 30/38 : RUN ln -s /usr/local/bin/docker-entrypoint.sh /entrypoint.sh # backwards compat
 ---> Using cache
 ---> 1a96ed741a01
Step 31/38 : COPY config/superset_config.py ${SUPERSET_HOME}config/
 ---> Using cache
 ---> 4f432a41b7ed
Step 32/38 : RUN chown -R superset:superset $SUPERSET_HOME
 ---> Using cache
 ---> ec5e1c50ac78
Step 33/38 : USER superset
 ---> Using cache
 ---> 972aef3c8b47
Step 34/38 : RUN cd superset/assets && yarn
 ---> Running in c69464913303
yarn install v1.15.2
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
info fsevents@1.2.4: The platform "linux" is incompatible with this module.
info "fsevents@1.2.4" is an optional dependency and failed compatibility check. Excluding it from installation.
[4/5] Linking dependencies...
warning " > @data-ui/event-flow@0.0.54" has unmet peer dependency "aphrodite@^1.2.0".
warning "@data-ui/event-flow > recompose@0.23.5" has incorrect peer dependency "react@^0.14.0 || ^15.0.0".
warning " > react-addons-css-transition-group@15.6.2" has incorrect peer dependency "react@^15.4.2".
warning " > react-addons-test-utils@15.6.2" has incorrect peer dependency "react-dom@^15.4.2".
warning " > react-test-renderer@15.6.2" has incorrect peer dependency "react@^15.6.2".
[5/5] Building fresh packages...
Done in 233.78s.

chharish commented 5 years ago

@abhioncbr Any chance to get this resolved, please?

abhioncbr commented 5 years ago

This step will take sometime, and their is a possibility that it will fail in your local environment because of low memory issue. You can submit your fix as a PR, I will test it and merge to main branch in coming couple of days.

chharish commented 5 years ago

I'm running it on a AWS Ec2 server. I created a new image with dependencies that I require.

mysqlclient==1.3.13                     # MySQL         mysql://
#cx_Oracle==7.0.0                       # Oracle        oracle://
#snowflake-sqlalchemy==1.1.6            # Snowflake     snowflake://
sqlalchemy-redshift==0.7.2              # Redshift      redshift+psycopg2://
pymssql==2.1.4                      # MSSQL             mssql://
#impyla==0.14.1                     # Impala        impala://
PyAthenaJDBC==2.0.1                     # Athena            awsathena+jdbc://
PyAthena==1.4.4                     # Athena        awsathena+rest://
#sqlalchemy-vertica-python==0.4.3       # Vertica           vertica+vertica_python://
#sqlalchemy-clickhouse==0.1.5.post0     # ClickHouse    clickhouse://
#kylinpy==2.1.2                     # Kylin             kylin://
#pybigquery==0.4.8                      # BigQuery          bigquery://
#sqlalchemy-teradata==0.9.0.dev0            # Teradata      teradata://
#psycopg2==2.7.6.1                      # Postgres, Greenplum       postgresql+psycopg2://
#PyHive==0.6.1                      # Presto  presto://, Hive    hive://, SparkSQL    jdbc+hive://

The docker with the new image now enters a restarting loop because couldn't start health check. I first tested it with nginx reverse proxy and then without reverse proxy. Same issue.

 [ec2-user@ip-10-0-1-71 docker-files]$ docker ps
CONTAINER ID        IMAGE                                     COMMAND                  CREATED             STATUS                            PORTS                                 NAMES
e1056e72de77        abhioncbr/docker-superset:0.29.0rc5-poc   "docker-entrypoint.sh"   6 minutes ago       Up 3 seconds (health: starting)   80/tcp, 443/tcp, 5555/tcp, 8088/tcp   docker-files_superset_1_6a56cd63bc7e
804496a1dd8e        mysql:5.7                                 "docker-entrypoint.s…"   6 minutes ago       Up 6 minutes                      0.0.0.0:3306->3306/tcp, 33060/tcp     docker-files_mysql_1_bb68f49dda83
e66b9947fcc2        redis:3.2                                 "docker-entrypoint.s…"   6 minutes ago       Up 6 minutes                      0.0.0.0:6379->6379/tcp                docker-files_redis_1_86e554da3cff
[ec2-user@ip-10-0-1-71 docker-files]$ docker ps
CONTAINER ID        IMAGE                                     COMMAND                  CREATED             STATUS                         PORTS                               NAMES
e1056e72de77        abhioncbr/docker-superset:0.29.0rc5-poc   "docker-entrypoint.sh"   7 minutes ago       Restarting (2) 7 seconds ago                                       docker-files_superset_1_6a56cd63bc7e
804496a1dd8e        mysql:5.7                                 "docker-entrypoint.s…"   7 minutes ago       Up 7 minutes                   0.0.0.0:3306->3306/tcp, 33060/tcp   docker-files_mysql_1_bb68f49dda83
e66b9947fcc2        redis:3.2                                 "docker-entrypoint.s…"   7 minutes ago       Up 7 minutes                   0.0.0.0:6379->6379/tcp              docker-files_redis_1_86e554da3cff

I'm now back at where I started 7 days ago :( Any help is really appreciated.

chharish commented 5 years ago

The docker logs shows error at line 125 in docker-entrypoint.sh in the script.

/usr/local/bin/docker-entrypoint.sh: line 125: syntax error: unexpected end of file
Environment Variable: SUPERSET_ENV: local
Environment Variable Exported: INVOCATION_TYPE: COMPOSE
Starting Initialization[if needed]
Apache-Superset Already Initialized.
Container deployment type: local
/usr/local/bin/docker-entrypoint.sh: line 125: syntax error: unexpected end of file
Environment Variable: SUPERSET_ENV: local
Environment Variable Exported: INVOCATION_TYPE: COMPOSE
Starting Initialization[if needed]
/usr/local/bin/docker-entrypoint.sh: line 125: syntax error: unexpected end of file
Apache-Superset Already Initialized.
Container deployment type: local

Not sure if this has anything to do with the restart loop issue.

chharish commented 5 years ago

Not an issue with the container. I was able to recreate a new image using background processes and made sure the session doesn't time out when the image is under creation. I only installed AWSAthenaJDBC/REST drivers. I don't think a merge is necessary as it is pretty straight forward, thanks to you, but it would be better if there are plugin arguments to pass instead of creating a whole new image just to integrate a single plugin dependency and using the new image instead.

Thanks for the help.

abhioncbr commented 5 years ago

@chharish glad that it's working now. I think, whatever the small change you did in DockerFile, it's good to incorporate so that others wouldn't face the issue.

abhioncbr / docker-superset

Can't load Athena JDBC plugin #24