search

amazon-archives / dynamodb-janusgraph-storage-backend

The Amazon DynamoDB Storage Backend for JanusGraph
Apache License 2.0
447 stars 99 forks source link

Getting Started without Docker #266

Closed nickbabcock closed 6 years ago

nickbabcock commented 6 years ago

The pre-packaged Janusgraph distribution comes with Cassandra/ES forked into a separate process. Since we're using dynamodb here, I figured we'd need to deploy the Janusgraph server alongside the dynamodb-janusgraph-storage-backend jar and update the storage.backend to reference this bridge. All instructions use docker, so I poked around the source code to see how to deploy myself.

Steps I took:

java.lang.IllegalArgumentException: Could not instantiate implementation: com.amazon.janusgraph.diskstorage.dynamodb.DynamoDBStoreManager
Caused by: java.lang.NoClassDefFoundError: com/amazonaws/auth/AWSCredentialsProvider
        at com.amazon.janusgraph.diskstorage.dynamodb.DynamoDBStoreManager.<init>(DynamoDBStoreManager.java:98)

I'm guessing I missed a step, so my question is, assuming dynamodb is set up according to the cloud formation templates, and the ec2 instance can communicate appropriately with it -- how does one deploy the Janusgraph server with dynamodb-janusgraph-storage-backend without docker?

danieljue commented 6 years ago

Did the gremlin-server installation script succeed during cloud formation? I had to modify the cloudformation scripts a bit to make it work for me, there was a Java version issue with Maven in my case, running on a M4Xlarge in West 2.

If the install was successful, the gremlin service should be started automatically every time the ec2 instance is started, through init.d I added some log files for the various stages of the cloud formation, so i could check where the install was breaking down. Here is the UserData portion of my storage backend yaml cloud formation script, as of last week:

Edit: copy and pasted directly from the cloud formation template that ran:

UserData: Fn::Base64: Fn::Join:

  • ''
    • "#!/bin/bash\n"
  • "export SDKMAN_DIR=/usr/local/sdkman && curl -s https://get.sdkman.io | bash && source /usr/local/sdkman/bin/sdkman-init.sh\n"
  • "echo 'export SDKMAN_DIR=/usr/local/sdkman; source /usr/local/sdkman/bin/sdkman-init.sh' > /etc/profile.d/sdkman.sh\n"
  • "yum update -y && yum upgrade -y && yum install -y java-1.8.0-openjdk > /home/ec2-user/yumupdates.log\n"
  • "yum remove -y java-1.7.0-openjdk > /home/ec2-user/yumremovejava7.log\n"
  • "java -version > /home/ec2-user/java-version-before-sdkmvn.log\n"
  • "sdk install java < /dev/null\n"
  • "sdk current java > /home/ec2-user/java-version-from-sdkman.log\n"
  • "sdk install maven < /dev/null && set -x\n"
  • "mvn -version > /home/ec2-user/maven-installation-settings.log\n"
  • "export GREMLIN_SERVER_USERNAME='ec2-user'\n"
  • "export LOG_DIR=/var/log/gremlin-server\n"
  • "export SERVER_DIRNAME=dynamodb-janusgraph-storage-backend-1.2.0\n"
  • "export SERVER_ZIP=${SERVER_DIRNAME}.zip\n"
  • "export PACKAGES_DIR=/usr/local/packages\n"
  • "export INSTALL_DIR=${PACKAGES_DIR}/${SERVER_DIRNAME}\n"
  • "export REPO_ARCHIVE_DIR=/dev/shm/dynamodb-janusgraph-storage-backend-master\n"
  • "mkdir -p ${LOG_DIR} ${INSTALL_DIR}\n"
  • "export SERVICE_SCRIPT=${INSTALL_DIR}/bin/gremlin-server-service.sh\n"
  • "pushd /dev/shm\n"
  • "wget https://github.com/awslabs/dynamodb-janusgraph-storage-backend/archive/master.zip && unzip -q master.zip\n"
  • "pushd ${REPO_ARCHIVE_DIR}\n"
  • "src/test/resources/install-gremlin-server.sh > /home/ec2-user/gremlin-server-installation.log && popd && popd\n"
  • "pushd ${PACKAGES_DIR}\n"
  • "mv ${REPO_ARCHIVE_DIR}/server/${SERVER_DIRNAME} . && rm -rf /dev/shm/* && chmod u+x ${SERVICE_SCRIPT} && ln -s ${SERVICE_SCRIPT} /etc/init.d/gremlin-server && chkconfig --add gremlin-server\n"
  • "export BACKEND_PROPERTIES=${INSTALL_DIR}/conf/gremlin-server/dynamodb.properties\n"
  • "aws s3 cp "
  • Ref: "StorageBackendPropertiesFileS3Url"
  • " ${BACKEND_PROPERTIES}\n"
  • "chown -R ${GREMLIN_SERVER_USERNAME}:${GREMLIN_SERVER_USERNAME} ${LOG_DIR} ${INSTALL_DIR}\n"
  • "ln -s ${INSTALL_DIR}/conf /home/ec2-user/conf && chmod a+r /home/ec2-user/conf\n"
  • "service gremlin-server start > /home/ec2-user/gremlin-server-start.log\n\n"
danieljue commented 6 years ago

My last two log files look like this: image

If the gremlin-server-start.log is empty, then something went wrong with the install.

nickbabcock commented 6 years ago

So I think I figured out everything (I didn't use cloud formation for the EC2 instance, only for dynamodb).

The step I was missing was I needed to copy dynamodb-janusgraph-storage-backend jar dependencies into /opt/jg/lib.

I started up gremin-server, see it spouting metrics, and the below works:

gremlin> :remote connect tinkerpop.server conf/remote.yaml
==>Configured localhost/127.0.0.1:8182
gremlin> 10 - 9
==>1

Thanks for chiming in, I think I'm all set