search

amazon-archives / dynamodb-janusgraph-storage-backend

The Amazon DynamoDB Storage Backend for JanusGraph
Apache License 2.0
447 stars 99 forks source link

Gremlin Server un-responsive #260

Closed ghost closed 6 years ago

ghost commented 6 years ago

Hello,

We have followed the solution for the CloudFormation templates, and it appears that nothing is listening on the server on port 8182. This was checked by opening the SSH tunnels to the instance and checking through a browser and also via the instance by curling the localhost on that port. The pre-requisits such as the roles and placing the dynamodb file in the S3 bucket were followed.

Another observance, the SSH access Security Group parameter is opening port 22 to 0.0.0.0/32, is this intentional?

Thanks in advance!

udayth commented 6 years ago

We are running into the same issue. We think the problem is that Gremlin server installation never happens. After all the resources like EC2, VPCs, route tables etc have been created; we believe one or many of these steps are failing. there are no logs we could access to provide you with more info. Also, looks like Java 8 is not being installed which could be causing other failures.

"export SDKMAN_DIR=/usr/local/sdkman && curl -s https://get.sdkman.io | bash && source /usr/local/sdkman/bin/sdkman-init.sh\n"

udayth commented 6 years ago

BTW: If we run all the needed steps/software installations 'manually' on the instance; things work. Cloudformation template (the above steps specifically) is broken.

amcp commented 6 years ago

@bradyclo we default to no access via SSH because you, the customer needs to determine which host you will SSH from and modify the mask. I will try starting up an instance to see if I can identify the problem.

amcp commented 6 years ago

@udayth If Java 8 is not being installed, that is a problem. I will confirm/refute and report here.

udayth commented 6 years ago

@amcp Thanks for the response. Any chance you had some time to look into this? Any help is appreciated.

danieljue commented 6 years ago

I am having a similar issue where the cloud formation script executes the shell commands only partially, I still can't get the gremlin service to install. I have created a manual.sh file that I copy to the EC2 instance, in order to play with getting it to run. Here is my manual.sh:

#!/bin/bash export SDKMAN_DIR=/usr/local/sdkman && curl -s https://get.sdkman.io | bash && source /usr/local/sdkman/bin/sdkman-init.sh echo 'export SDKMAN_DIR=/usr/local/sdkman; source /usr/local/sdkman/bin/sdkman-init.sh' > /etc/profile.d/sdkman.sh yum update && yum upgrade && yum install -y java-1.8.0-openjdk yum remove java-1.7.0-openjdk -y sdk install maven < /dev/null && set -x export GREMLIN_SERVER_USERNAME='ec2-user' export LOG_DIR=/var/log/gremlin-server export SERVER_DIRNAME=dynamodb-janusgraph-storage-backend-1.2.0 export SERVER_ZIP=${SERVER_DIRNAME}.zip export PACKAGES_DIR=/usr/local/packages export INSTALL_DIR=${PACKAGES_DIR}/${SERVER_DIRNAME} export REPO_ARCHIVE_DIR=/dev/shm/dynamodb-janusgraph-storage-backend-master mkdir -p ${LOG_DIR} ${INSTALL_DIR} export SERVICE_SCRIPT=${INSTALL_DIR}/bin/gremlin-server-service.sh pushd /dev/shm wget https://github.com/awslabs/dynamodb-janusgraph-storage-backend/archive/master.zip && unzip -q master.zip pushd ${REPO_ARCHIVE_DIR} src/test/resources/install-gremlin-server.sh && popd && popd pushd ${PACKAGES_DIR} mv ${REPO_ARCHIVE_DIR}/server/${SERVER_DIRNAME} . && rm -rf /dev/shm/* && chmod u+x ${SERVICE_SCRIPT} && ln -s ${SERVICE_SCRIPT} /etc/init.d/gremlin-server && chkconfig --add gremlin-server export BACKEND_PROPERTIES=${INSTALL_DIR}/conf/gremlin-server/dynamodb.properties aws s3 cp s3://ironnet-labs-cognition/dynamodb.properties ${BACKEND_PROPERTIES} chown -R ${GREMLIN_SERVER_USERNAME}:${GREMLIN_SERVER_USERNAME} ${LOG_DIR} ${INSTALL_DIR} ln -s ${INSTALL_DIR}/conf /home/ec2-user/conf && chmod a+r /home/ec2-user/conf service gremlin-server start

Confirming maven and java from the above:
[ec2-user@ip-10-0-0-68 ~]$ mvn -version Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) Maven home: /usr/local/sdkman/candidates/maven/current Java version: 1.7.0_161, vendor: Oracle Corporation Java home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.161.x86_64/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "4.9.20-11.31.amzn1.x86_64", arch: "amd64", family: "unix" [ec2-user@ip-10-0-0-68 ~]$

Looks like perhaps Java 1.7 needs to be removed.

Here is the output from running this manual.sh as the EC2 user on the EC2 instance, note that I choose 'N' when asked to overwrite the unzipped master.zip files. There are some maven errors about checkstyle not having permmisions to some dirs.

`[ec2-user@ip-10-0-0-68 ~]$ ./manual.sh 

                            -+syyyyyyys:
                        `/yho:`       -yd.
                     `/yh/`             +m.
                   .oho.                 hy                          .`
                 .sh/`                   :N`                `-/o`  `+dyyo:.
               .yh:`                     `M-          `-/osysoym  :hs` `-+sys:      hhyssssssssy+
             .sh:`                       `N:          ms/-``  yy.yh-      -hy.    `.N-````````+N.
           `od/`                         `N-       -/oM-      ddd+`     `sd:     hNNm        -N:
          :do`                           .M.       dMMM-     `ms.      /d+`     `NMMs       `do
        .yy-                             :N`    ```mMMM.      -      -hy.       /MMM:       yh
      `+d+`           `:/oo/`       `-/osyh/ossssssdNMM`           .sh:         yMMN`      /m.
     -dh-           :ymNMMMMy  `-/shmNm-`:N/-.``   `.sN            /N-         `NMMy      .m/
   `oNs`          -hysosmMMMMydmNmds+-.:ohm           :             sd`        :MMM/      yy
  .hN+           /d:    -MMMmhs/-.`   .MMMh   .ss+-                 `yy`       sMMN`     :N.
 :mN/           `N/     `o/-`         :MMMo   +MMMN-         .`      `ds       mMMh      do
/NN/            `N+....--:/+oooosooo+:sMMM:   hMMMM:        `my       .m+     -MMM+     :N.

/NMo -+ooooo+/:-.......:+hNMN.NMMMd.MM/ -m: oMMN. hs -NMd :mm -MMMm- .s/ -MMm. /m- mMMd -N. mMM/ .- /MMh. -dMo -MMMy od. .MMMs..---yh +MMM. sNo.sNMM+ :MMMM/ sh+MMMNmNm+++- mMMM- /--ohmMMM+ :MMMMm.hyymmmdddo MMMMh. `-+yy/`yMMM/ :MMMMMy -sm:.``..-:-.` dMMMMmo-.``````..-:/osyhddddho. `+shdh+. hMMM: :MmMMMM/ ./yy/` `:sys+/+sh/ .dMMMMMMmdddddmmNMMMNNNNNMMMMMs sNdo- dMMM- `-/yd/MMMMm-:sy+. :hs- /N` `/ymNNNNNNNmmdys+/::----/dMMm: +m- mMMM+ohmo/.` sMMMMdo- .om: `sh `.-----+/.` `.-+hh/` `od. NMMNmds/ `mmy:` +mMy `:yy. /moyso+//+ossso:. .yy` `dy+:` .. :MMMN+---/oys: /+m: `.-:::-` /d+ +MMMMMMMNh:` +MN/ -yh. `+hddhy+. /MM+ .sh: :NMo -sh/ -NMs `/yy: .NMy `:sh+. `mMm` ./yds- `dMMMmyo:-..-:oymNy: +NMMMMMMMMMMMMMMMMms: -+shmNMMMNmdy+:`

                                                             Now attempting installation...

Looking for a previous installation of SDKMAN... SDKMAN found.

====================================================================================================== You already have SDKMAN installed. SDKMAN was found at:

/usr/local/sdkman

Please consider running the following if you need to upgrade.

$ sdk selfupdate force

======================================================================================================

./manual.sh: line 3: /etc/profile.d/sdkman.sh: Permission denied Loaded plugins: priorities, update-motd, upgrade-helper You need to be root to perform this command. Loaded plugins: priorities, update-motd, upgrade-helper You need to be root to perform this command.

Stop! maven 3.5.2 is already installed.

master.zip.1 [ <=> ] 752.65K --.-KB/s in 0.1s

2018-03-15 19:41:53 (7.25 MB/s) - ‘master.zip.1’ saved [770709]

danieljue commented 6 years ago

It seems using yum to install mvn pulls in Java 1.7 as a dep. Yum removing Java 1.7 removes Maven as a dep!

danieljue commented 6 years ago

Update, I was able to get the server to install and run. I added a line to install java via SDKMAN. Here is the partial gist of the script you can put in dynamodb-janusgraph-storage-backend-cfn.yaml, replacing the existing ec2 bash scripts:

https://gist.github.com/danieljue/93e484de9ca9038a98526bf302390abe

sandeepdoctily commented 6 years ago

One more issue I faced when installing on t2.micro is space on "/dev/shm" which causes failure of script. Can we move to "/tmp" so that it can work on all types of instances?

sandeepdoctily commented 6 years ago

@danieljue @udayth @amcp

I can confirm this that the template is now working for t2.micro instance as well when I replace the "/dev/shm" to "/tmp/shm".

[ec2-user@ip-10-0-0-72 ~]$ service gremlin-server status Gremlin Server is running with PID: 9080

Thanks for help.

amcp commented 6 years ago

fixed in #263