Closed joebernard closed 8 years ago
Is this something you're running locally? Can you by any chance show the gremlin-server output of bin/titan.sh start -v
(you'll get a lot of output, the gremlin server part is the final one).
I've included the output below, but discovered something in the meantime.
When my server (an EC2 instance) initially starts, g
and graph
are not available to gremlin-javascript
. It is only after I stop
Titan, then start
it using bin/titan.sh start -v
, and then try to connect with gremlin-javascript
again that g
and graph
are available.
From the output I can see that g
is getting bound after I stop/start Titan. I'm stumped as to why gremlin-javascript
isn't seeing these objects until after the titan restart.
0 [main] INFO org.apache.tinkerpop.gremlin.server.GremlinServer -
\,,,/
(o o)
-----oOOo-(3)-oOOo-----
286 [main] INFO org.apache.tinkerpop.gremlin.server.GremlinServer - Configuring Gremlin Server from conf/gremlin-server/gremlin-server.yaml
.2081 [main] INFO com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - Entering ensureStore table:v100_system_properties
2864 [main] INFO com.thinkaurelius.titan.core.util.ReflectiveConfigOptionLoader - Loaded and initialized config classes: 12 OK out of 12 attempts in PT0.104S
.3946 [main] INFO com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - Closing table:v100_system_properties
3951 [main] INFO com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - Closing table:v100_system_properties
3954 [main] INFO com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration - Generated unique-instance-id=ac1f316d3098-ip-172-31-49-1091
3997 [main] INFO com.thinkaurelius.titan.diskstorage.Backend - Configuring index [search]
4157 [main] INFO org.elasticsearch.node - [Thinker] version[1.5.1], pid[3098], build[5e38401/2015-04-09T13:41:35Z]
4159 [main] INFO org.elasticsearch.node - [Thinker] initializing ...
4167 [main] INFO org.elasticsearch.plugins - [Thinker] loaded [], sites []
.7128 [main] INFO org.elasticsearch.node - [Thinker] initialized
7128 [main] INFO org.elasticsearch.node - [Thinker] starting ...
7137 [main] INFO org.elasticsearch.transport - [Thinker] bound_address {local[1]}, publish_address {local[1]}
7149 [main] INFO org.elasticsearch.discovery - [Thinker] elasticsearch/BTw38bJpRPWNxqfDjPzDkA
7152 [elasticsearch[Thinker][clusterService#updateTask][T#1]] INFO org.elasticsearch.cluster.service - [Thinker] master {new [Thinker][BTw38bJpRPWNxqfDjPzDkA][ip-172-31-49-109][local[1]]{local=true}}, removed {[Thinker][xl3xrB7WQneJwk25oFe53g][ip-172-31-49-109][local[1]]{local=true},}, reason: local-disco-initial_connect(master)
7334 [main] INFO org.elasticsearch.http - [Thinker] bound_address {inet[/0:0:0:0:0:0:0:0:9201]}, publish_address {inet[/172.31.49.109:9201]}
7334 [main] INFO org.elasticsearch.node - [Thinker] started
7881 [elasticsearch[Thinker][clusterService#updateTask][T#1]] INFO org.elasticsearch.gateway - [Thinker] recovered [1] indices into cluster_state
...............37886 [main] INFO com.thinkaurelius.titan.diskstorage.Backend - Initiated backend operations thread pool of size 2
37887 [main] INFO com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - Entering ensureStore table:v100_titan_ids
37937 [main] INFO com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - Entering ensureStore table:v100_edgestore
37958 [main] INFO com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - Entering ensureStore table:v100_graphindex
37977 [main] INFO com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - Entering ensureStore table:v100_txlog
38030 [main] INFO com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - Entering ensureStore table:v100_systemlog
38063 [main] INFO com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - Entering ensureStore table:v100_system_properties
38256 [main] INFO com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog - Loaded unidentified ReadMarker start time 2016-04-09T19:11:56.134Z into com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog$MessagePuller@14ed7ddf
38257 [main] INFO org.apache.tinkerpop.gremlin.server.GremlinServer - Graph [graph] was successfully configured via [conf/gremlin-server/dynamodb.properties].
38257 [main] INFO org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor - Initialized Gremlin thread pool. Threads in pool named with pattern gremlin-*
38812 [main] INFO org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines - Loaded nashorn ScriptEngine
.39602 [main] INFO org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines - Loaded gremlin-groovy ScriptEngine
40901 [main] INFO org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor - Initialized gremlin-groovy ScriptEngine with scripts/empty-sample.groovy
40907 [main] INFO org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor - Initialized GremlinExecutor and configured ScriptEngines.
40914 [main] INFO org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor - A GraphTraversalSource is now bound to [g] with graphtraversalsource[standardtitangraph[com.amazon.titan.diskstorage.dynamodb.DynamoDBStoreManager:[127.0.0.1]], standard]
41022 [main] INFO org.apache.tinkerpop.gremlin.server.AbstractChannelizer - Configured application/vnd.gremlin-v1.0+gryo with org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0
41022 [main] INFO org.apache.tinkerpop.gremlin.server.AbstractChannelizer - Configured application/vnd.gremlin-v1.0+gryo-stringd with org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0
41068 [main] INFO org.apache.tinkerpop.gremlin.server.AbstractChannelizer - Configured application/vnd.gremlin-v1.0+json with org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV1d0
41071 [main] INFO org.apache.tinkerpop.gremlin.server.AbstractChannelizer - Configured application/json with org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0
41193 [gremlin-server-boss-1] INFO org.apache.tinkerpop.gremlin.server.GremlinServer - Gremlin Server configured with worker thread pool of 1, gremlin pool of 8 and boss thread pool of 1.
41194 [gremlin-server-boss-1] INFO org.apache.tinkerpop.gremlin.server.GremlinServer - Channel started at port 8182.
. OK (connected to 127.0.0.1:8182).
Run gremlin.sh to connect.
[ec2-user@ip-172-31-49-109 dynamodb-titan100-storage-backend-1.0.0-hadoop1]$ 62497 [gremlin-server-worker-1] INFO org.apache.tinkerpop.gremlin.server.op.OpLoader - Adding the standard OpProcessor.
62498 [gremlin-server-worker-1] INFO org.apache.tinkerpop.gremlin.server.op.OpLoader - Adding the control OpProcessor.
62500 [gremlin-server-worker-1] INFO org.apache.tinkerpop.gremlin.server.op.OpLoader - Adding the session OpProcessor.
That does sound weird. Do you have any logs in the logs
folder of your titan for those initial launches (that don't seem to populate graph and g)?
I have a log
directory (sibling to bin
) that contains
-rw-rw-r-- 1 ec2-user ec2-user 242538 Apr 9 21:24 cassandra.log
-rw-rw-r-- 1 ec2-user ec2-user 0 Apr 9 18:15 elasticsearch_index_indexing_slowlog.log
-rw-rw-r-- 1 ec2-user ec2-user 0 Apr 9 18:15 elasticsearch_index_search_slowlog.log
-rw-rw-r-- 1 ec2-user ec2-user 6612 Apr 9 21:24 elasticsearch.log
-rw-rw-r-- 1 ec2-user ec2-user 232743 Apr 9 19:12 gremlin-server.log
gremlin-server.log only captured my manual attempt to start the server (showing g
is bound). It doesn't seem to capture the initial Titan startup. Is there another location for Titan specific logs? A quick find
doesn't seem to reveal any.
Hmm not that I recall. If you're using an image you haven't created on EC2 it's possible that an alternate log dir is being used (or logging is being suppressed). Try having a look inside /etc/init.d/
or /etc/init/
for any titan startup script (I'm assuming Debian, I don't know about other distros).
You were right, the AWS image specified a custom log file. Found it in /var/log/gremlin-server
. There is an error when it attempts to init graph
. It seems to indicate the graph is already open, but I'm not sure how that could be. This was captured after a server reboot, and is the only Titan server I'm using. Here are the contents:
0 [main] INFO org.apache.tinkerpop.gremlin.server.GremlinServer -
\,,,/
(o o)
-----oOOo-(3)-oOOo-----
266 [main] INFO org.apache.tinkerpop.gremlin.server.GremlinServer - Configuring Gremlin Server from /usr/local/packages/dynamodb-titan100-storage-backend-1.0.0-hadoop1/conf/gremlin-server/gremlin-server.yaml
2220 [main] INFO com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - Entering ensureStore table:v100_system_properties
3075 [main] INFO com.thinkaurelius.titan.core.util.ReflectiveConfigOptionLoader - Loaded and initialized config classes: 12 OK out of 12 attempts in PT0.098S
4151 [main] INFO com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - Closing table:v100_system_properties
4155 [main] INFO com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - Closing table:v100_system_properties
4158 [main] INFO com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration - Generated unique-instance-id=ac1f316d2400-ip-172-31-49-1091
4197 [main] INFO com.thinkaurelius.titan.diskstorage.Backend - Configuring index [search]
4400 [main] INFO org.elasticsearch.node - [Atom Bob] version[1.5.1], pid[2400], build[5e38401/2015-04-09T13:41:35Z]
4400 [main] INFO org.elasticsearch.node - [Atom Bob] initializing ...
4416 [main] INFO org.elasticsearch.plugins - [Atom Bob] loaded [], sites []
7331 [main] INFO org.elasticsearch.node - [Atom Bob] initialized
7331 [main] INFO org.elasticsearch.node - [Atom Bob] starting ...
7336 [main] INFO org.elasticsearch.transport - [Atom Bob] bound_address {local[1]}, publish_address {local[1]}
7346 [main] INFO org.elasticsearch.discovery - [Atom Bob] elasticsearch/9K4-3iseRpSYqVExWE4J9A
7348 [elasticsearch[Atom Bob][clusterService#updateTask][T#1]] INFO org.elasticsearch.cluster.service - [Atom Bob] master {new [Atom Bob][9K4-3iseRpSYqVExWE4J9A][ip-172-31-49-109][local[1]]{local=true}}, removed {[Atom Bob][E0dnS2NRS7C3Jz-vb_dirA][ip-172-31-49-109][local[1]]{local=true},}, reason: local-disco-initial_connect(master)
7489 [main] INFO org.elasticsearch.http - [Atom Bob] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/172.31.49.109:9200]}
7490 [main] INFO org.elasticsearch.node - [Atom Bob] started
8045 [elasticsearch[Atom Bob][clusterService#updateTask][T#1]] INFO org.elasticsearch.gateway - [Atom Bob] recovered [1] indices into cluster_state
38049 [main] INFO com.thinkaurelius.titan.diskstorage.Backend - Initiated backend operations thread pool of size 2
38049 [main] INFO com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - Entering ensureStore table:v100_titan_ids
38156 [main] INFO com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - Entering ensureStore table:v100_edgestore
38169 [main] INFO com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - Entering ensureStore table:v100_graphindex
38187 [main] INFO com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - Entering ensureStore table:v100_txlog
38220 [main] INFO com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - Entering ensureStore table:v100_systemlog
38246 [main] INFO com.amazon.titan.diskstorage.dynamodb.AbstractDynamoDBStore - Entering ensureStore table:v100_system_properties
38307 [main] WARN org.apache.tinkerpop.gremlin.server.GremlinServer - Graph [graph] configured at [conf/gremlin-server/dynamodb.properties] could not be instantiated and will not be available in Gremlin Server. GraphFactory message: GraphFactory could not instantiate this Graph implementation [class com.thinkaurelius.titan.core.TitanFactory]
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [class com.thinkaurelius.titan.core.TitanFactory]
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:82)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:70)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:104)
at org.apache.tinkerpop.gremlin.server.GraphManager.lambda$new$27(GraphManager.java:50)
at java.util.LinkedHashMap$LinkedEntrySet.forEach(LinkedHashMap.java:663)
at org.apache.tinkerpop.gremlin.server.GraphManager.<init>(GraphManager.java:48)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:94)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:88)
at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:290)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:78)
... 8 more
Caused by: com.thinkaurelius.titan.core.TitanException: A Titan graph with the same instance id [ac1f316d2400-ip-172-31-49-1091] is already open. Might required forced shutdown.
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.<init>(StandardTitanGraph.java:146)
at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:94)
at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:74)
... 13 more
38312 [main] INFO org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor - Initialized Gremlin thread pool. Threads in pool named with pattern gremlin-*
38908 [main] INFO org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines - Loaded nashorn ScriptEngine
39606 [main] INFO org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines - Loaded gremlin-groovy ScriptEngine
40778 [main] WARN org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor - Could not initialize gremlin-groovy ScriptEngine with scripts/empty-sample.groovy as script could not be evaluated - javax.script.ScriptException: groovy.lang.MissingPropertyException: No such property: graph for class: Script1
40778 [main] INFO org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor - Initialized GremlinExecutor and configured ScriptEngines.
40878 [main] WARN org.apache.tinkerpop.gremlin.server.AbstractChannelizer - Could not instantiate configured serializer class - org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0 - it will not be available. There is no graph named [graph] configured to be used in the useMapperFromGraph setting
40880 [main] INFO org.apache.tinkerpop.gremlin.server.AbstractChannelizer - Configured application/vnd.gremlin-v1.0+gryo-stringd with org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0
40923 [main] WARN org.apache.tinkerpop.gremlin.server.AbstractChannelizer - Could not instantiate configured serializer class - org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV1d0 - it will not be available. There is no graph named [graph] configured to be used in the useMapperFromGraph setting
40924 [main] WARN org.apache.tinkerpop.gremlin.server.AbstractChannelizer - Could not instantiate configured serializer class - org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0 - it will not be available. There is no graph named [graph] configured to be used in the useMapperFromGraph setting
41026 [gremlin-server-boss-1] INFO org.apache.tinkerpop.gremlin.server.GremlinServer - Gremlin Server configured with worker thread pool of 1, gremlin pool of 8 and boss thread pool of 1.
41026 [gremlin-server-boss-1] INFO org.apache.tinkerpop.gremlin.server.GremlinServer - Channel started at port 8182.
For reference I'm including the contents of gremlin-server-service.sh
which is the startup script for Titan (provided on the official AWS image for Titan).
# Init script for Gremlin Server so it automatically starts/stops with the machine.
#
# To install:
# 1) Add a symlink to this file in /etc/init.d/ under the name you'd like to see the service
# For example, to name the service "gremlin-server": ln -s /usr/local/packages/dynamodb-titan100-storage-backend-1.0.0-hadoop1/bin/gremlin-server-service.sh /etc/init.d/gremlin-server
# 2a) If you're running RH: chkconfig --add gremlin-server
# 2b) If you're running Ubuntu: update-rc.d gremlin-server defaults
#
# You have to SET the Gremlin Server installation directory here:
GREMLIN_SERVER_DIR="/usr/local/packages/dynamodb-titan100-storage-backend-1.0.0-hadoop1"
GREMLIN_SERVER_LOG_DIR="/var/log/gremlin-server"
# Specify the user to run Gremlin Server as:
GREMLIN_SERVER_USER="ec2-user"
# JAVA_OPTIONS only gets used on start
JAVA_OPTIONS="-server -Xms128m -Xmx512m -Dtitan.logdir=$GREMLIN_SERVER_LOG_DIR"
usage() {
echo "Usage: `basename $0`: <start|stop|status>"
exit 1
}
start() {
status
if [ $PID -gt 0 ]
then
echo "Gremlin Server has already been started. PID: $PID"
return $PID
fi
export JAVA_OPTIONS
echo "Starting Gremlin Server..."
su -c "cd \"$GREMLIN_SERVER_DIR\"; /usr/bin/nohup ./bin/gremlin-server.sh ${GREMLIN_SERVER_DIR}/conf/gremlin-server/gremlin-server.yaml 1>$GREMLIN_SERVER_LOG_DIR/service.log 2>$GREMLIN_SERVER_LOG_DIR/service.err &" $GREMLIN_SERVER_USER
}
stop() {
status
if [ $PID -eq 0 ]
then
echo "Gremlin Server has already been stopped."
return 0
fi
echo "Stopping Gremlin Server..."
su -c "kill -9 ${PID}" $GREMLIN_SERVER_USER
}
status() {
PID=`ps -ef | grep $GREMLIN_SERVER_USER | grep java | grep GremlinServer | grep -v grep | awk '{print $2}'`
if [ "x$PID" = "x" ]
then
PID=0
fi
# if PID is greater than 0 then Gremlin Server is running, else it is not
return $PID
}
if [ "x$1" = "xstart" ]
then
start
exit 0
fi
if [ "x$1" = "xstop" ]
then
stop
exit 0
fi
if [ "x$1" = "xstatus" ]
then
status
if [ $PID -gt 0 ]
then
echo "Gremlin Server is running with PID: $PID"
else
echo "Gremlin Server is NOT running"
fi
exit $PID
fi
usage
I need to head out so I'll give this some more thought. But I don't think it's an issue with the init script. A shot in the dark here would be to consider that you may have a lingering PID/lock file. This would be weird when using a pre-made instance but it could be a sign that titan didn't shutdown properly when the instance's state was last saved. It should be possible to test this theory. I haven't used amazon's services in a very long time but perhaps try to reboot titan to get the graph back up like you demonstrated earlier. Then shut Titan down safely and save your image. Cross your fingers when you boot the image back up :)
I'll try tomorrow. In the meantime I've opened an issue with the AWS project. Will report back. Thanks so much for all your help.
I'm going to close this since it isn't an issue with gremlin-javascript
. For anyone interested, follow the aforementioned AWS ticket.
I'm not sure if this is related to
gremlin-javascript
, but this issue started right after I updated to the latest version.g
andgraph
are no longer available to my calls by default. I'm still able to connect, but need to fully instantiate these objects on each call now:I found issue #21, but so far haven't been able to fix the problem. The contents of
gremlin-server.yaml
are attached, but I've made no recent updates to it.I will note that when operating on the console (
bin/gremlin.sh
), I've always needed to manually instantiate these objects. However up until recentlygremlin-javascript
always provided a client withg
andgraph
available. I'm not sure what changed to cause this.Gremlin version:
Output when starting
gremlin.sh
:gremlin-server.yaml: