search

datastax / cstar_perf

Apache Cassandra performance testing platform
Apache License 2.0
72 stars 34 forks source link

CSTAR-605: Alter spark-cassandra-stress execution to run on client node #226

Closed csplinter closed 8 years ago

csplinter commented 8 years ago

Covered in this PR for CSTAR-605

rocco408 commented 8 years ago

+1

csplinter commented 8 years ago

I have found an issue with DSE 4.7.9 and the current setup. When we try to run spark-cassandra-stress from the client node we get this trace which comes from calling dsetool to get the spark master. Similarly, the spark master that we pass to spark-cassandra-stress is not being properly set when we run the spark-submit command

java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is: 
    java.net.ConnectException: Connection refused]
    at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
    at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
    at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:192)
    at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:145)
    at com.datastax.bdp.tools.DseTool.run(DseTool.java:102)
    at com.datastax.bdp.tools.DseTool.run(DseTool.java:51)
    at com.datastax.bdp.tools.DseTool.main(DseTool.java:174)
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is: 
    java.net.ConnectException: Connection refused]
    at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:122)
    at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205)
    at javax.naming.InitialContext.lookup(InitialContext.java:417)
    at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955)
    at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922)
    at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287)
    ... 6 more
Caused by: java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is: 
    java.net.ConnectException: Connection refused
    at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
    at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
    at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
    at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342)
    at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
    at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
    ... 11 more
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at java.net.Socket.connect(Socket.java:538)
    at java.net.Socket.<init>(Socket.java:434)
    at java.net.Socket.<init>(Socket.java:211)
    at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
    at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
    at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
    ... 16 more

It works fine for 4.8.x and 5.0.0. Will dig into this when I get back on Wednesday (7/6)

nastra commented 8 years ago

+1

csplinter commented 8 years ago

4.7.x Problem:

In spark-cassandra-stress's run.sh we call DSE's spark submit.

Issue 1: Error from calling dsetool on the node running spark-cassandra-stress When we do this, we try to get the spark master by calling set_spark_master_env which calls dsetool on the node running spark-cassandra-stress. This produces the following trace as DSE is not started on this node

java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is: 
    java.net.ConnectException: Connection refused]
    at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
    at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
    at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:192)
    at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:145)
    at com.datastax.bdp.tools.DseTool.run(DseTool.java:102)
    at com.datastax.bdp.tools.DseTool.run(DseTool.java:51)
    at com.datastax.bdp.tools.DseTool.main(DseTool.java:174)
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is: 
    java.net.ConnectException: Connection refused]
    at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:122)
    at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205)
    at javax.naming.InitialContext.lookup(InitialContext.java:417)
    at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955)
    at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922)
    at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287)
    ... 6 more
Caused by: java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is: 
    java.net.ConnectException: Connection refused
    at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
    at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
    at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
    at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342)
    at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
    at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
    ... 11 more
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at java.net.Socket.connect(Socket.java:538)
    at java.net.Socket.<init>(Socket.java:434)
    at java.net.Socket.<init>(Socket.java:211)
    at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
    at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
    at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
    ... 16 more

Issue 2: Spark master is not set in call to spark-submit Because we can not get the master from the above call to dsetool, the MASTER environment variable remains unset. Even though there is an option to pass in our master to spark-cassandra-stress via the second argument, this value is not properly set in our call to spark-submit because it is looking for the --master flag.

Issue 3: 0 rows written to . (this occurs when spark-cassandra-stress is run on either client or server node) If both of the above problems are worked around, the rows are not being properly written to the C* table. The writeperfrow task should be using saveToCassandra but something else is going wrong here as I see the following in the executor logs and no rows when I check in cqlsh

INFO  2016-07-06 19:44:41 org.apache.spark.executor.Executor: Finished task 398.0 in stage 0.0 (TID 398). 1577 bytes result sent to driver
INFO  2016-07-06 19:44:41 org.apache.spark.executor.Executor: Running task 397.0 in stage 0.0 (TID 397)
INFO  2016-07-06 19:44:41 org.apache.spark.executor.Executor: Running task 399.0 in stage 0.0 (TID 399)
INFO  2016-07-06 19:44:41 com.datastax.spark.connector.writer.TableWriter: Wrote 0 rows to ks.tab in 0.001 s.

@rocco408 @nastra the easy solution here would just be to run spark-cassandra-stress on the server cluster if the DSE version is < 4.8.x but I do not feel great about this. Another solution that I am more fond of is to pass a --master argument to spark-cassandra-stress if DSE version is < 4.8.x and just live with the trace that is a result of calling dsetool sparkmaster from the dse script on the spark-cassandra-stress node. The biggest issue that I see here is the fact that 0 rows are actually being saved to C* and I saw this also when manually running spark-cassandra-stress (Connector Version = 1.2.5) on a 4.7.9 cluster. Below is my debugging output

cassandra.client.transport.factory=com.datastax.bdp.transport.client.TDseClientTransportFactory
cassandra.config.loader=com.datastax.bdp.config.DseConfigurationLoader
hadoop.security.authentication=simple
hadoop.security.authorization=false
hadoop.security.group.mapping=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
hadoop.security.token.service.use_ip=true
hadoop.security.uid.cache.secs=14400
spark.app.id=app-20160707150700-0006
spark.app.name=SparkStress_writeperfrow
spark.cassandra.auth.conf.factory=com.datastax.bdp.spark.DseAuthConfFactory
spark.cassandra.connection.factory=com.datastax.bdp.spark.DseCassandraConnectionFactory
spark.cassandra.connection.host=testcluster-01
spark.cassandra.connection.keep_alive_ms=25000
spark.driver.extraJavaOptions= -DcassandraUserNameProp= -DcassandraPasswordProp= -XX:MaxPermSize=256M 
spark.driver.host=testcluster-00
spark.driver.memory=512M
spark.driver.port=40722
spark.executor.extraJavaOptions=-XX:MaxPermSize=256M -DcassandraUserNameProp= -DcassandraPasswordProp= 
spark.executor.id=driver
spark.fileserver.uri=http://172.17.0.2:41614
spark.hadoop.cassandra.client.transport.factory=com.datastax.bdp.transport.client.TDseClientTransportFactory
spark.hadoop.cassandra.config.loader=com.datastax.bdp.config.DseConfigurationLoader
spark.hadoop.hadoop.security.authentication=simple
spark.hadoop.hadoop.security.authorization=false
spark.hadoop.hadoop.security.group.mapping=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
spark.hadoop.hadoop.security.token.service.use_ip=true
spark.hadoop.hadoop.security.uid.cache.secs=14400
spark.jars=file:/home/cstar/fab/spark-cassandra-stress/build/libs/SparkCassandraStress-1.0.jar
spark.kryoserializer.buffer.mb=10
spark.master=spark://172.17.0.3:7077
spark.metrics.conf=dse-spark-metrics.properties
spark.ssl.enabled=false
spark.ssl.enabledAlgorithms=TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA
spark.ssl.keyPassword=cassandra
spark.ssl.keyStore=.keystore
spark.ssl.keyStorePassword=cassandra
spark.ssl.protocol=TLS
spark.ssl.trustStore=.truststore
spark.ssl.trustStorePassword=cassandra
spark.ssl.useNodeLocalConf=true
spark.tachyonStore.folderName=spark-b41d0857-b47a-40ed-b062-b25bea562601
rocco408 commented 8 years ago

Just summing up private chat conversation so we keep some common docs on this.

Issues 1 and 2 were not reproducible on my setup. The issue 'may' be that CONNECTOR_VERSION or SPARK_VERSION got tinkered with at some point causing an incompatible build, just a guess though. My steps:

  1. launch two independent clusters, using openstack under the same vpn
  2. install 4.7.9 on both, start dse on both
  3. on remote machine (driver node) clone spark-stress, build as usual against dse
  4. run the test pointing to the master node. Example: ./run.sh dse -o 1000000 writeperfrow spark://10.200.180.160:7077 --conf spark.cassandra.connection.host=ip-10-200-180-160.datastax.lan

The thing with 0 rows landing in DSE has been seen before and is a minor race condition in spark-stress itself. If we insert too few rows spark-stress will kill the app before completion. I recommend inserting at least 1M rows until we fix this.

csplinter commented 8 years ago

@rocco408 and I figured it out, spark-submit can not find the master on the client when DSE is not started on the client and therefore throws the errors. If DSE is started on the client, then it uses the local master because of the check in set_spark_master_env. To fix this, we are going to add a --master <passed_spark_master> if it is passed to the spark-submit call in spark-cassandra-stress's run.sh if a master is passed. We could alternatively set SPARK_MASTER on the client node, but this requires an extra step by the user. I will retest this once the above change is made.

rocco408 commented 8 years ago

I thought I had figured it out but in the end I noticed basically the same behavior with or without setting --master on the client when calling spark-submit. Basically with or without using --master with spark-submit we see that the server master is used (UI accessible via the server IP), the worker however is running on the client where spark-stress was kicked off, but the executors are indeed running on the server. Since we're running in client mode we're going to inherit some additional latency between the driver (running on the client) and the executors (running on the server nodes). I'm not sure if having the worker also running on the client is an issue, although I've only tested this on single-node deployments so we may hit something else when expanding to multiple nodes on the server. Another option is to simply run spark-stress in "cluster" mode (spark-submit --deploy-mode cluster), this will run everything on the server including the driver. Caveats being we'll need to scp the the spark-stress jar to all the executors (or to cfs on the server) and some links on the UI might be broken since some may still link back to the client for the worker pages, but the data can still be accessed if we point the browser to the correct worker IP on the server. @RussellSpitzer what do you think is reasonable approach? I think Chris is simply trying to mimic, as closely as possible, the most common architecture that is used in the field when running these tests.

RussellSpitzer commented 8 years ago

I wouldn't really worry about latency between the Spark Stress driver and the executors. That should be minimally important (really only used for sending task metadata). Either running in client or cluster mode is reasonable, the only important thing is colocating the spark executors

csplinter commented 8 years ago

@nastra @rocco408 all above issues are resolved by this cstar commit and this spark-cassandra-stress commit. I retested with these changes and everything works as expected. Going to leave this open for another day in case you guys want to review one more time (I will squash the commits before merging)

rocco408 commented 8 years ago

Looks good to me 👍 :shipit:

nastra commented 8 years ago

+1