search

NVIDIA / spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs
https://nvidia.github.io/spark-rapids
Apache License 2.0
795 stars 232 forks source link

how to enter SparkSession function in SparkQueryCompareTestSuite.scala when I do unit test #977

Closed chenrui17 closed 3 years ago

chenrui17 commented 3 years ago

I want to do unit test of skewJoinTest, and I debug the unit test code, when it go to "withSparkSession" from withGpuSparkSession ,it exit with error code 1 image image

like this :
esting started at 9:30 下午 ... /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/bin/java -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:57325,suspend=y,server=n -javaagent:/Users/chenrui17/Library/Caches/IdeaIC2019.3/groovyHotSwap/gragent.jar -javaagent:/Users/chenrui17/Library/Caches/IdeaIC2019.3/captureAgent/debugger-agent.jar -Dfile.encoding=UTF-8 -classpath "/Users/chenrui17/Library/Application Support/IdeaIC2019.3/Scala/lib/runners.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/ext/jaccess.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/lib/packager.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/lib/tools.jar:/Users/chenrui17/Documents/2020-10-09/baidu/inf-cpt/spark-rapids/tests/target/test-classes:/Users/chenrui17/Documents/2020-10-09/baidu/inf-cpt/spark-rapids/tests/target/classes:/Users/chenrui17/.m2/repository/org/slf4j/jul-to-slf4j/1.7.30/jul-to-slf4j-1.7.30.jar:/Users/chenrui17/.m2/repository/org/slf4j/slf4j-api/1.7.30/slf4j-api-1.7.30.jar:/Users/chenrui17/.m2/repository/org/slf4j/jcl-over-slf4j/1.7.30/jcl-over-slf4j-1.7.30.jar:/Users/chenrui17/.m2/repository/org/scala-lang/scala-library/2.12.8/scala-library-2.12.8.jar:/Users/chenrui17/.m2/repository/org/apache/spark/spark-sql_2.12/3.0.1/spark-sql_2.12-3.0.1.jar:/Users/chenrui17/.m2/repository/com/univocity/univocity-parsers/2.9.0/univocity-parsers-2.9.0.jar:/Users/chenrui17/.m2/repository/org/apache/spark/spark-sketch_2.12/3.0.1/spark-sketch_2.12-3.0.1.jar:/Users/chenrui17/.m2/repository/org/apache/spark/spark-core_2.12/3.0.1/spark-core_2.12-3.0.1.jar:/Users/chenrui17/.m2/repository/com/thoughtworks/paranamer/paranamer/2.8/paranamer-2.8.jar:/Users/chenrui17/.m2/repository/org/apache/avro/avro/1.8.2/avro-1.8.2.jar:/Users/chenrui17/.m2/repository/org/apache/commons/commons-compress/1.8.1/commons-compress-1.8.1.jar:/Users/chenrui17/.m2/repository/org/tukaani/xz/1.5/xz-1.5.jar:/Users/chenrui17/.m2/repository/org/apache/avro/avro-mapred/1.8.2/avro-mapred-1.8.2-hadoop2.jar:/Users/chenrui17/.m2/repository/org/apache/avro/avro-ipc/1.8.2/avro-ipc-1.8.2.jar:/Users/chenrui17/.m2/repository/com/twitter/chill_2.12/0.9.5/chill_2.12-0.9.5.jar:/Users/chenrui17/.m2/repository/com/esotericsoftware/kryo-shaded/4.0.2/kryo-shaded-4.0.2.jar:/Users/chenrui17/.m2/repository/com/esotericsoftware/minlog/1.3.0/minlog-1.3.0.jar:/Users/chenrui17/.m2/repository/com/twitter/chill-java/0.9.5/chill-java-0.9.5.jar:/Users/chenrui17/.m2/repository/org/apache/hadoop/hadoop-client/2.7.4/hadoop-client-2.7.4.jar:/Users/chenrui17/.m2/repository/org/apache/hadoop/hadoop-common/2.7.4/hadoop-common-2.7.4.jar:/Users/chenrui17/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/Users/chenrui17/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/Users/chenrui17/.m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/Users/chenrui17/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar:/Users/chenrui17/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/Users/chenrui17/.m2/repository/org/mortbay/jetty/jetty-sslengine/6.1.26/jetty-sslengine-6.1.26.jar:/Users/chenrui17/.m2/repository/javax/servlet/jsp/jsp-api/2.1/jsp-api-2.1.jar:/Users/chenrui17/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/Users/chenrui17/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/Users/chenrui17/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/Users/chenrui17/.m2/repository/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar:/Users/chenrui17/.m2/repository/org/apache/hadoop/hadoop-auth/2.7.4/hadoop-auth-2.7.4.jar:/Users/chenrui17/.m2/repository/org/apache/httpcomponents/httpclient/4.2.5/httpclient-4.2.5.jar:/Users/chenrui17/.m2/repository/org/apache/httpcomponents/httpcore/4.2.4/httpcore-4.2.4.jar:/Users/chenrui17/.m2/repository/org/apache/directory/server/apacheds-kerberos-codec/2.0.0-M15/apacheds-kerberos-codec-2.0.0-M15.jar:/Users/chenrui17/.m2/repository/org/apache/directory/server/apacheds-i18n/2.0.0-M15/apacheds-i18n-2.0.0-M15.jar:/Users/chenrui17/.m2/repository/org/apache/directory/api/api-asn1-api/1.0.0-M20/api-asn1-api-1.0.0-M20.jar:/Users/chenrui17/.m2/repository/org/apache/directory/api/api-util/1.0.0-M20/api-util-1.0.0-M20.jar:/Users/chenrui17/.m2/repository/org/apache/curator/curator-client/2.7.1/curator-client-2.7.1.jar:/Users/chenrui17/.m2/repository/org/apache/htrace/htrace-core/3.1.0-incubating/htrace-core-3.1.0-incubating.jar:/Users/chenrui17/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4.jar:/Users/chenrui17/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar:/Users/chenrui17/.m2/repository/xerces/xercesImpl/2.9.1/xercesImpl-2.9.1.jar:/Users/chenrui17/.m2/repository/xml-apis/xml-apis/1.3.04/xml-apis-1.3.04.jar:/Users/chenrui17/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-app/2.7.4/hadoop-mapreduce-client-app-2.7.4.jar:/Users/chenrui17/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-common/2.7.4/hadoop-mapreduce-client-common-2.7.4.jar:/Users/chenrui17/.m2/repository/org/apache/hadoop/hadoop-yarn-client/2.7.4/hadoop-yarn-client-2.7.4.jar:/Users/chenrui17/.m2/repository/org/apache/hadoop/hadoop-yarn-server-common/2.7.4/hadoop-yarn-server-common-2.7.4.jar:/Users/chenrui17/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-shuffle/2.7.4/hadoop-mapreduce-client-shuffle-2.7.4.jar:/Users/chenrui17/.m2/repository/org/apache/hadoop/hadoop-yarn-api/2.7.4/hadoop-yarn-api-2.7.4.jar:/Users/chenrui17/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.4/hadoop-mapreduce-client-core-2.7.4.jar:/Users/chenrui17/.m2/repository/org/apache/hadoop/hadoop-yarn-common/2.7.4/hadoop-yarn-common-2.7.4.jar:/Users/chenrui17/.m2/repository/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar:/Users/chenrui17/.m2/repository/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar:/Users/chenrui17/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.9.13/jackson-jaxrs-1.9.13.jar:/Users/chenrui17/.m2/repository/org/codehaus/jackson/jackson-xc/1.9.13/jackson-xc-1.9.13.jar:/Users/chenrui17/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.7.4/hadoop-mapreduce-client-jobclient-2.7.4.jar:/Users/chenrui17/.m2/repository/org/apache/hadoop/hadoop-annotations/2.7.4/hadoop-annotations-2.7.4.jar:/Users/chenrui17/.m2/repository/org/apache/spark/spark-launcher_2.12/3.0.1/spark-launcher_2.12-3.0.1.jar:/Users/chenrui17/.m2/repository/org/apache/spark/spark-kvstore_2.12/3.0.1/spark-kvstore_2.12-3.0.1.jar:/Users/chenrui17/.m2/repository/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.jar:/Users/chenrui17/.m2/repository/org/apache/spark/spark-network-common_2.12/3.0.1/spark-network-common_2.12-3.0.1.jar:/Users/chenrui17/.m2/repository/org/apache/spark/spark-network-shuffle_2.12/3.0.1/spark-network-shuffle_2.12-3.0.1.jar:/Users/chenrui17/.m2/repository/org/apache/spark/spark-unsafe_2.12/3.0.1/spark-unsafe_2.12-3.0.1.jar:/Users/chenrui17/.m2/repository/javax/activation/activation/1.1.1/activation-1.1.1.jar:/Users/chenrui17/.m2/repository/org/apache/curator/curator-recipes/2.7.1/curator-recipes-2.7.1.jar:/Users/chenrui17/.m2/repository/org/apache/curator/curator-framework/2.7.1/curator-framework-2.7.1.jar:/Users/chenrui17/.m2/repository/com/google/guava/guava/16.0.1/guava-16.0.1.jar:/Users/chenrui17/.m2/repository/org/apache/zookeeper/zookeeper/3.4.14/zookeeper-3.4.14.jar:/Users/chenrui17/.m2/repository/org/apache/yetus/audience-annotations/0.5.0/audience-annotations-0.5.0.jar:/Users/chenrui17/.m2/repository/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar:/Users/chenrui17/.m2/repository/org/apache/commons/commons-lang3/3.9/commons-lang3-3.9.jar:/Users/chenrui17/.m2/repository/org/apache/commons/commons-math3/3.4.1/commons-math3-3.4.1.jar:/Users/chenrui17/.m2/repository/org/apache/commons/commons-text/1.6/commons-text-1.6.jar:/Users/chenrui17/.m2/repository/com/google/code/findbugs/jsr305/3.0.0/jsr305-3.0.0.jar:/Users/chenrui17/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/Users/chenrui17/.m2/repository/org/slf4j/slf4j-log4j12/1.7.30/slf4j-log4j12-1.7.30.jar:/Users/chenrui17/.m2/repository/com/ning/compress-lzf/1.0.3/compress-lzf-1.0.3.jar:/Users/chenrui17/.m2/repository/org/xerial/snappy/snappy-java/1.1.7.5/snappy-java-1.1.7.5.jar:/Users/chenrui17/.m2/repository/org/lz4/lz4-java/1.7.1/lz4-java-1.7.1.jar:/Users/chenrui17/.m2/repository/com/github/luben/zstd-jni/1.4.4-3/zstd-jni-1.4.4-3.jar:/Users/chenrui17/.m2/repository/org/roaringbitmap/RoaringBitmap/0.7.45/RoaringBitmap-0.7.45.jar:/Users/chenrui17/.m2/repository/org/roaringbitmap/shims/0.7.45/shims-0.7.45.jar:/Users/chenrui17/.m2/repository/commons-net/commons-net/3.1/commons-net-3.1.jar:/Users/chenrui17/.m2/repository/org/json4s/json4s-jackson_2.12/3.6.6/json4s-jackson_2.12-3.6.6.jar:/Users/chenrui17/.m2/repository/org/json4s/json4s-core_2.12/3.6.6/json4s-core_2.12-3.6.6.jar:/Users/chenrui17/.m2/repository/org/json4s/json4s-ast_2.12/3.6.6/json4s-ast_2.12-3.6.6.jar:/Users/chenrui17/.m2/repository/org/json4s/json4s-scalap_2.12/3.6.6/json4s-scalap_2.12-3.6.6.jar:/Users/chenrui17/.m2/repository/org/glassfish/jersey/core/jersey-client/2.30/jersey-client-2.30.jar:/Users/chenrui17/.m2/repository/jakarta/ws/rs/jakarta.ws.rs-api/2.1.6/jakarta.ws.rs-api-2.1.6.jar:/Users/chenrui17/.m2/repository/org/glassfish/hk2/external/jakarta.inject/2.6.1/jakarta.inject-2.6.1.jar:/Users/chenrui17/.m2/repository/org/glassfish/jersey/core/jersey-common/2.30/jersey-common-2.30.jar:/Users/chenrui17/.m2/repository/jakarta/annotation/jakarta.annotation-api/1.3.5/jakarta.annotation-api-1.3.5.jar:/Users/chenrui17/.m2/repository/org/glassfish/hk2/osgi-resource-locator/1.0.3/osgi-resource-locator-1.0.3.jar:/Users/chenrui17/.m2/repository/org/glassfish/jersey/core/jersey-server/2.30/jersey-server-2.30.jar:/Users/chenrui17/.m2/repository/org/glassfish/jersey/media/jersey-media-jaxb/2.30/jersey-media-jaxb-2.30.jar:/Users/chenrui17/.m2/repository/jakarta/validation/jakarta.validation-api/2.0.2/jakarta.validation-api-2.0.2.jar:/Users/chenrui17/.m2/repository/org/glassfish/jersey/containers/jersey-container-servlet/2.30/jersey-container-servlet-2.30.jar:/Users/chenrui17/.m2/repository/org/glassfish/jersey/containers/jersey-container-servlet-core/2.30/jersey-container-servlet-core-2.30.jar:/Users/chenrui17/.m2/repository/org/glassfish/jersey/inject/jersey-hk2/2.30/jersey-hk2-2.30.jar:/Users/chenrui17/.m2/repository/org/glassfish/hk2/hk2-locator/2.6.1/hk2-locator-2.6.1.jar:/Users/chenrui17/.m2/repository/org/glassfish/hk2/external/aopalliance-repackaged/2.6.1/aopalliance-repackaged-2.6.1.jar:/Users/chenrui17/.m2/repository/org/glassfish/hk2/hk2-api/2.6.1/hk2-api-2.6.1.jar:/Users/chenrui17/.m2/repository/org/glassfish/hk2/hk2-utils/2.6.1/hk2-utils-2.6.1.jar:/Users/chenrui17/.m2/repository/org/javassist/javassist/3.25.0-GA/javassist-3.25.0-GA.jar:/Users/chenrui17/.m2/repository/io/netty/netty-all/4.1.47.Final/netty-all-4.1.47.Final.jar:/Users/chenrui17/.m2/repository/com/clearspring/analytics/stream/2.9.6/stream-2.9.6.jar:/Users/chenrui17/.m2/repository/io/dropwizard/metrics/metrics-core/4.1.1/metrics-core-4.1.1.jar:/Users/chenrui17/.m2/repository/io/dropwizard/metrics/metrics-jvm/4.1.1/metrics-jvm-4.1.1.jar:/Users/chenrui17/.m2/repository/io/dropwizard/metrics/metrics-json/4.1.1/metrics-json-4.1.1.jar:/Users/chenrui17/.m2/repository/io/dropwizard/metrics/metrics-graphite/4.1.1/metrics-graphite-4.1.1.jar:/Users/chenrui17/.m2/repository/io/dropwizard/metrics/metrics-jmx/4.1.1/metrics-jmx-4.1.1.jar:/Users/chenrui17/.m2/repository/com/fasterxml/jackson/module/jackson-module-scala_2.12/2.10.0/jackson-module-scala_2.12-2.10.0.jar:/Users/chenrui17/.m2/repository/com/fasterxml/jackson/module/jackson-module-paranamer/2.10.0/jackson-module-paranamer-2.10.0.jar:/Users/chenrui17/.m2/repository/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar:/Users/chenrui17/.m2/repository/oro/oro/2.0.8/oro-2.0.8.jar:/Users/chenrui17/.m2/repository/net/razorvine/pyrolite/4.30/pyrolite-4.30.jar:/Users/chenrui17/.m2/repository/net/sf/py4j/py4j/0.10.9/py4j-0.10.9.jar:/Users/chenrui17/.m2/repository/org/apache/commons/commons-crypto/1.0.0/commons-crypto-1.0.0.jar:/Users/chenrui17/.m2/repository/org/apache/spark/spark-catalyst_2.12/3.0.1/spark-catalyst_2.12-3.0.1.jar:/Users/chenrui17/.m2/repository/org/scala-lang/modules/scala-parser-combinators_2.12/1.1.2/scala-parser-combinators_2.12-1.1.2.jar:/Users/chenrui17/.m2/repository/org/codehaus/janino/janino/3.0.16/janino-3.0.16.jar:/Users/chenrui17/.m2/repository/org/codehaus/janino/commons-compiler/3.0.16/commons-compiler-3.0.16.jar:/Users/chenrui17/.m2/repository/org/antlr/antlr4-runtime/4.7.1/antlr4-runtime-4.7.1.jar:/Users/chenrui17/.m2/repository/commons-codec/commons-codec/1.10/commons-codec-1.10.jar:/Users/chenrui17/.m2/repository/org/apache/arrow/arrow-vector/0.15.1/arrow-vector-0.15.1.jar:/Users/chenrui17/.m2/repository/org/apache/arrow/arrow-format/0.15.1/arrow-format-0.15.1.jar:/Users/chenrui17/.m2/repository/org/apache/arrow/arrow-memory/0.15.1/arrow-memory-0.15.1.jar:/Users/chenrui17/.m2/repository/org/apache/spark/spark-tags_2.12/3.0.1/spark-tags_2.12-3.0.1.jar:/Users/chenrui17/.m2/repository/org/apache/orc/orc-core/1.5.8/orc-core-1.5.8.jar:/Users/chenrui17/.m2/repository/org/apache/orc/orc-shims/1.5.8/orc-shims-1.5.8.jar:/Users/chenrui17/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar:/Users/chenrui17/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/chenrui17/.m2/repository/io/airlift/aircompressor/0.10/aircompressor-0.10.jar:/Users/chenrui17/.m2/repository/org/apache/orc/orc-mapreduce/1.5.8/orc-mapreduce-1.5.8.jar:/Users/chenrui17/.m2/repository/org/apache/hive/hive-storage-api/2.6.0/hive-storage-api-2.6.0.jar:/Users/chenrui17/.m2/repository/org/apache/parquet/parquet-column/1.10.1/parquet-column-1.10.1.jar:/Users/chenrui17/.m2/repository/org/apache/parquet/parquet-common/1.10.1/parquet-common-1.10.1.jar:/Users/chenrui17/.m2/repository/org/apache/parquet/parquet-encoding/1.10.1/parquet-encoding-1.10.1.jar:/Users/chenrui17/.m2/repository/org/apache/parquet/parquet-hadoop/1.10.1/parquet-hadoop-1.10.1.jar:/Users/chenrui17/.m2/repository/org/apache/parquet/parquet-format/2.4.0/parquet-format-2.4.0.jar:/Users/chenrui17/.m2/repository/org/apache/parquet/parquet-jackson/1.10.1/parquet-jackson-1.10.1.jar:/Users/chenrui17/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar:/Users/chenrui17/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar:/Users/chenrui17/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.10.0/jackson-databind-2.10.0.jar:/Users/chenrui17/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.10.0/jackson-annotations-2.10.0.jar:/Users/chenrui17/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.10.0/jackson-core-2.10.0.jar:/Users/chenrui17/.m2/repository/org/apache/xbean/xbean-asm7-shaded/4.15/xbean-asm7-shaded-4.15.jar:/Users/chenrui17/.m2/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:/Users/chenrui17/.m2/repository/org/scalatest/scalatest_2.12/3.0.5/scalatest_2.12-3.0.5.jar:/Users/chenrui17/.m2/repository/org/scalactic/scalactic_2.12/3.0.5/scalactic_2.12-3.0.5.jar:/Users/chenrui17/.m2/repository/org/scala-lang/scala-reflect/2.12.4/scala-reflect-2.12.4.jar:/Users/chenrui17/.m2/repository/org/scala-lang/modules/scala-xml_2.12/1.0.6/scala-xml_2.12-1.0.6.jar:/Users/chenrui17/.m2/repository/ai/rapids/cudf/0.15/cudf-0.15-cuda10-2.jar:/Users/chenrui17/Documents/2020-10-09/baidu/inf-cpt/spark-rapids/dist/target/classes:/Users/chenrui17/Documents/2020-10-09/baidu/inf-cpt/spark-rapids/sql-plugin/target/classes:/Users/chenrui17/.m2/repository/com/google/flatbuffers/flatbuffers-java/1.11.0/flatbuffers-java-1.11.0.jar:/Users/chenrui17/Documents/2020-10-09/baidu/inf-cpt/spark-rapids/shuffle-plugin/target/classes:/Users/chenrui17/.m2/repository/org/openucx/jucx/1.8.0/jucx-1.8.0.jar:/Users/chenrui17/Documents/2020-10-09/baidu/inf-cpt/spark-rapids/udf-compiler/target/classes:/Users/chenrui17/Documents/2020-10-09/baidu/inf-cpt/spark-rapids/shims/aggregator/target/classes:/Users/chenrui17/Documents/2020-10-09/baidu/inf-cpt/spark-rapids/shims/spark300/target/classes:/Users/chenrui17/Documents/2020-10-09/baidu/inf-cpt/spark-rapids/shims/spark301/target/classes:/Users/chenrui17/Documents/2020-10-09/baidu/inf-cpt/spark-rapids/shims/spark310/target/classes:/Users/chenrui17/Documents/2020-10-09/baidu/inf-cpt/spark-rapids/shims/spark302/target/classes:/Users/chenrui17/.m2/repository/org/mockito/mockito-core/2.28.2/mockito-core-2.28.2.jar:/Users/chenrui17/.m2/repository/net/bytebuddy/byte-buddy/1.9.10/byte-buddy-1.9.10.jar:/Users/chenrui17/.m2/repository/net/bytebuddy/byte-buddy-agent/1.9.10/byte-buddy-agent-1.9.10.jar:/Users/chenrui17/.m2/repository/org/objenesis/objenesis/2.6/objenesis-2.6.jar:/Users/chenrui17/.m2/repository/org/junit/jupiter/junit-jupiter-api/5.4.2/junit-jupiter-api-5.4.2.jar:/Users/chenrui17/.m2/repository/org/apiguardian/apiguardian-api/1.0.0/apiguardian-api-1.0.0.jar:/Users/chenrui17/.m2/repository/org/opentest4j/opentest4j/1.1.1/opentest4j-1.1.1.jar:/Users/chenrui17/.m2/repository/org/junit/platform/junit-platform-commons/1.4.2/junit-platform-commons-1.4.2.jar:/Applications/IntelliJ IDEA CE.app/Contents/lib/idea_rt.jar" org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner -s com.nvidia.spark.rapids.AdaptiveQueryExecSuite -testName "skewed inner join optimization" -C org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestReporter -showProgressMessages true Connected to the target VM, address: '127.0.0.1:57325', transport: 'socket'

Disconnected from the target VM, address: '127.0.0.1:57325', transport: 'socket'

Process finished with exit code 1

chenrui17 commented 3 years ago

after run this line in IDEA， it runs a long time , and then exit with code 1 , please help me how to debug this and run unit test of AdaptiveQueryExecSuite image my rapids version is branch-0.2 , and commit id is 9582bc87859a7d45985eda1c52281fc16fd88d88

jlowe commented 3 years ago

I was able to step into withSparkSession using a debugger successfully. However that's not how I would normally debug it. If you're interested in debugging the main part of skewJoinTest then I'd place the breakpoint with the lambda function you're actually interested in debugging rather than needing to step your way manually there through the general withSparkSession code.

As to why it's exiting during your debug session, you should check the logs for indication as to what went wrong. Look at recent files within tests/target/surefire-reports/ such as tests/target/surefire-reports/scala-test-detailed-output.log to see if an error was logged. For example, if you're seeing something like Could not allocate native memory: std::bad_alloc: CUDA error in the logs then likely another process is running on the GPU at the time and taking up GPU memory (e.g.: a hung process from an earlier debugging session). In that case, the nvidia-smi command can be used to find processes that are running on the GPU.

chenrui17 commented 3 years ago

I was able to step into withSparkSession using a debugger successfully. However that's not how I would normally debug it. If you're interested in debugging the main part of skewJoinTest then I'd place the breakpoint with the lambda function you're actually interested in debugging rather than needing to step your way manually there through the general withSparkSession code.

As to why it's exiting during your debug session, you should check the logs for indication as to what went wrong. Look at recent files within tests/target/surefire-reports/ such as tests/target/surefire-reports/scala-test-detailed-output.log to see if an error was logged. For example, if you're seeing something like Could not allocate native memory: std::bad_alloc: CUDA error in the logs then likely another process is running on the GPU at the time and taking up GPU memory (e.g.: a hung process from an earlier debugging session). In that case, the nvidia-smi command can be used to find processes that are running on the GPU.

thanks a lot ! I check the tests/target/surefire-reports/scala-test-detailed-output.log and i found that the error message is ,like this, ERROR NativeDepsLoader: Could not load cudf jni library... image and also ,now i am using MacBook-pro IDEA software to running suite test case ,Without GPU, can it be executed normally？

jlowe commented 3 years ago

Without GPU, can it be executed normally？

No, almost all of the RAPIDS Accelerator unit tests require a GPU because they are testing a plugin that executes code on the GPU. So now we know why it failed, as a unit test requiring a CUDA GPU was run on a platform without that environment.

I too have been running Intellij on a MacBook to step through the unit tests, but the difference is I'm using Intellij as a remote debugger, not running the unit tests locally. On a Linux machine with a CUDA GPU I executed this command:

mvn test -DwildcardSuites=com.nvidia.spark.rapids.AdaptiveQueryExecSuite -DdebugForkedProcess -Pspark301tests

And then once it said it was listening for a debugger attach I would launch a remote debug session in Intellij with my desired breakpoints set. Note that -DdebugForkedProcess causes a couple of spurious debugger sessions where no test are run because it discovers no tests in that project.

Closing this issue since the root cause is trying to debug a unit test where it cannot run normally. Please reopen if you encounter similar issues when trying to debug on a system where the unit test can normally run.

chenrui17 commented 3 years ago

@jlow thank you for your help , now I can debug rapids code with remote mode , and i met a new question : i modified some spark-sql source code and rapids code , i want to debug rapids code with my modified spark, what can i do for it?

i think now debug methods use the spark jar is in maven repository , it's in here : ~/.m2/repository/org/apache/spark/spark-sql_2.12/3.0.1/spark-sql_2.12-3.0.1.jar , so if i move my modify spark-sql_2.12-3.0.1.jar to here , then I run unit test command: mvn test -DwildcardSuites=com.nvidia.spark.rapids.AdaptiveQueryExecSuite -DdebugForkedProcess -Pspark301tests -DdebuggerPort=8888

can it work with my expect ？

jlowe commented 3 years ago

Yes, if you run the tests with -Pspark301tests and override the jars that will be picked up for Spark 3.0.1 in the local Maven repository then that should work. I recommend updating both the remote machine and your local laptop so the debugger is in sync with your remote machine in case you step into the modified Spark code.

A possibly simpler way to do this is to update the pom to add your own test profile for your custom version of Spark if that Spark is installed or published somewhere, e.g.: -Psparkcustomtests that changes the version of Spark that will be listed as the dependency during the tests run.