CreateReadCountPanelOfNormals cannot run on single machine

galaxy001 commented 5 years ago

Bug Report

Affected tool(s) or class(es)

CreateReadCountPanelOfNormals

Affected version(s)

[ ] Latest public release version [4.1.0.0]

Description

When you run it on a single machine, it trys to use hadoop and failed.

$ java -jar ../gatk-package-4.1.0.0-local.jar CreateReadCountPanelOfNormals --input in.counts.hdf5 --output out.pon.hdf5
12:33:52.103 WARN  SparkContextFactory - Environment variables HELLBENDER_TEST_PROJECT and HELLBENDER_JSON_SERVICE_ACCOUNT_KEY must be set or the GCS hadoop connector will not be configured properly
12:33:52.162 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/share/FGI2017B/pub/gatk-4.1.0.0/gatk-package-4.1.0.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
12:33:53.793 INFO  CreateReadCountPanelOfNormals - ------------------------------------------------------------
12:33:53.794 INFO  CreateReadCountPanelOfNormals - The Genome Analysis Toolkit (GATK) v4.1.0.0
12:33:53.794 INFO  CreateReadCountPanelOfNormals - For support and documentation go to https://software.broadinstitute.org/gatk/
12:33:53.797 INFO  CreateReadCountPanelOfNormals - Initializing engine
12:33:53.797 INFO  CreateReadCountPanelOfNormals - Done initializing engine
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/02/18 12:33:53 INFO SparkContext: Running Spark version 2.2.0
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/share/FGI2017B/pub/gatk-4.1.0.0/gatk-package-4.1.0.0-local.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
12:33:54.187 WARN  NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
12:33:54.263 INFO  CreateReadCountPanelOfNormals - Shutting down engine
[February 18, 2019 at 12:33:54 PM CST] org.broadinstitute.hellbender.tools.copynumber.CreateReadCountPanelOfNormals done. Elapsed time: 0.04 minutes.
Runtime.totalMemory()=2147483648
Exception in thread "main" java.lang.ExceptionInInitializerError
    at org.apache.spark.SparkConf.validateSettings(SparkConf.scala:546)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:373)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
    at org.broadinstitute.hellbender.engine.spark.SparkContextFactory.createSparkContext(SparkContextFactory.java:178)
    at org.broadinstitute.hellbender.engine.spark.SparkContextFactory.getSparkContext(SparkContextFactory.java:110)
    at org.broadinstitute.hellbender.engine.spark.SparkCommandLineProgram.doWork(SparkCommandLineProgram.java:28)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:138)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
    at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:162)
    at org.broadinstitute.hellbender.Main.mainEntry(Main.java:205)
    at org.broadinstitute.hellbender.Main.main(Main.java:291)
Caused by: java.net.UnknownHostException: cngb-nas-f17-1: cngb-nas-f17-1: Name or service not known
    at java.base/java.net.InetAddress.getLocalHost(InetAddress.java:1631)
    at org.apache.spark.util.Utils$.findLocalInetAddress(Utils.scala:891)
    at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$localIpAddress$lzycompute(Utils.scala:884)
    at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$localIpAddress(Utils.scala:884)
    at org.apache.spark.util.Utils$$anonfun$localHostName$1.apply(Utils.scala:941)
    at org.apache.spark.util.Utils$$anonfun$localHostName$1.apply(Utils.scala:941)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.util.Utils$.localHostName(Utils.scala:941)
    at org.apache.spark.internal.config.package$.<init>(package.scala:204)
    at org.apache.spark.internal.config.package$.<clinit>(package.scala)
    ... 12 more
Caused by: java.net.UnknownHostException: cngb-nas-f17-1: Name or service not known
    at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
    at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:924)
    at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1504)
    at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:843)
    at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1494)
    at java.base/java.net.InetAddress.getLocalHost(InetAddress.java:1626)
    ... 21 more

Steps to reproduce

On a Linux machine without Hadoop, run java -jar ../gatk-package-4.1.0.0-local.jar CreateReadCountPanelOfNormals --input in.counts.hdf5 --output out.pon.hdf5 locally.

Expected behavior

Produce out.pon.hdf5

Actual behavior

Exit with error.

galaxy001 commented 5 years ago

Well, after install hadoop 3.1.1 with Linuxbrew, I managed to run CreateReadCountPanelOfNormals.

Still, I think it is better not to copy input files to hadoop when running on single machine.

samuelklee commented 5 years ago

Glad you were able to resolve your issue. Not sure if this is specific to the CNV tool or if the exception caused by the Spark configuration is more general. Tagging engine team @droazen, but closing for now.

broadinstitute / gatk