search

sphereio / docker-predictionio

Docker container for PredictionIO-based machine learning services
49 stars 33 forks source link

Exception in thread "main" java.lang.NullPointerException at org.apache.hadoop.net.DNS.reverseDns(DNS.java:92) #10

Open djones opened 8 years ago

djones commented 8 years ago

Starting and stopping the PIO services, creating a new PIO app, and building an engine works.

But when I try to train an engine before deploying it, I get this error:

Exception in thread "main" java.lang.NullPointerException
    at org.apache.hadoop.net.DNS.reverseDns(DNS.java:92)

This issue is documented here http://docs.prediction.io/resources/faq/#running-hbase

Setting the DNS to use Google's doesn't fix the issue for me in the context of Docker at least. Any idea how to fix this?

Full error here:

pio train
[INFO] [Console$] Using existing engine manifest JSON at /UniversalRecommender/manifest.json
[INFO] [Runner$] Submission command: /PredictionIO-0.9.6/vendors/spark-1.6.2-bin-hadoop2.6/bin/spark-submit --class io.prediction.workflow.CreateWorkflow --jars file:/UniversalRecommender/target/scala-2.10/template-scala-parallel-universal-recommendation-assembly-0.2.3-deps.jar,file:/UniversalRecommender/target/scala-2.10/template-scala-parallel-universal-recommendation_2.10-0.2.3.jar --files file:/PredictionIO-0.9.6/conf/log4j.properties,file:/PredictionIO-0.9.6/vendors/hbase-1.2.2/conf/hbase-site.xml --driver-class-path /PredictionIO-0.9.6/conf:/PredictionIO-0.9.6/vendors/hbase-1.2.2/conf file:/PredictionIO-0.9.6/lib/pio-assembly-0.9.6.jar --engine-id AVewPpSLLYhPU8rUnbgesrtHhxzgjLu4 --engine-version 0cb6183979896c763dceb064bbf96a48e2bd0a12 --engine-variant file:/UniversalRecommender/engine.json --verbosity 0 --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_VERSION=0.9.6,PIO_FS_BASEDIR=/root/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost,PIO_STORAGE_SOURCES_HBASE_HOME=/PredictionIO-0.9.6/vendors/hbase-1.2.2,PIO_HOME=/PredictionIO-0.9.6,PIO_FS_ENGINESDIR=/root/.pio_store/engines,PIO_STORAGE_SOURCES_LOCALFS_PATH=/root/.pio_store/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=predictionio,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=/PredictionIO-0.9.6/vendors/elasticsearch-1.7.5,PIO_FS_TMPDIR=/root/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/PredictionIO-0.9.6/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
[INFO] [Engine] Extracting datasource params...
[INFO] [WorkflowUtils$] No 'name' is found. Default empty String will be used.
[INFO] [Engine] Datasource params: (,DataSourceParams(handmade,List(purchase, view)))
[INFO] [Engine] Extracting preparator params...
[INFO] [Engine] Preparator params: (,Empty)
[INFO] [Engine] Extracting serving params...
[INFO] [Engine] Serving params: (,Empty)
[INFO] [Remoting] Starting remoting
[INFO] [Remoting] Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@172.17.0.2:44144]
[INFO] [Engine$] EngineWorkflow.train
[INFO] [Engine$] DataSource: org.template.DataSource@2f09e6b2
[INFO] [Engine$] Preparator: org.template.Preparator@13f40d71
[INFO] [Engine$] AlgorithmList: List(org.template.URAlgorithm@7dbae40)
[INFO] [Engine$] Data sanity check is on.
Exception in thread "main" java.lang.NullPointerException
    at org.apache.hadoop.net.DNS.reverseDns(DNS.java:92)
    at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.reverseDNS(TableInputFormatBase.java:228)
    at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:191)
    at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:120)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
    at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
    at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
    at org.apache.spark.Partitioner$.defaultPartitioner(Partitioner.scala:65)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$aggregateByKey$3.apply(PairRDDFunctions.scala:206)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$aggregateByKey$3.apply(PairRDDFunctions.scala:206)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
    at org.apache.spark.rdd.PairRDDFunctions.aggregateByKey(PairRDDFunctions.scala:205)
    at io.prediction.data.storage.PEventAggregator$.aggregateProperties(PEventAggregator.scala:198)
    at io.prediction.data.storage.PEvents$class.aggregateProperties(PEvents.scala:118)
    at io.prediction.data.storage.hbase.HBPEvents.aggregateProperties(HBPEvents.scala:33)
    at io.prediction.data.store.PEventStore$.aggregateProperties(PEventStore.scala:112)
    at org.template.DataSource.readTraining(DataSource.scala:83)
    at org.template.DataSource.readTraining(DataSource.scala:46)
    at io.prediction.controller.PDataSource.readTrainingBase(PDataSource.scala:37)
    at io.prediction.controller.Engine$.train(Engine.scala:641)
    at io.prediction.controller.Engine.train(Engine.scala:174)
    at io.prediction.workflow.CoreWorkflow$.runTrain(CoreWorkflow.scala:65)
    at io.prediction.workflow.CreateWorkflow$.main(CreateWorkflow.scala:247)
    at io.prediction.workflow.CreateWorkflow.main(CreateWorkflow.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
djones commented 8 years ago

Here's a related issue https://github.com/mingfang/docker-predictionio/issues/10

I notice the mingfang version of hbase-site.xml has some extra DNS interface configuration set. https://github.com/mingfang/docker-predictionio/blob/master/hbase-site.xml#L10-L17

otidh commented 7 years ago

Hi @djones, I also encountered this issue. Have you found the solution for it?

I have also done "docker run -it --dns=8.8.8.8" as suggested in the link you've given above, but the issue is still existing.

djones commented 7 years ago

@otidh running docker run -it --dns=8.8.8.8 fixed it for me in the end.

otidh commented 7 years ago

@djones I still don't know why it's still not working even though I have used that command when running the container. But in the end the issue was fixed after I manually edited /etc/dhcp/dhclient.conf and put 8.8.8.8 as value for domain-name-servers