When needing to train an NER model I'm unable to specify a tensorflow graph to be used with the graphFolder parameter. I have the graph file "blstm_45_100_128_99.pb" in a directory on HDFS (/user/hadoop) and specify this directory in the setGraphFolder() on the NerDLApproach object.
I create a pipeline that includes this NerDLApproach object and then try to call .fit() using the dataset created from the CoNLL file I have.
The following error is thrown:
Error: java.lang.IllegalArgumentException: Wrong FS: hdfs://ip-XXX-XX-XXX-XX.emr.datafusion.idexx.com:8020/user/hadoop/blstm_45_100_128_99.pb, expected: file:/// at
org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:669) at
org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:86) at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:635) at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866) at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630) at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:452) at
org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340) at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2049) at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2017) at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1982) at
com.johnsnowlabs.nlp.util.io.ResourceHelper$SourceStream.copyToLocal(ResourceHelper.scala:61) at
com.johnsnowlabs.nlp.util.io.ResourceHelper$.copyToLocal(ResourceHelper.scala:83) at
com.johnsnowlabs.nlp.annotators.ner.dl.WithGraphResolver$$anonfun$5.apply(NerDLApproach.scala:371) at
com.johnsnowlabs.nlp.annotators.ner.dl.WithGraphResolver$$anonfun$5.apply(NerDLApproach.scala:371) at
scala.Option.map(Option.scala:146) at
com.johnsnowlabs.nlp.annotators.ner.dl.WithGraphResolver$class.searchForSuitableGraph(NerDLApproach.scala:371) at
com.johnsnowlabs.nlp.annotators.ner.dl.NerDLApproach$.searchForSuitableGraph(NerDLApproach.scala:433) at
com.johnsnowlabs.nlp.annotators.ner.dl.NerDLApproach.train(NerDLApproach.scala:313) at
com.johnsnowlabs.nlp.annotators.ner.dl.NerDLApproach.train(NerDLApproach.scala:30) at
com.johnsnowlabs.nlp.AnnotatorApproach._fit(AnnotatorApproach.scala:55) at
com.johnsnowlabs.nlp.AnnotatorApproach.fit(AnnotatorApproach.scala:61) at
org.apache.spark.ml.Pipeline$$anonfun$fit$2.apply(Pipeline.scala:153) at
org.apache.spark.ml.Pipeline$$anonfun$fit$2.apply(Pipeline.scala:149) at
scala.collection.Iterator$class.foreach(Iterator.scala:891) at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at
scala.collection.IterableViewLike$Transformed$class.foreach(IterableViewLike.scala:44) at
scala.collection.SeqViewLike$AbstractTransformed.foreach(SeqViewLike.scala:37) at
org.apache.spark.ml.Pipeline.fit(Pipeline.scala:149) at org.apache.spark.ml.Pipeline.fit(Pipeline.scala:96) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at
java.lang.reflect.Method.invoke(Method.java:498) at sparklyr.Invoke.invoke(invoke.scala:147) at
sparklyr.StreamHandler.handleMethodCall(stream.scala:136) at sparklyr.StreamHandler.read(stream.scala:61) at
sparklyr.BackendHandler$$anonfun$channelRead0$1.apply$mcV$sp(handler.scala:58) at
scala.util.control.Breaks.breakable(Breaks.scala:38) at sparklyr.BackendHandler.channelRead0(handler.scala:38) at
sparklyr.BackendHandler.channelRead0(handler.scala:14) at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138) at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) at
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at
java.lang.Thread.run(Thread.java:748)
Description
When needing to train an NER model I'm unable to specify a tensorflow graph to be used with the graphFolder parameter. I have the graph file "blstm_45_100_128_99.pb" in a directory on HDFS (/user/hadoop) and specify this directory in the setGraphFolder() on the NerDLApproach object.
I create a pipeline that includes this NerDLApproach object and then try to call
.fit()
using the dataset created from the CoNLL file I have.The following error is thrown:
Context
Unable to train our own NER model.
Your Environment