Closed auroredea closed 5 years ago
@auroredea Could you show me some use cases? Do you usually need per method config or it is usually a global config? We also need to add tests, but let's discuss APIs first.
Of course. I have to manage a new model file, and I would like to pass the config on all steps. Also I think it's useful when we met different languages on a corpus, and have to switch by language the ner, for example.
My use case is to use a new french ner model (on datas
which is a Dataframe of french sentences) :
val frenchProperties = new Properties()
frenchProperties.load(IOUtils.readerFromString("StanfordCoreNLP-french.properties"));
frenchProperties.put("pos.model", "edu/stanford/nlp/models/pos-tagger/french/french.tagger")
val frenchConfig = CoreNLPConfig(frenchProperties)
val frenchNer = ner(frenchConfig)
val withNer = datas.withColumn("ner", frenchNer(col("sentences")))
About new tests, as it's only passing custom properties, shall we test only the correct usage of the passed props or you'd prefer a full test with each method ?
Any thoughts @mengxr about that ?
Sorry for late reply! I spent some time to upgrade Spark and CoreNLP versions.
ju.Props
seems sufficient to me.
Hello, We (with @nekonyuu ) added some new features :