search

SearchScale / dataimporthandler

Repository for DIH (Document Import Handler)
Apache License 2.0
68 stars 48 forks source link

When loading configuration from data config file, variable substitution does not work for "solr.core.instanceDir" #65

Open szhou1998 opened 10 months ago

szhou1998 commented 10 months ago

I'm running into an issue where variable substitution is not working properly when loading data config from an XML file. I have an entity element with attribute url="${solr.core.instanceDir}/workDir/claimcontact.xml". When running full-import, I get this error:

"level":"ERROR","loggerName":"org.apache.solr.handler.dataimport.DocBuilder",
"Exception while processing: claimcontact document : SolrInputDocument(fields: [])"
"org.apache.solr.handler.dataimport.DataImportHandlerException: 
java.security.AccessControlException: access denied ("java.io.FilePermission" "/workDir/claimcontact.xml" "read")"

"java.security.AccessControlContext","method":"checkPermission","file":"AccessControlContext.java","line":472
"java.security.AccessController","method":"checkPermission","file":"AccessController.java","line":897
"java.lang.SecurityManager","method":"checkPermission","file":"SecurityManager.java","line":322
"java.lang.SecurityManager","method":"checkRead","file":"SecurityManager.java","line":661
"java.io.File","method":"isFile","file":"File.java","line":889
"org.apache.solr.handler.dataimport.FileDataSource","method":"getFile","file":"FileDataSource.java","line":119
"org.apache.solr.handler.dataimport.FileDataSource","method":"getData","file":"FileDataSource.java","line":86
"org.apache.solr.handler.dataimport.FileDataSource","method":"getData","file":"FileDataSource.java","line":48
"org.apache.solr.handler.dataimport.XPathEntityProcessor","method":"initQuery","file":"XPathEntityProcessor.java","line":291
"org.apache.solr.handler.dataimport.DataImportHandlerException","method":"wrapAndThrow","file":"DataImportHandlerException.java","line":61
"org.apache.solr.handler.dataimport.XPathEntityProcessor","method":"initQuery","file":"XPathEntityProcessor.java","line":294
"org.apache.solr.handler.dataimport.XPathEntityProcessor","method":"fetchNextRow","file":"XPathEntityProcessor.java","line":232
"org.apache.solr.handler.dataimport.XPathEntityProcessor","method":"nextRow","file":"XPathEntityProcessor.java","line":212
"org.apache.solr.handler.dataimport.EntityProcessorWrapper","method":"nextRow","file":"EntityProcessorWrapper.java","line":267
"org.apache.solr.handler.dataimport.DocBuilder","method":"buildDocument","file":"DocBuilder.java","line":494
"org.apache.solr.handler.dataimport.DocBuilder","method":"buildDocument","file":"DocBuilder.java","line":433
"org.apache.solr.handler.dataimport.DocBuilder","method":"doFullDump","file":"DocBuilder.java","line":348
"org.apache.solr.handler.dataimport.DocBuilder","method":"execute","file":"DocBuilder.java","line":236
"org.apache.solr.handler.dataimport.DataImporter","method":"doFullImport","file":"DataImporter.java","line":424
"org.apache.solr.handler.dataimport.DataImporter","method":"runCmd","file":"DataImporter.java","line":483
"org.apache.solr.handler.dataimport.DataImporter","method":"lambda$runAsync$0","file":"DataImporter.java","line":466

As you can see, the solr.core.instanceDir variable is not being substituted correctly. I believe the issue is with this bit of code in DocBuilder: https://github.com/SearchScale/dataimporthandler/blob/branch_9x/src/main/java/org/apache/solr/handler/dataimport/DocBuilder.java#L116-L120

In Solr 8, a change was made to get the properties from CoreDescriptor substitutableProperties: https://github.com/apache/lucene-solr/commit/17ae79b0905b2bf8635c1b260b30807cae2f5463#diff-9eadac54fe1ace540cb05b6711791be03a9289a7d6f16615fb4efa4ee31641f9

CoreDescriptor substitutableProperties has solr.core.instanceDir as a property: https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/core/CoreDescriptor.java#L269

However, note that in the Solr 9 DocBuilder change in this repo, the Solr 8 "before" code did not have this change: https://github.com/SearchScale/dataimporthandler/commit/ec730d5ed48822001d7281619876a9a631e0f25a#diff-88595e01694c11981cecf4b71fdc54969870ff49737208856685388eb3bb0ee5

Should this be changed to new VariableResolver(dataImporter.getCore().getCoreDescriptor().getSubstitutableProperties());?