Closed rajeshwrn closed 2 years ago
Oh my, we were hoping to get around to this soon, but this is great! Let me check it out and we'll try to get a new major release out in June :)
@rajeshwrn Having some trouble testing this out via unit tests and with integration right now. Will continue working on it to see where I'm going wrong :)
Oh my, we were hoping to get around to this soon, but this is great! Let me check it out and we'll try to get a new major release out in June :) When new release of spark-solr will be out and what's actual version support table? (Seems README.md is not up to date) @falloutdurham
Hi guys Do you know what is the status of this PR? Is it planned to be released soon?
Btw, @rajeshwrn great work!!
I tried today to build this PR locally and use it in a flow which dumps from Solr v8 one of the collections
e.g. scala> val containersMetadataDF = spark.read.format("solr").options(Map("zkhost" -> s"zookeeper1:2181,zookeeper2:2181,zookeeper3:2181/solr-green", "collection"-> "containerMetadata")).load()
This basically failed with:
2021-08-24 12:19:02 WARN BaseHttpClusterStateProvider - Attempt to fetch cluster state from zookeeper1:2181,zookeeper2:2181,zookeeper3:2181/solr-green failed.
org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: zookeeper1:2181,zookeeper2:2181,zookeeper3:2181/solr-green
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:695)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.fetchLiveNodes(BaseHttpClusterStateProvider.java:190)
at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.init(BaseHttpClusterStateProvider.java:64)
at org.apache.solr.client.solrj.impl.HttpClusterStateProvider.<init>(HttpClusterStateProvider.java:34)
at org.apache.solr.client.solrj.impl.CloudSolrClient$Builder.build(CloudSolrClient.java:464)
at com.lucidworks.spark.util.SolrSupport$.getSolrCloudClient(SolrSupport.scala:220)
at com.lucidworks.spark.util.SolrSupport$.getNewSolrCloudClient(SolrSupport.scala:242)
at com.lucidworks.spark.util.CacheCloudSolrClient$$anon$1.load(SolrSupport.scala:38)
at com.lucidworks.spark.util.CacheCloudSolrClient$$anon$1.load(SolrSupport.scala:36)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)
at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)
at com.lucidworks.spark.util.SolrSupport$.getCachedCloudClient(SolrSupport.scala:250)
at com.lucidworks.spark.util.SolrSupport$.getSolrBaseUrl(SolrSupport.scala:254)
at com.lucidworks.spark.SolrRelation.dynamicSuffixes$lzycompute(SolrRelation.scala:99)
at com.lucidworks.spark.SolrRelation.dynamicSuffixes(SolrRelation.scala:97)
at com.lucidworks.spark.SolrRelation.getBaseSchemaFromConfig(SolrRelation.scala:298)
at com.lucidworks.spark.SolrRelation.querySchema$lzycompute(SolrRelation.scala:238)
at com.lucidworks.spark.SolrRelation.querySchema(SolrRelation.scala:107)
at com.lucidworks.spark.SolrRelation.schema(SolrRelation.scala:427)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:448)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:326)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:308)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:308)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:226)
at $line26.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:23)
at $line26.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:27)
at $line26.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:29)
at $line26.$read$$iw$$iw$$iw$$iw$$iw.<init>(<console>:31)
at $line26.$read$$iw$$iw$$iw$$iw.<init>(<console>:33)
at $line26.$read$$iw$$iw$$iw.<init>(<console>:35)
at $line26.$read$$iw$$iw.<init>(<console>:37)
at $line26.$read$$iw.<init>(<console>:39)
at $line26.$read.<init>(<console>:41)
at $line26.$read$.<init>(<console>:45)
at $line26.$read$.<clinit>(<console>)
at $line26.$eval$.$print$lzycompute(<console>:7)
at $line26.$eval$.$print(<console>:6)
at $line26.$eval.$print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:745)
at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1021)
at scala.tools.nsc.interpreter.IMain.$anonfun$interpret$1(IMain.scala:574)
at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:41)
at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:37)
at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:41)
at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:573)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:600)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:570)
at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:894)
at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:762)
at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:464)
at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:485)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:239)
at org.apache.spark.repl.Main$.doMain(Main.scala:78)
at org.apache.spark.repl.Main$.main(Main.scala:58)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:959)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1038)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1047)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: shaded.apache.http.client.ClientProtocolException: URI does not specify a valid host name: zookeeper1:2181,zookeeper2:2181,zookeeper3:2181/solr-green/admin/collections?action=CLUSTERSTATUS&wt=javabin&version=2
at shaded.apache.http.impl.client.CloseableHttpClient.determineTarget(CloseableHttpClient.java:95)
at shaded.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at shaded.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:571)
... 79 more
com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Couldn't initialize a HttpClusterStateProvider (is/are the Solr server(s), [zookeeper1:2181,zookeeper2:2181,zookeeper3:2181/solr-green], down?)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2263)
at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)
at com.lucidworks.spark.util.SolrSupport$.getCachedCloudClient(SolrSupport.scala:250)
at com.lucidworks.spark.util.SolrSupport$.getSolrBaseUrl(SolrSupport.scala:254)
at com.lucidworks.spark.SolrRelation.dynamicSuffixes$lzycompute(SolrRelation.scala:99)
at com.lucidworks.spark.SolrRelation.dynamicSuffixes(SolrRelation.scala:97)
at com.lucidworks.spark.SolrRelation.getBaseSchemaFromConfig(SolrRelation.scala:298)
at com.lucidworks.spark.SolrRelation.querySchema$lzycompute(SolrRelation.scala:238)
at com.lucidworks.spark.SolrRelation.querySchema(SolrRelation.scala:107)
at com.lucidworks.spark.SolrRelation.schema(SolrRelation.scala:427)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:448)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:326)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:308)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:308)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:226)
... 47 elided
Caused by: java.lang.RuntimeException: Couldn't initialize a HttpClusterStateProvider (is/are the Solr server(s), [zookeeper1:2181,zookeeper2:2181,zookeeper3:2181/solr-green], down?)
at org.apache.solr.client.solrj.impl.CloudSolrClient$Builder.build(CloudSolrClient.java:466)
at com.lucidworks.spark.util.SolrSupport$.getSolrCloudClient(SolrSupport.scala:220)
at com.lucidworks.spark.util.SolrSupport$.getNewSolrCloudClient(SolrSupport.scala:242)
at com.lucidworks.spark.util.CacheCloudSolrClient$$anon$1.load(SolrSupport.scala:38)
at com.lucidworks.spark.util.CacheCloudSolrClient$$anon$1.load(SolrSupport.scala:36)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)
... 64 more
Caused by: java.lang.RuntimeException: Tried fetching live_nodes using Solr URLs provided, i.e. [zookeeper1:2181,zookeeper2:2181,zookeeper3:2181/solr-green]. However, succeeded in obtaining the cluster state from none of them.If you think your Solr cluster is up and is accessible, you could try re-creating a new CloudSolrClient using working solrUrl(s) or zkHost(s).
at org.apache.solr.client.solrj.impl.BaseHttpClusterStateProvider.init(BaseHttpClusterStateProvider.java:73)
at org.apache.solr.client.solrj.impl.HttpClusterStateProvider.<init>(HttpClusterStateProvider.java:34)
at org.apache.solr.client.solrj.impl.CloudSolrClient$Builder.build(CloudSolrClient.java:464)
... 72 more
I know this is not yet merged / released but I was wondering if during your tests you found the same issue. I've checked both on both Solr and ZK and these can be accessed from the cluster. So there are no connectivity issues.
@barcac updated the code. It will work now
There was an issue in solr client creation. While updating the deprecated solr method, missed adding one param.
From solrj7.3, the withZkHost() method was deprecated. I tried to change this with Builder constructor but in code solrConfig options only has zkhost. So I reverted this back to the older method. It is working now. Since it is solr version dependent, for spark3 this change is not required.
Looks like this change requires effort and some changes in the design. we need to find a way to pass zkhosts and zkchroot separately in the options. as it is below
val options = Map( "collection" -> "{solr_collection_name}", "zkhost" -> "{zkhost1, zkhost2,zkhost3}", "zkchroot" -> "{zkchroot string}" )
val df = spark.read.format("solr") .options(options) .load
and
val zkServers = List(cloudClientParams.zkhost1, cloudClientParams.zkhost2, cloudClientParams.zkhost3)
val solrClientBuilder = new CloudSolrClient.Builder(zkServers, Optional.of(cloudClientParams.zkchroot))
@falloutdurham this may be the issue that you encountered while executing the unit test cases.
hi @rajeshwrn ! thanks for the quick fix. will give it a try
looking forward for the release of it
@rajeshwrn I will give it another whirl this week! I notice that it's still failing on the test run. If it works after skipTests, though, I'll see if I can look into those failures and get them resolved.
The good news is I've built the new version and it seems promising! I've fixed up some of the failing tests - just one more to go now, but that seems to be an issue with some of the SolrRelations interaction with Catalyst, so it may take me a little longer to track down and eliminate it.
Just got a complete clean run! 💃
Let me get some ducks in a row internally over the next few days - I'll try to merge some of the outstanding PRs and rebase this on top of them before cutting a 4.0.0 release.
Awesome Rajesh
@rajeshwrn tried it now and with this PR's latest changes the snapshot-build works now in our case; no more connectivity issues
Build from Source mvn clean package -DskipTests