I am using spark-solr to feed data from spark to solr for a while now. I dont have any issues, and I usually have process which will run for 2 hrs, feeding more then 80 millions of records. I was doing this all the time without any kind of issues.
However, after adding basic authentication to the solr side, I cannot make this work. I provided parameters as per documentation, but I am still getting 401 from solr side during execution.
I am not sure am I doing something wrong, or its the issue with versions I am using on spark-solr, solr or spark side? Credentials provided are definitely ok, and I am able to login to the solr admin ui without any kind of issues using these creds.
Platform for execution: aws - emr Emr version: emr-5.33.0 Spark version: 2.4.7 - amzn1 Solr version: 8.9.0
Execution with arguments: spark-submit --deploy-mode cluster --master yarn --conf spark.yarn.submit.waitAppCompletion=true --conf spark.driver.extraJavaOptions=-Dbasicauth=admin:test1234 --conf spark.executor.extraJavaOptions=-Dbasicauth=admin:test1234 --py-files s3a://my-data-scripts/pydependencies.zip --files s3a://my-data-scripts/config/config.emr.ini --jars /home/hadoop/spark-solr-3.9.0-shaded.jar s3a://emr-openaccess-scripts/main.py step_6
I am using spark-solr to feed data from spark to solr for a while now. I dont have any issues, and I usually have process which will run for 2 hrs, feeding more then 80 millions of records. I was doing this all the time without any kind of issues.
However, after adding basic authentication to the solr side, I cannot make this work. I provided parameters as per documentation, but I am still getting 401 from solr side during execution.
I am not sure am I doing something wrong, or its the issue with versions I am using on spark-solr, solr or spark side? Credentials provided are definitely ok, and I am able to login to the solr admin ui without any kind of issues using these creds.