SolrCore Initialization Failures

[apotek]

I am following the steps in the README.

1. ddev get ddev/ddev-solr
2. ddev restart
3. "Once up and running, access Solr's UI within your browser by opening http://<projectname>.ddev.site:8983"

So, I go to http://<my project>.ddev.site:8983 and log in and see the following:

SolrCore Initialization Failures

dev: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Please check your logs for more information

So I check my logs:

Log4j2

Time (Local)	Level	Core	Logger	Message
4/22/2024, 10:18:14 PM	WARN false		ZkController	Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/zookeeper-access-control.html
4/22/2024, 10:18:14 PM	WARN false		CoreContainer	Solr authentication is enabled,​ but SSL is off. Consider enabling SSL to protect user credentials and data with encryption.
4/22/2024, 10:18:14 PM	ERROR false	x:dev	ZkController
4/22/2024, 10:18:14 PM	ERROR false	CoreContainer	Error waiting for SolrCore to be loaded on startup

Consequences

1. I also note that no cores or collections have been initialized on startup (according to the solr UI). I was expecting that one would be created in the docker compose build steps.

2. Sure enough, just as I feared, when I configure my connection to solr in Drupal with:

Server Name: solr Backend: Solr Solr Connector: Solr Cloud with Basic Auth Solr node: solr Solr port: 8983 Solr Path: / Default solr collection: mycollection

http basic auth: (username and password as specified).

Save

I get the following message:

Warning message Server solr is reachable but the configured collection is not available.

I check the solr admin page and I see that solr, as I feared, does not see my collection. I forge ahead...

On the server's "view" page click the Upload Configset button and check the "Upload (and overwrite)

Ok. I follow these instructions. Number of shards 1, etc, And I click the upload button.

I get the following error:

Error message The configs of the essential Solr field types are missing or broken for server "solr".

So it looks like the container is failing to be initialized with something essential pre-requisite setups.

When I ssh in to the container, I see indeed that no collection has been created on the file system:

$ ls -al /var/solr/data
total 24
drwxrwx--- 4 solr solr 4096 Apr 23 02:52 .
drwxrwx--- 4 solr root 4096 Apr 23 02:12 ..
drwxr-xr-x 2 solr solr 4096 Apr 23 02:18 filestore
-rw-r--r-- 1 solr solr 2427 Jan 12 17:42 solr.xml
drwxr-xr-x 2 solr solr 4096 Apr 23 02:18 userfiles
-rw-r--r-- 1 solr solr 1048 Jan 12 17:42 zoo.cfg

Viewing the solr server I set up, I see that I can connect, but the collection is not available. "The Solr collection could not be accessed. Further data is therefore unavailable."

You can see also in the search_api_solr module output that the module is making a very odd request to solr:8983/solr/benz/admin/luke??? I have no idea why it would be requesting such a path.

I have made no changes to anything other than the ones specified in the README. No customizations to the image, or to .ddev/docker-compose.solr.yaml

Changing connection from auto detect to 8.x doesn't help.

Trying to make it work

Using the solr admin UI, I created a collection, using the same name as the failed attempt from the search_api_solr module.

That succeeded, and when I visited the server "view" page again, I get a message saying that the module can see the collection:

The Solr collection could be accessed (latency: 11.675165725708 ms).

Ok Great! Time to upload the configset!

I click the Upload configset button and say yes to overwrite the _default config set for my collection.

And then I get this error in search_api_solr:

Error message The configs of the essential Solr field types are missing or broken for server "solr". You are using an incompatible Solr schema. Please follow the instructions described in the README.md file for setting up Solr.

Looking in the logs on the solr server, I see:

solrconfig.xml: is no longer supported,​ use solr.xml:/metrics/reporter section instead

I don't know how to interpret this but it sure seems like there is an incompatibility between the config set search_api_solr wants to generate and what solr-cloud is expecting.

Well, let's try with drush:

% ddev drush --numShards=1 search-api-solr:upload-configset solr

In SolrConfigSetController.php line 336:

The configs of the essential Solr field types are missing or broken for server "solr".

Failed to run drush --numShards=1 search-api-solr:upload-configset solr: exit status 1

I'm stumped. I tore down and removed my ddev setup, purged my volumes and docker images and tried again. Same result.

[apotek]

Just wiped everything, including docker volumes and images and went through all the steps in the README, exactly as specified.

I think this is the source of all the problems. When I attempt to upload my config set (either via drush or via the UI), the following error message is returned:

The configs of the essential Solr field types are missing or broken for server "solr".

I interpret this to mean, given that other users seem to be having success with this ddev add-on, that something has changed in the solr docker container and ddev-solr needs to supply or run a setup command to install the essential solr field types.

% docker --version Docker version 25.0.3, build 4debf41 % ddev --version ddev version v1.22.7

search_api_solr module version: 4.3.2 search_api_solr_admin module is enabled.

[rfay]

You're doing ddev drush @adobe ? Doesn't that mean that this entire problem is happening on the remote server and has nothing to do with this add-on?

[apotek]

You're doing ddev drush @adobe ? Doesn't that mean that this entire problem is happening on the remote server and has nothing to do with this add-on?

Thanks @rfay, but no, "adobe" is just my drush alias for the drupal site in the ddev project I am working on. Nothing is happening with a remote server. I have removed it from the error description in order to remove confusion, but I do wonder if this add on has been tested in multi-site setups. The old ddev-drupal9-solr add works fine in multi-site mode.

One thing I appreciate about this add-on (ddev-solr) is that it doesn't force you to name your "core" (now collection) any one specific thing. But the cost of this seems to be that since nothing is pre-built/pre-configured on the server, that there are more problems in getting to a non-error state. When I ran into the error with the missing basic fields config, I wondered if it could be a change in the upstream docker container ... that this add on used to work but now is assuming certain defaults and configuration that no longer holds.

[rfay]

The old ddev-drupal9-solr add works fine in multi-site mode

I guess there's nothing wrong with using that one.

I do wonder if this add on has been tested in multi-site setups

I have not heard of anyone using ddev-solr with multi-site, but don't know why that would actually be relevant.

this add on used to work but now is assuming certain defaults and configuration that no longer holds

There have been no non-trivial changes to this add-on in 10 months, see https://github.com/ddev/ddev-solr/commits/main/

Please try it in a limited test environment and see if you can sort out what's going on. (Create a trivial project using a quickstart, extend it to a multisite after you have the add-on working, etc.)

[apotek]

Some interesting reading that might have some connection to the issues encountered here:

• DDEV Solr Cloud Instructions do not work: no collection created
• entity type needs to be installed - Cache, Request Dispatcher, Request Handler

[rfay]

Was https://www.drupal.org/project/search_api_solr/issues/3104189 actually the issue for you?

[apotek]

Was https://www.drupal.org/project/search_api_solr/issues/3104189 actually the issue for you?

Not sure. I am waiting for another time window where I can dig deeper and hopefully come up with some useful knowledge to help others.

[mkalkbrenner]

@apotek try to follow the README with a fresh installation of drupal. It seems that you messed up your Solr configs in Drupal.

If that works we can take a closer look.

[mkalkbrenner]

no further feedback

[apotek]

no further feedback

Sorry for the lack of feedback. I didn't have time budget to keep troubleshooting this add-on. We ended up having to use https://github.com/ddev/ddev-drupal-solr in order to get things working. If I ever get the budget to revisit this, I will add more info, but it looks doubtful.

[johnbburg]

Pardon me commenting on a closed issue, but it is the same error.

The readme in https://github.com/ddev/ddev-drupal-solr urges the use of this add on instead. I had initially tried that add on, but saw that note, so trying to use this one, but I am seeing the same errors as apotek originally reported.

I was posting in this thread in Drupal slack https://drupal.slack.com/archives/C3E9QDZ5M/p1720043648456549 about it.

When I try to "Upload and create collection", I get

Creating collection dev failed with error code 400: Solr HTTP error: OK (400) 
{
  "responseHeader": {
    "status": 400,
    "QTime": 855
  },
  "error": {
    "metadata": [
      "error-class",
      "org.apache.solr.common.SolrException",
      "root-error-class",
      "org.apache.solr.common.SolrException"
    ],
    "msg": "Underlying core creation failed while creating collection: dev",
    "code": 400
  }
}

The solr log in ddev shows these errors:

org.apache.solr.core.SolrCoreInitializationException: SolrCore 'dev' is not available due to init failure: 
...
Caused by: org.apache.solr.common.SolrException: 
...
Caused by: org.apache.solr.common.SolrException: No coreNodeName for CoreDescriptor[name=dev;instanceDir=/var/solr/data/dev]

Even when running the commands directly in the solr container.

I tried setting it up with a fresh installatation, and it worked as expected. But I need to add it to this existing project, so I'm just scratching my head here on what's causing this. What's even more mind boggling is that I copied the same exact ddev config ( the .ddev/solr directory as well as the docker compose) from that fresh setup to the existing project I'm trying to apply it to, and it still runs into the error.

Edit: Fat fingered to save button too soon.

[mkalkbrenner]

What happens if you delete the docker solr volume?

[johnbburg]

I think the fix was to simply update to the latest Search API Solr module in Drupal. I had thought I was on the latest version, but after checking, I was on 4.3.2. Updating to 4.3.4 seems to fix the error when uploading the config set.

Sorry to bug you!

[rfay]

Thanks! As you discovered, it's always great to make sure latest stable "everything" before asking for support.