Default liveliness and readiness probes return 401

[gi-dorio]

I have installed the solr operator, and I'm now trying to install solr. To do so, I'm using the chart i found here. I would like to install it and have LDAP authentication, so I went for the option of providing my own security.json in a Secret. The content of the Secret stringData is as follows

    {
      "authentication": {
        "class": "solr.HadoopAuthPlugin",
        "sysPropPrefix": "solr.",
        "type": "ldap",
        "authConfigs": [
          "ldap.providerurl",
          "ldap.basedn",
          "ldap.enablestarttls"
        ],
        "defaultConfigs": {
          "ldap.providerurl": <my-ldap-url>,
          "ldap.basedn": <my-ldap-base-dn>,
          "ldap.enablestarttls": "false"
        }
      },
      "authorization": {
        "class": "solr.RuleBasedAuthorizationPlugin",
        "user-role": {
          "solr-admin": ["admin", "k8s"],
          "k8s-oper": ["k8s"],
          "solr": ["users", "k8s"]
        },
        "permissions": [
          { "name": "k8s-probe-0", "role": null, "collection": null, "path":"/admin/info/system" },
          { "name": "k8s-probe-1", "role": null, "collection": null, "path":"/admin/info/health" },
          { "name": "k8s-status", "role": "k8s", "collection": null, "path":"/admin/collections" },
          { "name": "k8s-metrics", "role": null, "collection": null, "path":"/admin/metrics" },
          { "name": "k8s-zk", "role": "k8s", "collection": null, "path":"/admin/zookeeper/status" },
          { "name": "k8s-ping", "role": "k8s", "collection": "*", "path":"/admin/ping" },
          { "name": "read", "role": ["admin","users"] },
          { "name": "update", "role": ["admin"] },
          { "name": "security-read", "role": ["admin"] },
          { "name": "security-edit", "role": ["admin"] },
          { "name": "all", "role": ["admin"] }
        ]
      }
    }

I've also modified the relevant values of the helm chart in the following way

solrOptions
  security: 
    authenticationType: Basic
    basicAuthSecret: solr-basic-auth
    probesRequireAuth: false
    bootstrapSecurityJson:
      name: security-json-secret
      key: security.json

Now, if I do not add custom probes in the podOptions section of the values, my pod gets killed because both probes return 401. I've followed the documentation found here, but this doesn't even mention the possibility of getting a 401.

I've also tried to add some custom probes as explained in this section, but they still return 401 and kill my pod.

The only way I was able to have the pod survive was by adding some custom probes that make a curl with the credentials, but, as also mentioned in the docs, probes shouldn't have authentication.

Am i doing something wrong? The docs explicitly tell that one can set the probes to require no auth and it seems to me I have done the necessary steps.

[HoustonPutman]

Under authentication I think you need to add "blockUnknown": false. You are authorized for those endpoints, but since you are not providing the basic auth header, you are getting rejected because you are not authenticated.

In the docs you can find the following snippet:

A few aspects of the default security.json configuration warrant a closer look. First, the probesRequireAuth setting (defaults to false) governs the value for blockUnknown (under authentication) and whether the probe endpoint(s) require authentication:

[gi-dorio]

I think i misinterpreted this part

First, the probesRequireAuth setting (defaults to false) governs the value for blockUnknown (under authentication) and whether the probe endpoint(s) require authentication

I thought it meant that it would pilot the value for blockUnknown in general and not only for the auto bootstrapped security json. Anyways, i tried to add "blockUnknown": false to my security.json authentication part, but no luck. The pod dies as usual and if i describe it, it says

  Warning  Unhealthy          104s (x7 over 2m14s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 401
  Warning  Unhealthy          104s (x3 over 2m4s)   kubelet            Liveness probe failed: HTTP probe failed with statuscode: 401

[HoustonPutman]

Did you try to update an existing cluster or create a new one?

Also even if you deleted and recreated, you need to make sure that Zookeeper didn't use the same persistent volumes as before. Because if the security.json already exists in Solr it won't update it.

If the documentation was confusing, we always appreciate contributions (especially for docs)!

[gi-dorio]

I tried both options, and they both failed. Is there some way I can understand what's going on? Maybe looking at some file? For example, what's the location of the security.json in zookeeper? Maybe I can look there and see if it's using the file I'm giving it (even if I suppose it does, since if I pass the ldap credentials to my probes they work)

[HoustonPutman]

it should be at the top level of your chroot. So /<chroot>/security.json.

Since you are using LDAP, I would expect other Solr Operator commands to fail, since BasicAuth is the only supported auth type. Its so strange that the non-auth part is failing. Definitely see if its using the right security.json!