search

silverstripe / addons.silverstripe.org

Website hosting Silverstripe Framework extensions
BSD 3-Clause "New" or "Revised" License
13 stars 16 forks source link

Issues with search re-indexing #241

Closed brynwhyman closed 5 years ago

brynwhyman commented 5 years ago

Following the SS4 upgrade is appears there are issues with search reindexing. This needs to be resolved

sminnee commented 5 years ago

This seems to be the error:

[2019-04-01 12:10:03][INFO] Job caused exception mapper [downloads] of different type, current_type [long], merged_type [text] in /var/www/mysite/releases/f62b7d0443bd65b2e0e71cb04651b00d73f80b41/vendor/ruflin/elastica/lib/Elastica/Transport/Guzzle.php at line 101
[2019-04-01 12:10:03][INFO] Defining the mappings
ERROR [Emergency]: Uncaught Elastica\Exception\ResponseException: mapper [downloads] of different type, current_type [long], merged_type [text]
IN GET dev/tasks/ProcessJobQueueTask
Line 101 in /var/www/mysite/releases/f62b7d0443bd65b2e0e71cb04651b00d73f80b41/vendor/ruflin/elastica/lib/Elastica/Transport/Guzzle.php

Source
======
92: 
93: $response->setTransferInfo(
94: [
95: 'request_header' => $request->getMethod(),
96: 'http_code' => $res->getStatusCode(),
97: ]
98: );
99: 
100: if ($response->hasError()) {
* 101: throw new ResponseException($request, $response);
102: }
103: 
104: if ($response->hasFailedShards()) {
105: throw new PartialShardFailureException($request, $response);
106: }
107: 

Trace
=====
Elastica\Transport\Guzzle->exec({"path":"ss_mysite\/Addon\/_mapping","method":"PUT","data":{"Addon":{"properties":{"name":{"type":"string"},"description":{"type":"string"},"type":{"type":"string"},"compatibility":{"type":"string"},"vendor":{"type":"string"},"tags":{"type":"string"},"released":{"type":"date"},"downloads":{"type":"string"},"readme":{"type":"string"}}}},"query":[],"contentType":"application\/json","connection":{"config":{"url":null,"bigintConversion":false},"host":"search-cloudy-ssaddon1-prod2xes-toi7xpgnfuyoa3x7zkipva7j5q.ap-southeast-2.es.amazonaws.com","port":80,"path":null,"proxy":null,"transport":"AwsAuthV4","persistent":true,"timeout":5,"connections":[],"roundRobin":false,"log":false,"retryOnConflict":0,"username":null,"password":null,"aws_region":"ap-southeast-2","enabled":true}}, Array)
Request.php:193

Elastica\Request->send()
Client.php:674

Elastica\Client->request(ss_mysite/Addon/_mapping, PUT, Array, Array)
Client.php:706

Elastica\Client->requestEndpoint(Elasticsearch\Endpoints\Indices\Mapping\Put)
Index.php:562

Elastica\Index->requestEndpoint(Elasticsearch\Endpoints\Indices\Mapping\Put)
Type.php:523

Elastica\Type->requestEndpoint(Elasticsearch\Endpoints\Indices\Mapping\Put)
Mapping.php:250

Elastica\Type\Mapping->send()
ElasticaService.php:218

Heyday\Elastica\ElasticaService->define()
ReindexTask.php:41

Heyday\Elastica\ReindexTask->run(SilverStripe\Control\HTTPRequest)
RunBuildTaskJob.php:80

Symbiote\QueuedJobs\Jobs\RunBuildTaskJob->process()
QueuedJobService.php:723

Symbiote\QueuedJobs\Services\QueuedJobService->runJob(568)
QueuedJobService.php:1117

Symbiote\QueuedJobs\Services\QueuedJobService->processJobQueue(2)
QueueRunner.php:23

Symbiote\QueuedJobs\Tasks\Engines\QueueRunner->runQueue(2)
QueuedJobService.php:1094

Symbiote\QueuedJobs\Services\QueuedJobService->runQueue(2)
ProcessJobQueueTask.php:59

Symbiote\QueuedJobs\Tasks\ProcessJobQueueTask->run(SilverStripe\Control\HTTPRequest)
TaskRunner.php:104

SilverStripe\Dev\TaskRunner->runTask(SilverStripe\Control\HTTPRequest)
RequestHandler.php:323

SilverStripe\Control\RequestHandler->handleAction(SilverStripe\Control\HTTPRequest, runTask)
Controller.php:284

SilverStripe\Control\Controller->handleAction(SilverStripe\Control\HTTPRequest, runTask)
RequestHandler.php:202

SilverStripe\Control\RequestHandler->handleRequest(SilverStripe\Control\HTTPRequest)
Controller.php:212

SilverStripe\Control\Controller->handleRequest(SilverStripe\Control\HTTPRequest)
RequestHandler.php:226

SilverStripe\Control\RequestHandler->handleRequest(SilverStripe\Control\HTTPRequest)
Controller.php:212

SilverStripe\Control\Controller->handleRequest(SilverStripe\Control\HTTPRequest)
Director.php:361

SilverStripe\Control\Director->SilverStripe\Control\{closure}(SilverStripe\Control\HTTPRequest)
VersionedHTTPMiddleware.php:41

SilverStripe\Versioned\VersionedHTTPMiddleware->process(SilverStripe\Control\HTTPRequest, Closure)
HTTPMiddlewareAware.php:62

SilverStripe\Control\Director->SilverStripe\Control\Middleware\{closure}(SilverStripe\Control\HTTPRequest)
BasicAuthMiddleware.php:68

SilverStripe\Security\BasicAuthMiddleware->process(SilverStripe\Control\HTTPRequest, Closure)
HTTPMiddlewareAware.php:62

SilverStripe\Control\Director->SilverStripe\Control\Middleware\{closure}(SilverStripe\Control\HTTPRequest)
AuthenticationMiddleware.php:61

SilverStripe\Security\AuthenticationMiddleware->process(SilverStripe\Control\HTTPRequest, Closure)
HTTPMiddlewareAware.php:62

SilverStripe\Control\Director->SilverStripe\Control\Middleware\{closure}(SilverStripe\Control\HTTPRequest)
CanonicalURLMiddleware.php:188

SilverStripe\Control\Middleware\CanonicalURLMiddleware->process(SilverStripe\Control\HTTPRequest, Closure)
HTTPMiddlewareAware.php:62

SilverStripe\Control\Director->SilverStripe\Control\Middleware\{closure}(SilverStripe\Control\HTTPRequest)
HTTPCacheControlMiddleware.php:42

SilverStripe\Control\Middleware\HTTPCacheControlMiddleware->process(SilverStripe\Control\HTTPRequest, Closure)
HTTPMiddlewareAware.php:62

SilverStripe\Control\Director->SilverStripe\Control\Middleware\{closure}(SilverStripe\Control\HTTPRequest)
ChangeDetectionMiddleware.php:27

SilverStripe\Control\Middleware\ChangeDetectionMiddleware->process(SilverStripe\Control\HTTPRequest, Closure)
HTTPMiddlewareAware.php:62

SilverStripe\Control\Director->SilverStripe\Control\Middleware\{closure}(SilverStripe\Control\HTTPRequest)
FlushMiddleware.php:29

SilverStripe\Control\Middleware\FlushMiddleware->process(SilverStripe\Control\HTTPRequest, Closure)
HTTPMiddlewareAware.php:62

SilverStripe\Control\Director->SilverStripe\Control\Middleware\{closure}(SilverStripe\Control\HTTPRequest)
RequestProcessor.php:66

SilverStripe\Control\RequestProcessor->process(SilverStripe\Control\HTTPRequest, Closure)
HTTPMiddlewareAware.php:62

SilverStripe\Control\Director->SilverStripe\Control\Middleware\{closure}(SilverStripe\Control\HTTPRequest)
SessionMiddleware.php:20

SilverStripe\Control\Middleware\SessionMiddleware->process(SilverStripe\Control\HTTPRequest, Closure)
HTTPMiddlewareAware.php:62

SilverStripe\Control\Director->SilverStripe\Control\Middleware\{closure}(SilverStripe\Control\HTTPRequest)
AllowedHostsMiddleware.php:60

SilverStripe\Control\Middleware\AllowedHostsMiddleware->process(SilverStripe\Control\HTTPRequest, Closure)
HTTPMiddlewareAware.php:62

SilverStripe\Control\Director->SilverStripe\Control\Middleware\{closure}(SilverStripe\Control\HTTPRequest)
TrustedProxyMiddleware.php:176

SilverStripe\Control\Middleware\TrustedProxyMiddleware->process(SilverStripe\Control\HTTPRequest, Closure)
HTTPMiddlewareAware.php:62

SilverStripe\Control\Director->SilverStripe\Control\Middleware\{closure}(SilverStripe\Control\HTTPRequest)
HTTPMiddlewareAware.php:65

SilverStripe\Control\Director->callMiddleware(SilverStripe\Control\HTTPRequest, Closure)
Director.php:370

SilverStripe\Control\Director->handleRequest(SilverStripe\Control\HTTPRequest)
HTTPApplication.php:48

SilverStripe\Control\HTTPApplication->SilverStripe\Control\{closure}(SilverStripe\Control\HTTPRequest)

call_user_func(Closure, SilverStripe\Control\HTTPRequest)
HTTPApplication.php:66

SilverStripe\Control\HTTPApplication->SilverStripe\Control\{closure}(SilverStripe\Control\HTTPRequest)
HTTPMiddlewareAware.php:65

SilverStripe\Control\HTTPApplication->callMiddleware(SilverStripe\Control\HTTPRequest, Closure)
HTTPApplication.php:67

SilverStripe\Control\HTTPApplication->execute(SilverStripe\Control\HTTPRequest, Closure, )
HTTPApplication.php:47

SilverStripe\Control\HTTPApplication->handle(SilverStripe\Control\HTTPRequest)
cli-script.php:22

[2019-04-01 12:10:03][INFO]
[2019-04-01 12:10:03][INFO] Job paused at 2019-04-01 12:10:03
sminnee commented 5 years ago

In theory this could be fixed by changing this to "long" or something, but it's been a string since the SS3 version too, which is a big confusing.

https://github.com/silverstripe/addons.silverstripe.org/blob/master/app/src/dataobjects/Addon.php#L140

Perhaps nuke the index and reindex?

robbieaverill commented 5 years ago

I'm running 5.6.16 locally and not having this issue. Will try and find out exactly which version SSP is running to see if there's a discrepancy.

robbieaverill commented 5 years ago

I've tried recreating a new index. I can't see a way to clear the current index. That didn't help anyway, so I'll look at adjusting the type instead.

sminnee commented 5 years ago

"long" feels like a more correct type for an Int than "string", in any case.

robbieaverill commented 5 years ago

I've had the index reset, it's still evaluating to "long", and ES is trying to save it as a string. My local environment's index looks the same as the UAT environment, but the production environment looks different.

I don't know enough about this, for now I'm going to try setting the downloads field to be indexed as a string via config.

Addon::getElasticaMapping() has explicit casting for each field, but it looks like this isn't being used on production. The elastica module is probably being allowed to introspect the DB types and make its own decisions, which I assume the presence of the getElasticaMapping() method should prevent. I don't know why this happens on production and not on UAT (and local) which would explain the difference in the indexes.

robbieaverill commented 5 years ago

Here's a summary of where I'm at:

Failing call to set new index mapping:

Elastica\Transport\Guzzle->exec({"path":"ss_mysite\/Addon\/_mapping","method":"PUT","data":{"Addon":{"properties":{"name":{"type":"string"},"description":{"type":"string"},"type":{"type":"string"},"compatibility":{"type":"string"},"vendor":{"type":"string"},"tags":{"type":"string"},"released":{"type":"date"},"downloads":{"type":"string"},"readme":{"type":"string"}}}},"query":[],"contentType":"application\/json","connection":{"config":{"url":null,"bigintConversion":false},"host":"MYELASTICHOSTHERE","port":80,"path":null,"proxy":null,"transport":"AwsAuthV4","persistent":true,"timeout":5,"connections":[],"roundRobin":false,"log":false,"retryOnConflict":0,"username":null,"password":null,"aws_region":"ap-southeast-2","enabled":true}}, Array)
Request.php:193

Note "downloads":{"type":"string"} above.

Production ES index spec:

{
  "ss_mysite": {
    "aliases": {},
    "mappings": {
      "Addon": {
        "properties": {
          "SS_Published": {
            "type": "boolean"
          },
          "compatibility": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "description": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "downloads": {
            "type": "long"
          },
          "name": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "readme": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "released": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "tags": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "type": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "vendor": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    },
    "settings": {
      "index": {
        "creation_date": "1554338601480",
        "number_of_shards": "5",
        "number_of_replicas": "1",
        "uuid": "qMeCpAhgR_SyptSOimjWZA",
        "version": {
          "created": "5060899"
        },
        "provided_name": "ss_mysite"
      }
    }
  }
}

UAT/local index spec:

{
  "ss_mysite": {
    "aliases": {},
    "mappings": {
      "Addon": {
        "properties": {
          "SS_Published": {
            "type": "boolean"
          },
          "compatibility": {
            "type": "text"
          },
          "description": {
            "type": "text"
          },
          "downloads": {
            "type": "text"
          },
          "name": {
            "type": "text"
          },
          "readme": {
            "type": "text"
          },
          "released": {
            "type": "date"
          },
          "tags": {
            "type": "text"
          },
          "type": {
            "type": "text"
          },
          "vendor": {
            "type": "text"
          }
        }
      }
    },
    "settings": {
      "index": {
        "creation_date": "1551313022147",
        "number_of_shards": "5",
        "number_of_replicas": "1",
        "uuid": "wXdfr8TySb-R6KiVxVLUJw",
        "version": {
          "created": "5060899"
        },
        "provided_name": "ss_mysite"
      }
    }
  }
}

@stogj manually cleared the index for me and said that when he checked it again it still had an index definition. It should have responded with something like this:

{
  "error": {
    "root_cause": [
      {
        "type": "index_not_found_exception",
        "reason": "no such index",
        "resource.type": "index_or_alias",
        "resource.id": "addons",
        "index_uuid": "_na_",
        "index": "addons"
      }
    ],
    "type": "index_not_found_exception",
    "reason": "no such index",
    "resource.type": "index_or_alias",
    "resource.id": "addons",
    "index_uuid": "_na_",
    "index": "addons"
  },
  "status": 404
}

Nevertheless, he did something and I tried again. The reindex task then worked successfully, and when I refreshed to run it again (via browser) it reverted back to failing again.

It feels like something is interjecting and changing the index spec outside of the application layer (to me).

I'm going to try switching the mapping to a long explicitly instead.

sminnee commented 5 years ago

Cool. If this fixes it, it might be worth filing some upstream issues relating to changing index types.

robbieaverill commented 5 years ago

Alright well now I'm getting the inverse error (string vs long) on UAT from this change. At least I can reproduce the issue locally now (addons is my index name):

For whatever reason the ES infrastructure in production isn't clearing correctly, but when reindexing ES from the app layer it should delete the entire index including mappings. I'll see if I can adjust the module to do this.

robbieaverill commented 5 years ago

Ok I've raised the issue at https://github.com/heyday/silverstripe-elastica/issues/19 and made a patch PR at https://github.com/silverstripe/addons.silverstripe.org/pull/243, and have fixed this in our project with #243 so it deletes the existing index every time it goes to reindex it.

Tested on UAT and deployed to production, working. Happy to close this now.