Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default.

TorlockC commented 1 year ago

Describe your question

Hi there,

I have the following error on all searches at the moment. Not sure if it's related to a recent update or rsync but resyncing doesn't seem to resolve the issue.

We do have a lot of terms so setting fielddata=true on terms.pa_10g-support.slug just returns the same error on another term and assuming that setting it to true on all of these is going to have some major performance impact.

Any idea on the best approach to fix this one?

[body] => {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [terms.pa_10g-support.slug] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"transparentukcom-post-1","node":"TO4JT9PYT5qNflqLF4oogA","reason":{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [terms.pa_10g-support.slug] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}}],"caused_by":{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [terms.pa_10g-support.slug] in order to load field data by uninverting the inverted index. Note that this can use significant memory.","caused_by":{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [terms.pa_10g-support.slug] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}}},"status":400}

Code of Conduct

[X] I agree to follow this project's Code of Conduct

felipeelia commented 1 year ago

Hi @TorlockC,

As you can see here, the terms.*.slug field is already a keyword, which makes me think your mapping is missing. Did you already try to run a full sync (delete all data and sync)?

Also, do you mind sharing your status report and any custom code related to EP you have (if any)? Thanks

github-actions[bot] commented 1 year ago

This issue has been automatically closed because there has been no response to our request for more information in the past 3 days. With only the information that is currently available, we are unable to take further action on this ticket. Please reach out if you have found or find the answers we need so that we can investigate further. When the information is ready, you can re-open this ticket to share it with us.

TorlockC commented 1 year ago

Hi @felipeelia,

This was after a complete full sync. Is there a way to resend the mapping and check it over or would that just be trying a full sync again? Status report is below and no custom code at the moment. I've removed wp_args and some of the body text from the report errors as it is too much text to paste on Github so sorry if it's a little tricky to read and please let me know if I left out anything important to diagnose further.

I think we could actually live without the vast majority of these terms being in the index altogether if that is an option as well? We search more on the short/long desc, some meta data and the pa_brand attribute, most of the others are not really needed.

result: array (
  'error' => 
  array (
    'root_cause' => 
    array (
      0 => 
      array (
        'type' => 'query_shard_exception',
        'reason' => 'No mapping found for [post_title.sortable] in order to sort on',
        'index_uuid' => 'NfZbo9TZRX-QQb2vC6Mnlw',
        'index' => 'transparentukcom-post-1',
      ),
    ),
    'type' => 'search_phase_execution_exception',
    'reason' => 'all shards failed',
    'phase' => 'query',
    'grouped' => true,
    'failed_shards' => 
    array (
      0 => 
      array (
        'shard' => 0,
        'index' => 'transparentukcom-post-1',
        'node' => 'TO4JT9PYT5qNflqLF4oogA',
        'reason' => 
        array (
          'type' => 'query_shard_exception',
          'reason' => 'No mapping found for [post_title.sortable] in order to sort on',
          'index_uuid' => 'NfZbo9TZRX-QQb2vC6Mnlw',
          'index' => 'transparentukcom-post-1',
        ),
      ),
    ),
  ),
  'status' => 400,
)

### https://transparent-uk.com/prod-cat/peripherals/power-products/page/94 (2023-08-10 15:41:30) ###
error: No mapping found for [post_title.sortable] in order to sort on
recommended_solution: The field <code>post_title.sortable</code> was not found. Make sure it is added to the list of indexed fields and run <a href="https://transparent-uk.com/wp-admin/admin.php?page=elasticpress-sync">a new sync</a> to fix the issue.
es_req: POST http://amazonaws.com:9200/post-1/_search
request_id: fecedcc2a09842028c6a59cfe36a5ca4
query_time: 10.022878646851
result: array (
  'error' => 
  array (
    'root_cause' => 
    array (
      0 => 
      array (
        'type' => 'query_shard_exception',
        'reason' => 'No mapping found for [post_title.sortable] in order to sort on',
        'index_uuid' => 'NfZbo9TZRX-QQb2vC6Mnlw',
        'index' => 'transparentukcom-post-1',
      ),
    ),
    'type' => 'search_phase_execution_exception',
    'reason' => 'all shards failed',
    'phase' => 'query',
    'grouped' => true,
    'failed_shards' => 
    array (
      0 => 
      array (
        'shard' => 0,
        'index' => 'transparentukcom-post-1',
        'node' => 'TO4JT9PYT5qNflqLF4oogA',
        'reason' => 
        array (
          'type' => 'query_shard_exception',
          'reason' => 'No mapping found for [post_title.sortable] in order to sort on',
          'index_uuid' => 'NfZbo9TZRX-QQb2vC6Mnlw',
          'index' => 'transparentukcom-post-1',
        ),
      ),
    ),
  ),
  'status' => 400,
)

### https://transparent-uk.com/?s=FD-W-2-S2401&post_type=product (2023-08-10 15:40:51) ###
error: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [terms.product_cat.slug] in order to load field data by uninverting the inverted index. Note that this can use significant memory.
recommended_solution: It seems you saved a post without doing a full sync first because <code>terms.product_cat.slug</code> is missing the correct mapping type. <a href="https://transparent-uk.com/wp-admin/admin.php?page=elasticpress-sync">Delete all data and sync</a> to fix the issue.
es_req: POST http://amazonaws.com:9200/post-1/_search
request_id: d794b6fa5ae04400a8de277ccaabc34c
query_time: 9.8168849945068
result: array (
  'error' => 
  array (
    'root_cause' => 
    array (
      0 => 
      array (
        'type' => 'illegal_argument_exception',
        'reason' => 'Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [terms.product_cat.slug] in order to load field data by uninverting the inverted index. Note that this can use significant memory.',
      ),
    ),
    'type' => 'search_phase_execution_exception',
    'reason' => 'all shards failed',
    'phase' => 'query',
    'grouped' => true,
    'failed_shards' => 
    array (
      0 => 
      array (
        'shard' => 0,
        'index' => 'transparentukcom-post-1',
        'node' => 'TO4JT9PYT5qNflqLF4oogA',
        'reason' => 
        array (
          'type' => 'illegal_argument_exception',
          'reason' => 'Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [terms.product_cat.slug] in order to load field data by uninverting the inverted index. Note that this can use significant memory.',
        ),
      ),
    ),
    'caused_by' => 
    array (
      'type' => 'illegal_argument_exception',
      'reason' => 'Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [terms.product_cat.slug] in order to load field data by uninverting the inverted index. Note that this can use significant memory.',
      'caused_by' => 
      array (
        'type' => 'illegal_argument_exception',
        'reason' => 'Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [terms.product_cat.slug] in order to load field data by uninverting the inverted index. Note that this can use significant memory.',
      ),
    ),
  ),
  'status' => 400,
)

### https://transparent-uk.com/?s=FD-W-2-S2401&post_type=product (2023-08-10 15:40:42) ###
error: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [terms.product_cat.slug] in order to load field data by uninverting the inverted index. Note that this can use significant memory.
recommended_solution: It seems you saved a post without doing a full sync first because <code>terms.product_cat.slug</code> is missing the correct mapping type. <a href="https://transparent-uk.com/wp-admin/admin.php?page=elasticpress-sync">Delete all data and sync</a> to fix the issue.
es_req: POST http://amazonaws.com:9200/post-1/_search
request_id: 77d49c54924340a59f727319ff37de3f
query_time: 22.351980209351

result: array (
  'error' => 
  array (
    'root_cause' => 
    array (
      0 => 
      array (
        'type' => 'illegal_argument_exception',
        'reason' => 'Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [terms.product_cat.slug] in order to load field data by uninverting the inverted index. Note that this can use significant memory.',
      ),
    ),
    'type' => 'search_phase_execution_exception',
    'reason' => 'all shards failed',
    'phase' => 'query',
    'grouped' => true,
    'failed_shards' => 
    array (
      0 => 
      array (
        'shard' => 0,
        'index' => 'transparentukcom-post-1',
        'node' => 'TO4JT9PYT5qNflqLF4oogA',
        'reason' => 
        array (
          'type' => 'illegal_argument_exception',
          'reason' => 'Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [terms.product_cat.slug] in order to load field data by uninverting the inverted index. Note that this can use significant memory.',
        ),
      ),
    ),
    'caused_by' => 
    array (
      'type' => 'illegal_argument_exception',
      'reason' => 'Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [terms.product_cat.slug] in order to load field data by uninverting the inverted index. Note that this can use significant memory.',
      'caused_by' => 
      array (
        'type' => 'illegal_argument_exception',
        'reason' => 'Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [terms.product_cat.slug] in order to load field data by uninverting the inverted index. Note that this can use significant memory.',
      ),
    ),
  ),
  'status' => 400,
)

## WordPress ##

### WordPress Environment ###
wp_version: 6.2.2
home_url: https://transparent-uk.com
site_url: https://transparent-uk.com
is_multisite: false
theme: astra-child (1.0.0)
parent_theme: Astra (4.1.6)
plugins: Advanced Shipment Tracking for WooCommerce (3.5.3), Clever Mega Menu for Elementor (1.1.2), Debug Bar (1.1.4), Debug Bar ElasticPress (3.0.0), ElasticPress (4.6.1), Elementor (3.14.1), Elementor Pro (3.14.1), External Images (2.91), FluentSMTP (2.2.5), GTM4WP (1.16.2), HubSpot All-In-One Marketing - Forms, Popups, Live Chat (10.1.30), MWB HubSpot for WooCommerce (1.4.8), Paid On Results - WooCommerce (3.0.3), PDF Invoices & Packing Slips for WooCommerce (3.5.6), PDF Invoices & Packing Slips for WooCommerce - Premium Templates (2.20.2), Premmerce Product Filter for WooCommerce (3.7.2), Purchase Orders for WooCommerce (1.8.3), Rvvup for WooCommerce (1.20230705160521.main.d5d4a371), S3 Uploads (2.1.0-RC2), Scalability Pro (5.09), Shipping Method Description for WooCommerce (1.2.6), Trustpilot-reviews (2.5.925), W3 Total Cache (2.4.0), WooCommerce (7.8.2), WooCommerce EU VAT Assistant (2.1.2.230718), WooCommerce PayPal Payments (2.2.0), WooCommerce PDF Invoices & Packing Slips Professional (2.9.2), WooCommerce Split Orders (1.6.8), Wordfence Security (7.10.2), WP All Export - WooCommerce Export Add-On Pro (1.0.6), WP All Export Pro (1.8.3), WP All Import - WooCommerce Import Add-On Pro (3.3.4) and WP All Import Pro (4.8.0)
revisions: all

### Server Environment ###
php_version: 7.4.28
memory_limit: 40M
timeout: 60

## Indexable Content ##

### Transparent &mdash; https://transparent-uk.com ###
post_count: 18
page_count: 32
e-landing-page_count: 33
elementor_library_count: 71
product_count: 89,507
post-type-template_count: 0
cmm4e_menu_count: 18
post_meta_keys: 1
page_meta_keys: 15
e-landing-page_meta_keys: 7
elementor_library_meta_keys: 7
product_meta_keys: 46
post-type-template_meta_keys: 0
cmm4e_menu_meta_keys: 8
total-all-post-types: 53
distinct-meta-keys: _thumbnail_id, layout, sidebar, footer, header_title_bar, header_transparency, external_image_url, eigallery, site-post-title, site-sidebar-layout, site-content-layout, ast-breadcrumbs-content, theme-transparent-header-meta, ast-title-bar-display, ast-featured-img, ei_converted, _backorders, _downloadable, _manage_stock, _price, _product_attributes, _product_image_gallery, _product_version, _regular_price, _sku, _sold_individually, _stock, _stock_status, _tax_class, _tax_status, _virtual, _wc_average_rating, cost_price, hubwoo_ecomm_pro_id, supplierid, total_sales, ep_exclude_from_search, _crosssell_ids, _sale_price, onsale, _weight, eili, ei, wpmm_postgrid_views, _height, _length, _width, _upsell_ids, _visibility, hubwoo_ecomm_invalid_pro, hubwoo_product_synced, cmm4e_menu_id and cmm4e_menu_item_id

## ElasticPress ##

### Settings ###
host: http://eu-west-2.compute.amazonaws.com:9200
index_prefix: 
language: en_GB
per_page: 50
network_active: false

### Timeouts ###
request_timeout: 5
index_document_timeout: 15
bulk_request_timeout: 30

## Elasticsearch Indices ##

### transparentukcom-post-1 ###
health: yellow
status: open
index: transparentukcom-post-1
uuid: NfZbo9TZRX-QQb2vC6Mnlw
pri: 1
rep: 1
docs.count: 46
docs.deleted: 3
store.size: 1.1mb
pri.store.size: 1.1mb
total_fields_limit: 

## Last Sync ##

### 2023/08/03 2:12:38 pm ###
method: WP Dashboard
is_full_sync: Yes
end_date_time: 2023/08/08 10:41:22 am
total_time: 20 hours, 28 minutes, 44 seconds
total: 89659
synced: 89690
skipped: 0
failed: 10
errors: array (
  0 => '179628 (Post): [illegal_argument_exception] Limit of total fields [1000] has been exceeded',
  1 => '179628 (Post): [illegal_argument_exception] Limit of total fields [1000] has been exceeded',
  2 => '179628 (Post): [illegal_argument_exception] Limit of total fields [1000] has been exceeded',
  3 => '179628 (Post): [illegal_argument_exception] Limit of total fields [1000] has been exceeded',
  4 => '179628 (Post): [illegal_argument_exception] Limit of total fields [1000] has been exceeded',
  5 => '179628 (Post): [illegal_argument_exception] Limit of total fields [1000] has been exceeded',
  6 => '179628 (Post): [illegal_argument_exception] Limit of total fields [1000] has been exceeded',
  7 => '179628 (Post): [illegal_argument_exception] Limit of total fields [1000] has been exceeded',
  8 => '179628 (Post): [illegal_argument_exception] Limit of total fields [1000] has been exceeded',
  9 => '179628 (Post): [illegal_argument_exception] Limit of total fields [1000] has been exceeded',
)

## Feature Settings ##

### Custom Search Results ###
active: true
force_inactive: false

### Facets ###
active: true
force_inactive: false
match_type: all

### Post Search ###
active: true
decaying_enabled: 0
force_inactive: false
highlight_enabled: 0
highlight_excerpt: 0
highlight_tag: mark
synonyms_editor_mode: simple

### Related Posts ###
active: true
force_inactive: false

### WooCommerce ###
active: true
force_inactive: false
orders: 0

felipeelia commented 1 year ago

Hi @TorlockC, for some reason ElasticPress is not able to send the mapping for your website. That error [illegal_argument_exception] Limit of total fields [1000] has been exceeded indicates it is using the default 1,000 limit instead of the 5,000 limit EP sends. Also, post_title.sortable should be recognized as a field.

To better debug it you'll need WP-CLI, and I recommend you split the full sync into smaller steps:

Delete the index
Send the new mapping
Send the content

For step #1, run wp elasticpress delete-index --index-name=transparentukcom-post-1. This will delete the index. Run wp elasticpress get-cluster-indices to double-check -- you should see no indices listed.

For step #2, run wp elasticpress put-mapping. If you see any errors in the output of this command, stop and share it here. If that doesn't output any errors, I still recommend you double-check it:

Run wp elasticpress get-cluster-indices to get the index name
Run wp elasticpress get-index-settings <index-name> and check if index.mapping.total_fields.limit is 5,000

If the above looks correct, you can then sync the content with wp elasticpress sync --show-errors

github-actions[bot] commented 1 year ago

This issue has been automatically closed because there has been no response to our request for more information in the past 3 days. With only the information that is currently available, we are unable to take further action on this ticket. Please reach out if you have found or find the answers we need so that we can investigate further. When the information is ready, you can re-open this ticket to share it with us.

TorlockC commented 1 year ago

Hi @felipeelia,

Thanks for the help.

I have run the above in the sequence suggested and got no errors for the mapping. In fact, it appears that the correct 5,000 limit for index.mapping.total_fields.limit is applied when I run wp elasticpress get-index-settings.

However, after a few hours of indexing index.mapping.total_fields.limit reverts back to 1,000 again and the same error comes up: [illegal_argument_exception] Limit of total fields [1000] has been exceeded.

I did attempt to manually update this field directly on my ES server with: curl -H 'Content-Type:application/json' -X PUT http://localhost:9200/indexname/_settings -d '{"index.mapping.total_fields.limit":5000}'

This does allow me to continue the reindex but again, once it's 100% complete I get the error(s) from my original comment:

Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [terms.pa_10g-support.slug] in order to load field data by uninverting the inverted index. Note that this can use significant memory.

Any idea why it would be reverting on the ES server as when I first downloaded and installed EP it seemed to work fine for months so not sure if I'm missing some vital bit of information here or have misconfigured somehow?

felipeelia commented 1 year ago

@TorlockC you can think about index mappings as database table schemas. The wp elasticpress put-mapping command will delete your index and recreate it with the correct mapping. The wp elasticpress get-index-settings command will only read that index setting, it won't change it.

The fact that it is changing after a few hours might indicate that you have something going wrong with your infrastructure: likely the index is being deleted for some reason (ElasticPress would not do that without a full sync or a WP-CLI command), and gets recreated in the wrong way (the 1,000 limit indicates an index was recreated "automatically", without the proper mapping being sent.)

Unfortunately, as it is something related to your infrastructure, it is beyond our support here.

10up / ElasticPress

Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. #3582

Describe your question

Code of Conduct