[BUG] null_pointer_exception when requesting with search_after on a null field

[TatianaNeuer]

Describe the bug

OpenSearch returns a 500 Internal server error when search_after contains a null field and "track_total_hits:false". This is the error:

{
    "error": {
        "root_cause": [
            {
                "type": "null_pointer_exception",
                "reason": "Cannot read field \"bytes\" because \"other\" is null"
            }
        ],
        "type": "search_phase_execution_exception",
        "reason": "all shards failed",
        "phase": "query",
        "grouped": true,
        "failed_shards": [
            {
                "shard": 0,
                "index": "sample_index_null",
                "node": "l3SwxdKCRpWEexkAj0IVVA",
                "reason": {
                    "type": "null_pointer_exception",
                    "reason": "Cannot read field \"bytes\" because \"other\" is null"
                }
            }
        ],
        "caused_by": {
            "type": "null_pointer_exception",
            "reason": "Cannot read field \"bytes\" because \"other\" is null",
            "caused_by": {
                "type": "null_pointer_exception",
                "reason": "Cannot read field \"bytes\" because \"other\" is null"
            }
        }
    },
    "status": 500
}

Related component

Search

To Reproduce

1. Index some documents with null fields: POST /_bulk

   
   { "index": { "_index": "sample_index_null", "_id": "1" } }
   { "doc": "doc1", "name": "bob"}
   { "index": { "_index": "sample_index_null", "_id": "2" } }
   { "doc": "doc2", "name": null}
   { "index": { "_index": "sample_index_null", "_id": "3" } }
   { "doc": "doc3", "name": null}

2. Search documents:

{ "size": 20, "track_total_hits": false, "sort": [ { "name.keyword": { "order": "desc" } }, { "doc.keyword": { "order": "asc" } } ], "search_after": [ null, "doc2" ] }

3. The response is a 500 Internal server error. The same request but with "track_total_hits:true" returns the correct document with no error.

### Expected behavior

OpenSearch should return the documents.

### Additional Details

**Host/Environment (please complete the following information):**
- OS: Windows 10 with WSL2 and docker
- Version : docker image: opensearchproject/opensearch:2.15.0
- 1 opensearch node run with the following docker compose file:

version: '3' services: opensearch: image: opensearchproject/opensearch:2.15.0 container_name: opensearch environment:

• cluster.name=opensearch-cluster # Name the cluster
• node.name=opensearch # Name the node that will run in this container
• discovery.seed_hosts=opensearch # Nodes to look for when discovering the cluster
• cluster.initial_cluster_manager_nodes=opensearch # Nodes eligibile to serve as cluster manager
• bootstrap.memory_lock=true # Disable JVM heap memory swapping
• "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
• "DISABLE_INSTALL_DEMO_CONFIG=true" # Prevents execution of bundled demo script which installs demo certificates and security configurations to OpenSearch
• "DISABLE_SECURITY_PLUGIN=true" # Disables Security plugin ulimits: memlock: soft: -1 # Set memlock to unlimited (no soft or hard limit) hard: -1 nofile: soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536 hard: 65536 volumes:
• opensearch:/usr/share/opensearch/data # Creates volume called opensearch-data1 and mounts it to the container ports:
• 9200:9200 # REST API
• 9600:9600 # Performance Analyzer networks:
• opensearch-net # All of the containers will join the same Docker bridge network

volumes: opensearch:

networks: opensearch-net:

[dblock]

Looks like a bug. Would you have a moment to try and write a YAML REST test for this?

https://github.com/opensearch-project/OpenSearch/blob/main/TESTING.md#testing-the-rest-layer

[TatianaNeuer]

I tried writing a YAML REST test, heavily inspired from https://github.com/opensearch-project/OpenSearch/blob/main/rest-api-spec/src/main/resources/rest-api-spec/test/search/90_search_after.yml, I did not run the test so I hope the syntax is correct:

"null values":
  - do:
      indices.create:
          index:  test
  - do:
      bulk:
        refresh: true
        index: test
        body: |
          {"index":{}}
          { "doc": "doc1", "name": "bob"}
          {"index":{}}
          { "doc": "doc2", "name": null}
          {"index":{}}
      { "doc": "doc3", "name": null}

  - do:
      search:
        rest_total_hits_as_int: true
        index: test
        body:
          size: 2
      track_total_hits: false
          sort: [{ name.keyword: desc }, { doc.keyword: desc }]

  - match: {hits.total: 3 }
  - length: {hits.hits: 2 }
  - match: {hits.hits.0._index: test }
  - match: {hits.hits.0._source.doc: doc1 }
  - match: {hits.hits.1._index: test }
  - match: {hits.hits.1._source.doc: doc2 }
  - match: {hits.hits.1.sort: [null, "doc2"] }

  - do:
      search:
        rest_total_hits_as_int: true
        index: test
        body:
          size: 1
      track_total_hits: false
          sort: [{ name.keyword: desc }, { doc.keyword: desc }]
      search_after: [null, "doc2"]

  - match: {hits.total: 1 }
  - length: {hits.hits: 1 }
  - match: {hits.hits.0._index: test }
  - match: {hits.hits.0._source.doc: doc3 }
  - match: {hits.hits.1.sort: [null, "doc3"] }

[dblock]

Try running it? :)