Random errors when bulk-updating document objects via painless script

[consulthys]

Elasticsearch version (bin/elasticsearch --version): 7.6.1 on Elastic Cloud

Plugins installed: N/A

JVM version (java -version): Elastic Cloud

OS version (uname -a if on a Unix-like system): Elastic Cloud

Description of the problem including expected versus actual behavior:

I'm getting inconsistent and sporadic errors while updating documents (via script) through the _bulk API. The issue only happens when trying to update objects (i.e. JSON hash) inside the document.

Steps to reproduce:

Here is a way to easily reproduce the problem.

1. First create a sample index+document

POST test/_doc/1
{
  "hash": {}
}

2. Create a script to be used in the bulk update

The script simply replaces the existing hash field with whatever it receives inside the params and adds another field to that hash:

POST _scripts/add-data
{
  "script": {
    "lang": "painless",
    "source": """
    ctx._source.hash = params;
    ctx._source.hash.additionalField = 32;
    """
  }
}

3. Try to bulk update the document

POST test/_doc/_bulk
{"update":{"_id": "1"}}
{"script": {"id":"add-data", "params":{"addedField": 44}}}

Now the thing is that sometimes the response is OK

{
  "took" : 427,
  "errors" : false,
  "items" : [
    {
      "update" : {
        "_index" : "test",
        "_type" : "_doc",
        "_id" : "1",
        "_version" : 2,
        "result" : "updated",
        "_shards" : {
          "total" : 2,
          "successful" : 2,
          "failed" : 0
        },
        "_seq_no" : 1,
        "_primary_term" : 1,
        "status" : 200
      }
    }
  ]
}

and sometimes it yields this:

{
  "took" : 7,
  "errors" : true,
  "items" : [
    {
      "update" : {
        "_index" : "test",
        "_type" : "_doc",
        "_id" : "1",
        "status" : 400,
        "error" : {
          "type" : "illegal_argument_exception",
          "reason" : "failed to execute script",
          "caused_by" : {
            "type" : "script_exception",
            "reason" : "runtime error",
            "script_stack" : [
              "java.base/java.util.Collections$UnmodifiableMap.put(Collections.java:1457)",
              "ctx._source.hash.additionalField = 32;\n    ",
              "                ^---- HERE"
            ],
            "script" : "add-data",
            "lang" : "painless",
            "caused_by" : {
              "type" : "unsupported_operation_exception",
              "reason" : null
            }
          }
        }
      }
    }
  ]
}

Now I know that params is not modifiable and I'm trying to add a field to it. The error is pretty clear. But I would expect that it always fails, not randomly.

Also worth noting that it makes no difference whether the index has one primary shard or more.

[consulthys]

@dimitris-athanasiou I'm not sure how this fixes the issue I've reported above... Are you sure about the issue number ? Wasn't it #56419 ?

[hendrikmuhs]

@consulthys I think you are right, this bug has been accidentally closed via a commit message

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Scripting)

[stu-elastic]

This should always fail, we'll try to reproduce.

[consulthys]

@stu-elastic Note that I can still reproduce this in 8.13.3