`metaClass` subfields for "Elasticsearch Unreachable" events

[vpavlushkov]

Logstash produces some garbage for "Elasticsearch Unreachable" events:

...
      "loggerName": "logstash.outputs.elasticsearch",
      "levelString": "warn",
      "thread": "Ruby-0-Thread-34@[main]>worker1: :1",
      "level": "40",
      "timeMillis": 1521381519666,
      "logEvent": {
        "message": "Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://logs-sandbox:xxxxxx@<instance>.europe-west1.gcp.cloud.es.io:9243/][Manticore::SocketException] Connection reset",
        "url": {
          "metaClass": {
            "metaClass": {
              "metaClass": {
                "error_message": "Elasticsearch Unreachable: [https://logs-sandbox:xxxxxx@<instance>.europe-west1.gcp.cloud.es.io:9243/][Manticore::SocketException] Connection reset",
                "error_class": "LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError",
                "url": "https://logs-sandbox:xxxxxx@<instance>.europe-west1.gcp.cloud.es.io:9243/"
              }
            }
          }
        }
      }
...

Those extra metaClass layers are really unnecessary and, when Logstash logs are getting synced into Elastic, the field type for logEvent.url is getting violated as it is not a URL/string anymore but becomes an object.

For example "restored connection" event is getting logged like that:

...
      "levelString": "warn",
      "loggerName": "logstash.outputs.elasticsearch",
      "logEvent": {
        "message": "Restored connection to ES instance",
        "url": "https://logs-sandbox:xxxxxx@<instance>.europe-west1.gcp.cloud.es.io:9243/"
      },
      "timeMillis": 1521063811404,
      "thread": "Ruby-0-Thread-29: :1",
      "level": "40"
...

• Version: Logstash 6.2.2, chart config is taken from https://github.com/kubernetes/charts/tree/f9826507a13abc719ca7240f9eb40ad9be984b17/incubator/logstash
• Operating System: Dockerized deployment with Kubernetes (v1.9.3-gke.0) Container-Optimized OS from Google (kernel version 4.4.111+)
• Config File (if you have sensitive info, please remove it): o log4j2.properties o logstash.yml
• Sample Data: see description
• Steps to Reproduce: Let the logging pipeline run for a while so that connection to Elasticsearch gets interrupted (and then auto-restored)

[maglo]

same here, seen in version 7.6.2

And we are using JSON logging, which causes this event to end up in the DLQ with:

failed to parse field [logstash.log.log_event.url] of type [keyword] in document with id '1TZEBXQBEZXZq8lu6DHu'. Preview of field's value: '{metaClass={metaClass={metaClass={error_message=Elasticsearch Unreachable: [https://xxx:9200/][Manticore::ClientProtocolException]xxx:9200 failed to respond, error_class=LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, url=https://xxx:9200/}}}}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:1227"}}}}