PUT and POST endpoints require '---' header line when using yaml

[ycombinator]

Elasticsearch version (bin/elasticsearch --version):

Version: 8.0.0-SNAPSHOT, Build: default/docker/fcbbc7e7967c3adf05b54c08eb6eae18e3b847a0/2020-10-04T09:46:56.752224Z, JVM: 15

Plugins installed: Default

JVM version (java -version):

openjdk version "15" 2020-09-15
OpenJDK Runtime Environment AdoptOpenJDK (build 15+36)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 15+36, mixed mode, sharing)

OS version (uname -a if on a Unix-like system):

Linux 594dbf55013f 4.19.76-linuxkit #1 SMP Tue May 26 11:42:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

Like other Elasticsearch APIs that accept request bodies, the PUT _ingest/pipeline/{id} API is able to accept a YAML document in its request body. However, despite passing it a valid YAML document (according to the YAML spec) and providing the Content-Type: application/yaml request header, the request fails as shown below.

$  curl -H 'Content-Type: application/yaml' -X PUT 'http://localhost:9200/_ingest/pipeline/my-pipeline-id' -d 'processors: []
'

---
error:
  root_cause:
  - type: "not_x_content_exception"
    reason: "Compressor detection can only be called on some xcontent bytes or compressed\
      \ xcontent bytes"
  type: "not_x_content_exception"
  reason: "Compressor detection can only be called on some xcontent bytes or compressed\
    \ xcontent bytes"
status: 500

It turns out that the workaround for this is to provide the --- header in the request body as shown below.

$ curl -H 'Content-Type: application/yaml' -X PUT 'http://localhost:9200/_ingest/pipeline/my-pipeline-id' -d '---
processors: []
'

---
acknowledged: true

Note, however, that other Elasticsearch APIs that also accept request bodies do not require the --- header, as shown below.

curl -H 'Content-Type: application/yaml' 'http://localhost:9200/_search' -d '
query:
  match_none: {}
'

---
took: 1
timed_out: false
_shards:
  total: 1
  successful: 1
  skipped: 0
  failed: 0
hits:
  total:
    value: 0
    relation: "eq"
  max_score: null
  hits: []

So in the interest of a) consistency amongst Elasticsearch APIs and b) adherence to the YAML spec, specifically the section about bare documents, could we make the PUT _ingest/pipeline/{id} API accept bare YAML documents, i.e. ones without the --- header line?

Practically speaking it would help us avoid building exceptions such as https://github.com/elastic/package-spec/pull/54 for code about Ingest Pipelines intended for the PUT _ingest/pipeline/{id} API.

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Ingest)

[HiDAl]

Hello,

Checking out the issue, I've found out that this situation it's not only affecting our endpoint but any endpoint using PUT/POST:

# invalid body
curl -H 'Content-Type: application/yaml' -X PUT 'http://localhost:9200/_template/some-template' -d '
index_patterns:
- testing-index-ingest
version: 20190512
order: 0
settings:
  index:
    default_pipeline: testing-ingest-pipeline
'
---
error:
  root_cause:
  - type: "not_x_content_exception"
    reason: "Compressor detection can only be called on some xcontent bytes or compressed\
      \ xcontent bytes"
  type: "not_x_content_exception"
  reason: "Compressor detection can only be called on some xcontent bytes or compressed\
    \ xcontent bytes"
status: 500

# valid body
curl -H 'Content-Type: application/yaml' -X PUT 'http://localhost:9200/_template/some-template' -d '---
index_patterns:
- testing-index-ingest
version: 20190512
order: 0
settings:
  index:
    default_pipeline: testing-ingest-pipeline
'
---
acknowledged: true

# invalid body
curl -H 'Content-Type: application/yaml' -X POST 'http://localhost:9200/some-index/_doc/1' -d '
quote> field: value
quote> '
---
error:
  root_cause:
  - type: "mapper_parsing_exception"
    reason: "failed to parse"
  type: "mapper_parsing_exception"
  reason: "failed to parse"
  caused_by:
    type: "not_x_content_exception"
    reason: "Compressor detection can only be called on some xcontent bytes or compressed\
      \ xcontent bytes"
status: 400

# valid body
curl -H 'Content-Type: application/yaml' -X POST 'http://localhost:9200/some-index/_doc/1' -d '---
field: value
'
---
_index: "some-index"
_id: "1"
_version: 1
result: "created"
_shards:
  total: 2
  successful: 1
  failed: 0
_seq_no: 0
_primary_term: 1

Due to it looks like a broader issue, I'll reassign the ticket to Core/Infra

thanks

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[tmoschou]

Still an issue with 8.15.0.

$ curl -i -XPUT http://localhost:9200/test-index -H 'content-type: application/yaml' -d '
settings:
  number_of_shards: 1
'
HTTP/1.1 500 Internal Server Error
X-elastic-product: Elasticsearch
content-type: application/yaml
content-length: 333

---
error:
  root_cause:
  - type: "not_x_content_exception"
    reason: "Compressor detection can only be called on some xcontent bytes or compressed\
      \ xcontent bytes"
  type: "not_x_content_exception"
  reason: "Compressor detection can only be called on some xcontent bytes or compressed\
    \ xcontent bytes"
status: 500

and stacktrace

org.elasticsearch.common.compress.NotXContentException: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes
    at org.elasticsearch.server@8.15.0/org.elasticsearch.common.compress.CompressorFactory.compressor(CompressorFactory.java:42)
    at org.elasticsearch.server@8.15.0/org.elasticsearch.common.xcontent.XContentHelper.createParser(XContentHelper.java:121)
    at org.elasticsearch.server@8.15.0/org.elasticsearch.common.xcontent.XContentHelper.parseToType(XContentHelper.java:223)
    at org.elasticsearch.server@8.15.0/org.elasticsearch.common.xcontent.XContentHelper.parseToType(XContentHelper.java:210)
    at org.elasticsearch.server@8.15.0/org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:167)
    at org.elasticsearch.server@8.15.0/org.elasticsearch.rest.action.admin.indices.RestCreateIndexAction.prepareRequest(RestCreateIndexAction.java:116)
    at org.elasticsearch.server@8.15.0/org.elasticsearch.rest.action.admin.indices.RestCreateIndexAction.prepareRequest(RestCreateIndexAction.java:61)
    at org.elasticsearch.server@8.15.0/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:94)
    at org.elasticsearch.server@8.15.0/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:490)
    at org.elasticsearch.server@8.15.0/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:484)
    at org.elasticsearch.security@8.15.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.intercept(SecurityRestFilter.java:69)
    at org.elasticsearch.server@8.15.0/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:484)
    at org.elasticsearch.server@8.15.0/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:648)
    at org.elasticsearch.server@8.15.0/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:339)
    at org.elasticsearch.server@8.15.0/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:487)
    at org.elasticsearch.server@8.15.0/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:583)
    at org.elasticsearch.server@8.15.0/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:460)
    at org.elasticsearch.transport.netty4@8.15.0/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:141)
    at org.elasticsearch.transport.netty4@8.15.0/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:128)
    at io.netty.transport@4.1.109.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
 ...