[Enhancement] better error handling

[GopherJ]

Describe the bug A clear and concise description of what the bug is.

bulk request if failling silently, curl works but bulk request fails and gave nothing

To Reproduce Steps to reproduce the behavior:

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS: [e.g. Windows 10 Pro]
• rustc version [e.g. rustc --version]

Additional context Add any other context about the problem here.

[russcam]

There's not enough information provided to be begin looking at this.

@GopherJ, please can you add a small reproducible example that clearly demonstrates the issue you're seeing? Please also provide details about the Elasticsearch cluster you're running against.

[GopherJ]

@russcam Yes I'm trying to do so. Actually we had a problem on production, we are using elasticsearch 7.7.1, and elasticsearch client 7.7 alpha.

The problem is, our application on production server fails silently but on local all the things work.

I'm stilly trying to reproduce and understand the problem

[GopherJ]

@russcam Hi, finally I tracked down the error, could you see if you can help.

This error hasn't been reported when I use elasticsearch::bulk but once switch to elasticsearch::index it shows up.

 faild to save: Elastic Error: `Error { kind: Http(reqwest::Error { kind: Status(400), url: "http://localhost:9200/alerts.bd1459f8d1b44aec8acedf059a1b53ed/_doc/7901124001906198514" }) }`

[GopherJ]

@russcam Hi, please check: https://github.com/elastic/elasticsearch/issues/58398#issuecomment-648040065

Do you have any idea on it? Have I made an error in my mappings?

Even I made an error in mappings, the strange thing here is when I use elasticsearch::Bulk it doesn't give me an error message.

I found this error by switching to Index

[russcam]

@GopherJ with a bulk request, you need to check the response body to understand if there have been any errors. The HTTP status code of the response does not signal if any of the bulk operations have failed and relates only to the overall bulk request. The errors property on the top level object indicates if any of the operations did not complete successfully. A 400 response looks like there is an error with what is being sent.

[GopherJ]

@russcam How can I have better error reporting, this error still happens on our production server. We have two servers:

• production
• pre-production

They have the same version of elasticsearch(v7.7.1) and same commit hash of software.

The last week when this error happens on production server, we have spent four days to debug but still nothing found. It just cannot save some type of data.

Then we changed DNS because pre-production works fine, but this week the problem happens again...

With this error message:

 faild to save: Elastic Error: `Error { kind: Http(reqwest::Error { kind: Status(400), url: "http://localhost:9200/alerts.bd1459f8d1b44aec8acedf059a1b53ed/_doc/7901124001906198514" }) }`

I really don't know what to do

[GopherJ]

Our mappings:

{
    "index_pattern": ["alerts.*"],
    "priority": 1,
    "template": {
        "settings": {
            "number_of_shards": 5,
            "number_of_replicas": 1,
            "max_result_window": 100000
        },
        "mappings": {
            "properties": {
                "client_id": {
                    "type": "keyword"
                },
                "timestamp": {
                    "type": "date"
                },
                "label": {
                    "type": "keyword"
                },
                "label_detected": {
                    "type": "keyword"
                },
                "location": {
                    "type": "geo_point"
                },
                "rssi": {
                    "type": "integer"
                },
                "distance": {
                    "type": "integer"
                },
                "duration": {
                    "type": "integer"
                },
                "tags": {
                    "type": "text",
                    "fields": {
                        "keyword": {
                            "ignore_above": 256,
                            "type": "keyword"
                        }
                    }
                },
                "alert_level": {
                    "type": "integer"
                },
                "source": {
                    "properties": {
                        "ty": {
                            "type": "keyword"
                        },
                        "mac": {
                            "type": "keyword"
                        }
                    }
                }
            }
        }
    }
}

Our script to PUT mapping:

curl -X PUT "localhost:9200/_template/events*" -H 'Content-Type: application/json' -d @mappings.json

And the data we sent to elasticsearch:

1. first type

...
source: {
  type: "mobile"
}
...

2. second type

...
source: {
  type: "gateway",
  mac: "xxxxxxxxxxxx"
}
...

Also when we send data to elasticsearch, sometimes we do have the field duration, sometimes we don't, we add this field later when the calculation finishes

[russcam]

@GopherJ It would probably be better to ask this question on the discuss forums to begin; more people visit and answer questions on the forum than this repository and it doesn't look to be a bug with the client at this stage.

How can I have better error reporting, this error still happens on our production server. With this error message: faild to save: Elastic Error: Error { kind: Http(reqwest::Error { kind: Status(400), url: "http://localhost:9200/alerts.bd1459f8d1b44aec8acedf059a1b53ed/_doc/7901124001906198514" }) }

A HTTP 400 response indicates a problem with what is being sent to the server. The body of the response will contain more details in this case, so can be deserialized to a serde_json::Value and the properties read from it.

For your mappings

                "source": {
                    "properties": {
                        "ty": {
                            "type": "keyword"
                        },
                        "mac": {
                            "type": "keyword"
                        }
                    }
                }

source has a ty property, but documents are being sent with a "type" property

source: {
  type: "mobile"
}

[GopherJ]

hi @russcam sorry it's a typo there. We discovered finally it's because geo_point has been too big due to settings problem on UI side.

However I do think elastic-rs should have better error handling. With the above message，it's really hard to understand the problem.