bulk processor error with Opensearch 2.11

dhiaayachi commented 2 months ago

Expected Behavior

Seamless integration with Opensearch 2.11

Actual Behavior

Getting an error on the console and ui temporal | {"level":"error","ts":"2023-11-09T06:36:44.512Z","msg":"elastic: bulk processor \"visibility-processor\" failed but may retry: elastic: Error 400 (Bad Request): Illegal character ((CTRL-CHAR, code 31)): only regular white space (\\r, \\n, \\t) is allowed between tokens\n at [Source: (byte[])\"\\u001F�\\u0008\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000��Ko�F\\u0017\\u0006��3f�1�~�2�\"; line: 1, column: 2] [type=json_parse_exception]","logging-call-at":"logger.go:48","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/persistence/visibility/store/elasticsearch/client.(*errorLogger).Printf\n\t/home/builder/temporal/common/persistence/visibility/store/elasticsearch/client/logger.go:48\ngithub.com/olivere/elastic/v7.(*Client).errorf\n\t/go/pkg/mod/github.com/olivere/elastic/v7@v7.0.32/client.go:840\ngithub.com/olivere/elastic/v7.(*bulkWorker).commit.func2\n\t/go/pkg/mod/github.com/olivere/elastic/v7@v7.0.32/bulk_processor.go:559\ngithub.com/olivere/elastic/v7.RetryNotify\n\t/go/pkg/mod/github.com/olivere/elastic/v7@v7.0.32/retry.go:51\ngithub.com/olivere/elastic/v7.(*bulkWorker).commit\n\t/go/pkg/mod/github.com/olivere/elastic/v7@v7.0.32/bulk_processor.go:580\ngithub.com/olivere/elastic/v7.(*bulkWorker).work\n\t/go/pkg/mod/github.com/olivere/elastic/v7@v7.0.32/bulk_processor.go:501"}

Steps to Reproduce the Problem

Install Opensearch
Connect temporal to opensearch 2.11 for advanced visibility

Specifications

Version: Opensearch 2.11
Platform: Server 1.2.0 / UI 2.11.0

dhiaayachi commented 1 month ago

Thanks for reporting this issue!

It appears that Opensearch 2.11 is not compatible with the Temporal Visibility store for Elasticsearch. The error you are seeing appears to be related to an invalid JSON parse. This usually indicates that there are some characters in the JSON that Opensearch is not recognizing.

Temporal officially supports Elasticsearch versions 6.8, 7.10, and 8.

We recommend checking with Opensearch if there is any information available on compatibility with Temporal's Elasticsearch Visibility store. You can also reach out to us if you have any other questions or issues.

dhiaayachi commented 1 month ago

Thank you for reporting this issue. We are investigating the issue you are experiencing with OpenSearch 2.11 and Temporal.

It looks like the error message indicates an issue with the JSON parsing while sending visibility data to OpenSearch. This might be due to an incompatible character in the data being sent, possibly coming from the commonpb.Payload.Data.

Please answer the following questions to help us troubleshoot the issue:

What specific version of OpenSearch are you using?
What is the exact command or code snippet you're using to connect Temporal to OpenSearch?
Could you provide a sample of the data being sent to OpenSearch?
Are you using any custom codecs or data converters for handling data between Temporal and OpenSearch?

With this information, we can better understand the situation and find a solution for you.

dhiaayachi commented 1 month ago

Thanks for reporting the issue. The error message indicates that there is an illegal character in the JSON payload sent to OpenSearch. Please check your application code to identify and remove the illegal character.

If the issue persists, please provide the following information to help us diagnose the problem:

The exact JSON payload that is causing the error.
The code snippet where you're creating the JSON payload.

To learn more about configuring OpenSearch as the visibility store, see the following documentation:

https://docs.temporal.io/self-hosted-guide/visibility#elasticsearch

dhiaayachi / temporal