search

dhiaayachi / temporal

Temporal service
https://docs.temporal.io
MIT License
0 stars 0 forks source link

SecondaryVisbilityStore validation failed #191

Open dhiaayachi opened 2 weeks ago

dhiaayachi commented 2 weeks ago

Expected Behavior

I want to migrate the visibility store from ES to Postgres using Dual Visibility Store feature. Expected behavior to be able to dual write to both ES and Postgres. Temporal config:

  visibilityStore: es-visibility
  secondaryVisibilityStore: pg-visibility
  datastores:
    pg-visibility:
      SQL:
    es-visibility:
      elasticsearch:

Actual Behavior

Upon starting the server, I got the following error persistence config: cannot set secondaryVisibilityStore when visibilityStore is setting elasticsearch datastore

Steps to Reproduce the Problem

  1. Setup the config as above
  2. Start temporal server

Specifications

dhiaayachi commented 2 days ago

The error persistence config: cannot set secondaryVisibilityStore when visibilityStore is setting elasticsearch datastore is a known limitation in Temporal Server versions prior to 1.23. This is because dual Visibility is a new feature introduced in Temporal Server 1.21 and later. You can use dual Visibility with SQL databases like PostgreSQL for advanced Visibility, but not with Elasticsearch.

Here's a breakdown of the issue and a solution:

Problem:

Solution:

  1. Upgrade to Temporal Server 1.23 or later: Dual Visibility with Elasticsearch is supported in Temporal Server 1.23 and later.

  2. Update your configuration: You can then configure your temporal-server.yaml file to use Elasticsearch as the primary Visibility store and PostgreSQL as the secondary Visibility store:

    persistence:
 visibilityStore: es-visibility
 secondaryVisibilityStore: pg-visibility
 datastores:
   pg-visibility:
     SQL: 
       pluginName: "postgres12"
       databaseName: "temporal_visibility"
       connectAddr: "your_postgres_host:5432" # Replace with your PostgreSQL connection details
       connectProtocol: "tcp"
       user: "your_postgres_user"
       password: "your_postgres_password"
       maxConns: 2
       maxIdleConns: 2
       maxConnLifetime: "1h" 
   es-visibility:
     elasticsearch:
       version: "v7" # Or "v8"
       url:
         scheme: "http"
         host: "your_elasticsearch_host:9200" # Replace with your Elasticsearch connection details
       indices:
         visibility: temporal_visibility_v1
         secondary_visibility: temporal_visibility_v1_new

Additional Information:

By upgrading to Temporal Server 1.23 or later and updating your configuration, you can leverage the dual Visibility feature to migrate your Visibility data from Elasticsearch to PostgreSQL.

dhiaayachi commented 2 days ago

Thanks for reporting the issue. This is currently a known limitation with Temporal 1.22.3 and earlier, where dual writing to both ES and Postgres is not supported. You can find more information about dual visibility and the limitation in the Dual Visibility dynamic configuration reference.

You will need to upgrade to Temporal Server v1.21 or later to have dual visibility enabled.

dhiaayachi commented 2 days ago

Thanks for reporting this issue. This error message "persistence config: cannot set secondaryVisibilityStore when visibilityStore is setting elasticsearch datastore" indicates that the Temporal Server does not support setting a secondary Visibility store when the primary Visibility store is Elasticsearch.

The Dual Visibility feature is designed to work with SQL-based Visibility stores (MySQL, PostgreSQL, SQLite). You can't use Elasticsearch as a primary Visibility store and then add a secondary SQL store.

As a workaround, you could try the following:

  1. Use SQL as your primary Visibility store: Switch to using MySQL, PostgreSQL, or SQLite as your primary Visibility store. Then, you can configure Elasticsearch as your secondary store for dual writing.
  2. Migrate to Temporal Cloud: Temporal Cloud supports both Elasticsearch and SQL databases for advanced Visibility features, along with automated failover and 99.99% uptime.

For more information about setting up a Visibility store and advanced Visibility features, refer to the documentation here: https://docs.temporal.io/visibility

dhiaayachi commented 2 days ago

Thanks for reporting the issue. You are encountering this error because the secondaryVisibilityStore setting can only be used with a visibilityStore that uses a SQL datastore, and in your configuration, you are using the elasticsearch datastore for the visibilityStore.

To work around this, you can use a SQL datastore for the visibilityStore and then set the secondaryVisibilityStore to your ES store. For example:

  visibilityStore: pg-visibility
  secondaryVisibilityStore: es-visibility
  datastores:
    pg-visibility:
      SQL:
    es-visibility:
      elasticsearch:

You can find more information on the configuration options in the Temporal Service configuration documentation.

dhiaayachi commented 2 days ago

Thanks for reporting the issue. You are encountering this error because dual Visibility Store feature currently only supports SQL based datastores, not elasticsearch. To migrate your Visibility Store from ES to Postgres, first set up a secondary Postgres Visibility Store. Then enable dual Visibility Store feature by setting secondaryVisibilityStore and system.secondaryVisibilityWritingMode in dynamic config, and write to both stores. Once the data is migrated, you can disable writing to the ES store and then make your Postgres Visibility Store the primary store. This is the suggested workaround.

dhiaayachi commented 1 day ago

Thanks for reporting this issue. It appears you are encountering a known issue where Temporal does not allow setting a secondary Visibility store when using Elasticsearch as the primary. You can work around this by using one of the following supported databases for your visibility store: