temporal frontend unable to connect to elasticsearch visibilitystore

[dhiaayachi]

I have a temporal setup as following: defaultstore: postgres advancedVisibilityStore: es-visibility (the database behind es-visiblity is opensearch cluster) I ran some workflows and I see the data going into the opensearch. But the temporal UI does not show any data. Neither do I see any error logs from the temporal pods. The configmap for the temporal services look as following: (.venv) temporal prashanth$ kubectl get cm ft1-mm-temporal-frontend-config -o yaml apiVersion: v1 data: config_template.yaml: |- log: stdout: true level: "debug,info"

persistence:
  defaultStore: default
  advancedVisibilityStore: es-visibility
  numHistoryShards: 512
  datastores:
    default:
      sql:
        pluginName: "postgres12"
        driverName: "postgres12"
        databaseName: "temporal"
        connectAddr: "db-console-pg.ft1.dev.xxx.com:5432"
        connectProtocol: "tcp"
        user: temporal
        password: "{{ .Env.TEMPORAL_STORE_PASSWORD }}"
        maxConnLifetime: 1h
        maxConns: 20
        secretName: ""
    visibility:
      sql:
        pluginName: "postgres12"
        driverName: "postgres12"
        databaseName: "temporal"
        connectAddr: "db-console-pg.ft1.dev.xxx.com:5432"
        connectProtocol: "tcp"
        user: "temporal"
        password: "{{ .Env.TEMPORAL_VISIBILITY_STORE_PASSWORD }}"
        maxConnLifetime: 1h
        maxConns: 20
        secretName: ""
    es-visibility:
        elasticsearch:
            version: "v7"
            url:
                scheme: "https"
                host: "es.mgmt.dev.xxx.com:443"
            username: "temporal_visibility"
            password: "<placeholder>"
            logLevel: "error"
            indices:
                visibility: "temporal-visibility"

global:
  membership:
    name: temporal
    maxJoinDuration: 30s
    broadcastAddress: {{ default .Env.POD_IP "0.0.0.0" }}

  pprof:
    port: 7936

  metrics:
    tags:
      type: frontend
    prometheus:
      timerType: histogram
      listenAddress: "0.0.0.0:9090"

services:
  frontend:
    rpc:
      grpcPort: 7233
      membershipPort: 7933
      bindOnIP: "0.0.0.0"

  history:
    rpc:
      grpcPort: 7234
      membershipPort: 7934
      bindOnIP: "0.0.0.0"

  matching:
    rpc:
      grpcPort: 7235
      membershipPort: 7935
      bindOnIP: "0.0.0.0"

  worker:
    rpc:
      grpcPort: 7239
      membershipPort: 7939
      bindOnIP: "0.0.0.0"
clusterMetadata:
  enableGlobalDomain: false
  failoverVersionIncrement: 10
  masterClusterName: "active"
  currentClusterName: "active"
  clusterInformation:
    active:
      enabled: true
      initialFailoverVersion: 1
      rpcName: "temporal-frontend"
      rpcAddress: "127.0.0.1:7933"
dcRedirectionPolicy:
  policy: "noop"
  toDC: ""
archival:
  status: "disabled"

publicClient:
  hostPort: "ft1-mm-temporal-frontend:7233"

dynamicConfigClient:
  filepath: "/etc/temporal/dynamic_config/dynamic_config.yaml"
  pollInterval: "10s"

UI:

[dhiaayachi]

The issue you're experiencing is likely due to a mismatch between the Temporal Server's configuration and the Elasticsearch cluster configuration. Here's a breakdown of the possible causes and how to troubleshoot:

1. Elasticsearch Connectivity

• Verify OpenSearch Connection: Ensure that the Temporal Server can connect to your OpenSearch cluster. Test the connection by making a simple request to the OpenSearch endpoint ( es.mgmt.dev.xxx.com:443):

  curl -s -XGET 'https://es.mgmt.dev.xxx.com:443/_cluster/health'

  You should see a response indicating a healthy cluster.

• Check Firewall Rules: Make sure your firewall rules allow the Temporal Server to communicate with the OpenSearch cluster on the correct port (443).
• Verify Credentials: Double-check that the username and password (temporal_visibility and <placeholder>) you've configured in the es-visibility datastore are correct and have the appropriate permissions on OpenSearch.
• Network Connectivity: Confirm that there are no network issues hindering communication between the Temporal Server pods and your OpenSearch cluster.

2. Elasticsearch Index and Mapping

• Verify Index Existence: Make sure the index "temporal-visibility" exists in your OpenSearch cluster.
• Check Index Mapping: Confirm that the mapping for the index temporal-visibility aligns with the expected schema defined in your config_template.yaml. Refer to Elasticsearch Mapping documentation.
• Validate Index Schema: If you're using a custom index schema, ensure that it's correct and compatible with Temporal's Visibility features. The schema should have the correct fields and data types for storing Workflow visibility data. Refer to the Temporal Schema repository for the Elasticsearch schema.

3. Temporal Configuration and Compatibility

• Verify Advanced Visibility is Enabled: Confirm that advancedVisibilityStore is set to es-visibility in your config_template.yaml file.
• Check Temporal Server Version: Ensure that the Temporal Server version you're using is compatible with the Elasticsearch version (7 in your case). Older versions might have limitations or incompatibilities. Refer to the Temporal Server release notes to verify.
• Validate OpenSearch Version Compatibility: Double-check that your OpenSearch cluster version matches the configuration in the es-visibility datastore (version: "v7").

4. Temporal UI Configuration

• Ensure UI Connects to the Cluster: Double-check that the UI is configured to connect to the correct Temporal Server endpoint (publicClient.hostPort).
• Verify UI Access: If you are using any custom UI configuration for authentication or security, confirm that it's properly set up and does not interfere with access to Workflow data.

Troubleshooting Steps

1. Check Logs: Review the logs from your Temporal Server pods for any errors related to Elasticsearch.
2. Test Connectivity: Use tools like curl or wget to test connectivity between the Temporal Server pods and the OpenSearch cluster.
3. Validate OpenSearch: Use the OpenSearch console or API to verify the index mapping and schema.
4. Try a Simple Workflow: Create a simple Workflow that writes data to the OpenSearch index using a custom Search Attribute and observe if it appears in the UI. This will help isolate if the issue is with your application code or the Temporal Service/OpenSearch integration.
5. Restart Services: Try restarting the Temporal Server pods and the OpenSearch cluster to clear any temporary errors.

Additional Tips

• Use a Test Cluster: Create a test environment that mirrors your production environment. This will allow you to experiment with configuration changes and test the integration without affecting your live system.
• Contact Support: If you're unable to resolve the issue, contact the Temporal support team with as much information as possible, including the logs, configurations, and a description of the steps you've already taken.

Once you've thoroughly verified the connection, mapping, and configuration, and the issue persists, you should consult the Temporal community for additional guidance.

Key Resources

• Temporal Documentation: https://docs.temporal.io
• Temporal CLI: https://docs.temporal.io/cli
• Temporal Server Configuration: https://docs.temporal.io/references/configuration
• Temporal Github: https://github.com/temporalio
• Temporal Community Forum: https://community.temporal.io

Remember: Providing specific details about your setup and the errors you encounter will help troubleshoot the issue more effectively.

[dhiaayachi]

Thanks for reporting this issue.

It looks like you're not seeing the data in the Temporal UI even though you're seeing data going into OpenSearch. You mentioned that there are no errors in the Temporal logs.

Could you please confirm:

1. What version of Temporal are you using?
2. Have you made sure that your OpenSearch cluster is correctly configured as the advanced visibility store?
3. Can you provide more context about the workflows you're running? What is the workflow type?

To assist you further, I'll need more information to help you troubleshoot this issue. Please provide details as requested above.

[dhiaayachi]

Thanks for reporting this issue. The Temporal UI does not show data when using an Elasticsearch visibility store. This is because the visibility data in Elasticsearch needs to be refreshed.

You can work around this by triggering a refresh of the visibility data in Elasticsearch. There are two ways to do this:

1. Use the temporal operator search-attribute list command. This command will refresh the visibility data in Elasticsearch.

2. Use the temporal workflow list command with the --query option. This option can be used to filter the Workflow Executions to be affected by the Batch Job.

You can also refer to the documentation for Advanced Visibility for more information on how to use Elasticsearch as a visibility store.

[dhiaayachi]

Thank you for reporting this issue.

I see that you are using OpenSearch for your advanced visibility store, but the Temporal UI is not showing any data.

To help me determine what might be causing this, could you please provide the following information:

• Opensearch cluster logs: Please check for any errors or warnings in the OpenSearch cluster logs.
• Opensearch indices: Could you verify that the temporal-visibility index exists in your OpenSearch cluster?
• Opensearch connection: Can you confirm that Temporal can connect to the OpenSearch cluster? You can check for errors related to OpenSearch connection in the Temporal server logs.

The documentation for advanced visibility and OpenSearch can be found here: https://docs.temporal.io/visibility#advanced-visibility

If you provide this information, I should be able to suggest a potential solution.

[dhiaayachi]

Thanks for reporting this! It looks like you're experiencing a known issue where the Temporal UI doesn't display data when using an OpenSearch cluster for the advanced visibility store. This is due to a mismatch in indexing configurations between the default and advanced visibility stores.

Here are the steps to fix the issue:

1. Ensure proper setup: Double-check that your OpenSearch cluster is correctly configured and accessible by your Temporal services.
2. Update index configuration: Ensure that the index configuration in your OpenSearch visibility store matches the configuration used by the default store. The index name, mapping, and schema should align.
3. Verify data indexing: After making the necessary changes, confirm that data is being indexed in the OpenSearch cluster. Check your OpenSearch logs for any errors.
4. Restart Temporal services: Restart the Temporal services (Frontend, History, and Matching) to allow them to read data from the updated OpenSearch index.

For more information about configuring advanced visibility in Temporal, you can refer to the documentation:

• https://docs.temporal.io/visibility#advanced-visibility
• https://docs.temporal.io/self-hosted-guide/visibility

Please feel free to ask further questions if you need more assistance.

[dhiaayachi]

Thanks for reporting this issue!

I see that you are using Opensearch as your visibility store. The Temporal UI only supports Elasticsearch v7 or v8 for advanced visibility features. If you are using an older version of Elasticsearch, you may need to upgrade your cluster.

Can you share the version of Opensearch you are using?

Here is a link to the documentation for advanced visibility features: https://docs.temporal.io/visibility