Search attributes registration is lost after successfully adding

[dhiaayachi]

Expected Behavior

The search attributes should never got lost after registering

Actual Behavior

Sometimes it will get lost

Steps to Reproduce the Problem

This is the script that I used to register search attributes automatically after service is up running in docker. It happens occasionally that one of the three (randomly) will be lost after the CLI command show up the registering is succeeded.

E.g. in this integ test failure: https://github.com/indeedeng/iwf/actions/runs/4682724673/jobs/8297177058

In dump docker logs you can see:

 You are about to add search attributes [IwfWorkflowType:Keyword]. Continue? Y/N yes
  Search attributes have been added
  You are about to add search attributes [IwfGlobalWorkflowVersion:Int]. Continue? Y/N yes
  Search attributes have been added
  You are about to add search attributes [IwfExecutingStateIds:Keyword]. Continue? Y/N yes
  Search attributes have been added

And then test fails:

ERROR   loggerimpl/logger.go:135    encounter error for API {"error": "search attribute IwfWorkflowType is not defined", "logging-call-at": "service.go:389"}

Specifications

• Version: 1.20.1 , 1.19.1
• Platform: docker

[dhiaayachi]

Thank you for reporting this issue.

It seems like the search attributes you are trying to add are occasionally getting lost after registering.

From the provided information, it is difficult to determine the exact cause. Could you please provide more information about your environment:

• The specific Temporal version you are using.
• The Docker image you are using to run Temporal.
• The contents of the Dockerfile.
• The specific command you are using to run your service in Docker.

In the meantime, you can try these workarounds to avoid losing search attributes:

• Add search attributes during workflow registration: Instead of registering them after the service is up and running, you can register them during workflow registration using the WorkflowOptions struct.
• Manually verify search attributes: After registering search attributes, manually verify that they are correctly defined using the tctl describe-namespace command.

If these workarounds don't solve the issue, please provide the requested information and we can investigate further.

[dhiaayachi]

Thank you for reporting this issue. I understand that you are experiencing a problem where search attributes sometimes get lost after registering them.

Based on the information you provided, it seems like a potential issue with the search attribute registration process.

To better understand the issue, could you please provide the following information:

• What is the version of Temporal Server you are using?
• Can you share the logs from the Temporal Server when this issue occurs?
• Are you using the default Temporal Server configuration or have you made any customizations?

In the meantime, you can try restarting the Temporal Server to see if that resolves the issue. If the issue persists, please provide the requested information so I can help you further.

[dhiaayachi]

Thanks for reporting this issue. I'm sorry you're encountering this. I've reviewed the logs from the GitHub Actions test you linked and the information you provided about the issue occurring randomly.

Could you please provide more information about the specific steps you are taking to register the search attributes in your Docker environment? For example, are you using the Temporal CLI or a custom script?

Additionally, please share the following information to help us diagnose the problem:

• Docker image: The Docker image being used to run Temporal.
• Temporal configuration: The configuration file used to start Temporal in your Docker environment.
• Script for registering search attributes: The script used to automatically register the search attributes after the Temporal service is up and running.

This information will help me understand the context better and assist you in finding a solution.