Custom search attributes having hyphen("-") in the name is returning error while filtering Workflows

[dhiaayachi]

I have a self hosted Temporal server setup which uses auroradb and elasticsearch for visibility. I created a custom search attribute named "abc-test". While creation there was no error. I tried using this custom search attribute in one of my workflows which also got created without any error and I was able to see the newly created search attribute in the workflow history. However on trying to query the workflows through UI, I am getting error:

There is an error with filtering Workflows.
invalid query: unable to convert filter expression: unable to convert left side of "abc - test = 'xyz'": invalid expression: must be a column name but was *sqlparser.BinaryExpr

Expected Behavior

It should not return any error.

Actual Behavior

It seems that although there is no restriction on using hyphen it is somewhat acting as a delimeter and inserting spaces while filtering. If we are not using hyphen in the name then it works as expected.

Steps to Reproduce the Problem

1. On a self hosted server, create a custom search attribute "abc-test".
2. Use the newly created search attribute in any workflow execution
3. On UI, while querying, try to query the workflow using the search attribute as a filter.

Specifications

• Version: [1.22.6]
• Platform: All

Slack link: https://temporalio.slack.com/archives/CTRCR8RBP/p1710752031783399

[dhiaayachi]

This is an issue with the Temporal Server's handling of hyphens in Search Attribute names.

Analysis of the Problem

You're experiencing an issue with the Temporal Server's handling of hyphens (-) within custom Search Attribute names. Although Temporal allows creating custom attributes with hyphens, the server's query parser seems to treat them as delimiters, causing issues when querying workflows using the UI.

This behavior leads to an "invalid query" error because the query parser interprets the hyphen as a separator, attempting to parse the search attribute name as a complex expression rather than a simple column name.

Suggested Solution

As of now, the Temporal Server's query parser is not able to properly handle hyphens within Search Attribute names. To avoid this issue, consider these solutions:

• Use underscores instead of hyphens: Replace hyphens in your custom Search Attribute names with underscores (_). For example, rename "abc-test" to "abc_test". This will ensure that the query parser interprets the name correctly.
• Avoid using hyphens in Search Attribute names: While this might be less desirable for naming conventions, it's a temporary workaround until the issue is fixed.

References

• Search Attributes: https://docs.temporal.io/visibility#search-attribute
• Temporal Server Versions: https://github.com/temporalio/temporal/releases

Contributing to the Solution

You can contribute to the solution by:

• Reporting the issue: Ensure you have filed a bug report with Temporal. If one already exists, add your details to it. https://github.com/temporalio/temporal/issues
• Providing feedback: Share your experience and observations with the Temporal community. https://community.temporal.io/

Temporary Workaround

As a temporary workaround, you can update your code to use underscores instead of hyphens in your Search Attribute names. This will allow you to query workflows using the UI without encountering the "invalid query" error.

Temporal is aware of this issue and is actively working on a fix. However, until the fix is released, you will need to use one of the workarounds described above.

[dhiaayachi]

Thank you for reporting this issue. It seems that custom search attributes cannot have hyphens in their names. As Temporal UI uses them as delimiters, it is failing to recognize the custom search attribute "abc-test" as a valid attribute name.

You can work around this issue by:

• Using underscores instead of hyphens: You can rename your custom search attribute to "abc_test" and use it in your workflows and queries.

Please let me know if you have any further questions.

[dhiaayachi]

Thank you for reporting this issue. The error invalid query: unable to convert filter expression: unable to convert left side of "abc - test = 'xyz'": invalid expression: must be a column name but was *sqlparser.BinaryExpr in Temporal's UI while querying workflows, happens when a custom search attribute name contains a hyphen (-). The hyphen seems to be acting as a delimiter, causing spaces to be inserted during filtering.

This issue is not documented, however, there is a workaround for this:

• Use an underscore (_) in place of a hyphen (-) in the custom search attribute name.

For example, instead of creating a search attribute called "abc-test", you could create one called "abc_test".

[dhiaayachi]

Thank you for reporting this issue.

The error message you are seeing is caused by a limitation in the current version of Temporal (1.22.6). Hyphens are not supported as part of the custom search attribute names.

The Temporal team is aware of this issue and is actively working on a fix. You can find more information and updates about the issue on this Github issue: https://github.com/temporalio/temporal/issues/4351.

In the meantime, you can work around this issue by using an underscore ( _) instead of a hyphen ( -) when naming your custom search attributes.

If you have any other questions, please feel free to ask.

[dhiaayachi]

Thank you for reporting this issue! This is a known issue with Temporal server. It seems the hyphen is acting as a delimiter in the search attributes. Here is a link to the issue tracker: https://github.com/temporalio/temporal/issues/4908. We are working on a fix. For now, you can work around this by using underscores (_) instead of hyphens (-) in your custom search attribute names. This should avoid the issue and allow you to query your workflows correctly.

[dhiaayachi]

Thanks for reporting this issue. The issue you are seeing is a known limitation. We've documented this issue here. We're working on a fix for this, but in the meantime, you can work around it by using underscores instead of hyphens in your search attribute names.