Support query by close statue for S3 backed archival

[MatthewHou]

Is your feature request related to a problem? Please describe. We noticed that S3 backed archival does not support query by close status. This feature is supported when archival is backed by G Cloud or file store. Proposed Solution

• Option 1: Alter S3 object key structure to include close status A straight forward solution would be to add close status to archival file's S3 object key and incorporate that into the query.
  Current structure: {domain_id}/history/{workflow_id}/{run_id}/ Proposed structure: {domain_id}/history/{close_status}/{workflow_id}/{run_id}/

  • Pros: straight forward and relatively easy to implement
  • Cons: breaks backward compatibility. Records already archived in S3 need to be migrated or query no longer works

• Option 2: Add manually indexing to visibility archival Current S3 archiver implementation builds 2 manual indexes here https://github.com/uber/cadence/blob/e1f509e338babb29f2f1141393c69da8ddfb3034/common/archiver/s3store/visibilityArchiver.go#L152 (Note the secondary index (time range) does not work in current implementation as S3 does not support range query.) We could add another index based on the close status to support this type of query.

  • Pros: more backward compatible. Previously archived records still can't be searched by close status, but they can be queried without this filter
  • Cons: extra data needs to be stored on S3

[longquanzheng]

Another idea: we can have versioning and taking option#1, but people don't need to run migration. It's less work in writing tool and operation for migrating(also some possible downtime).

Proposed structure: {domain_id}/historyV2/{close_status}/{workflow_id}/{run_id}/

[longquanzheng]

@yycptt @andrewjdawson2016

[jontro]

Hi, I think you could still always query the old and new format, then it would still be backwards compatible but all new records would be written with the close status.

[MatthewHou]

Another idea: we can have versioning and taking option#1, but people don't need to run migration. It's less work in writing tool and operation for migrating(also some possible downtime).

Proposed structure: {domain_id}/historyV2/{close_status}/{workflow_id}/{run_id}/

I thought about versioning it too. But users still have to query archival records in 2 separate places (old records will not show up if they query by close statue). I'm also a bit skeptical about how the user experience in the Cadence UI if older archival records need to be queried separately