search

DataDog / temporal-large-payload-codec

HTTP service and accompanying Temporal Payload Codec which allows Temporal clients to automatically persist large payloads outside of workflow histories.
MIT License
33 stars 2 forks source link

Support Nexus (cross-namespace/region) calls #75

Open pedroalb opened 3 weeks ago

pedroalb commented 3 weeks ago

I am a new adopter of Temporal, more specifically Temporal Cloud. My intention is not necessarily to deal with large payloads but to avoid any sensitive/PII data in the payload being pushed to Temporal Cloud. I know I can use payload encryption (I have been using it), but I do not want to deal with key management toil. The internal decision is to look at using pointers for any kind of payload size, so any potential sensitive/PII data mistakenly passed to payload is stored within our realm. However, this implementation doesn't seem to support Nexus cross-region/namespace calls. What are your plans to support it in the near future?

MortadhaTeffaha commented 2 weeks ago

Hello @pedroalb, i'm not familiar with the Nexus cross-region/namespace calls that you're referring to, if you could share more information on that I can tell you if we ever would add it. that being said I think the use case that I got from the issue would be better solved in a different codec. if I understand correctly what you would like to achieve is the following:

when sending/receiving temporal Workflow Argument you would like to detect that PII information has been passed and replace this information with a pointer to an internal data store containing this data.

Temporal support chaining codecs and you could create a seperate Pii-codec that would do exactly that and the resulting payload would be passed to temporal-large-payload-codec to be stored in a blob storage.