Extend AttachmentProcessor so that it can read Maester Attachments

[jhorbulyk]

Migration to Use New API Features

Description

Currently, the AttachmentProcessor works with reading from and writing to our steward service. In the longer term, there is a desire to migrate from Steward to Maester. Currently, when storing data in Maester, in general, client side encryption is used. As a result, if we simply swap out Steward URLs with Maester URLs (and the existing Maester encryption logic) then existing components which work with attachments would not be functional. We should add logic such that if the provided URL is a Maester URL, then attachment processor should read it seamlessly.

Due Date

None, but:

• It would be nice to deprecate Steward
• It would nice to be able to have all these changes implemented prior to doing rework on components that handle attachments
• In the future, we may start billing for Maester usage

Also makes sense to work in parallel on https://github.com/elasticio/component-commons-library/issues/41

[A3a3e1]

@jhorbulyk How should we distinguish between Steward and Maester links? Maester URL looks like this: ELASTICIO_OBJECT_STORAGE_URI: 'http://maester-service.platform.svc.cluster.local:3002' (probably is a subject to be changed by the platform team)

Should we parse the URL by any rule or use any kind of UI selector switching between Maester and Steward?

[jhorbulyk]

How should we distinguish between Steward and Maester links? Maester URL looks like this: ELASTICIO_OBJECT_STORAGE_URI: 'http://maester-service.platform.svc.cluster.local:3002' (probably is a subject to be changed by the platform team) Should we parse the URL by any rule or use any kind of UI selector switching between Maester and Steward?

My suggestion would be to parse the URL and see if the URL's domain matches the ELASTICIO_OBJECT_STORAGE_URI env variable. @zuker Do you see any problems with this approach?

[zuker]

I'd suggest a more explicit approach: set some header or property in the message that indicates which storage is used. Relying on a domain is not a good idea since it can be changed at any time.

[jhorbulyk]

Latest thoughts on how to distinguish between Maester and Steward URLs: Add a query string parameter ?attachmentType=maester to all attachments that are created in Maester. This way:

• The flagging of the attachment is explicit and does not make assumptions about potentially changing DNS records
• There is no additional field that an integrator would have to remember to map

At the moment, all attachments are currently created by components themselves apart from incoming attachments on webhooks. (This will have to be transitioned at somepoint in the future.) This means that the logic of AttachmentProcessor also controls the logic of attachment URL creation.

[EmptyInfinity]

In getAttachment there is a parameter responseType, which can't be used along with maester-library, because in maester-library responseType is hardcoded to "stream" https://github.com/elasticio/component-commons-library/blob/master/lib/attachment/AttachmentProcessor.ts#L11 https://github.com/elasticio/maester-client/blob/3.0.0/src/StorageClient.ts#L81 Moreover, ObjectStorage (we use methods from this class) which wraps StorageClient expects "stream" to be returned https://github.com/elasticio/maester-client/blob/3.0.0/src/ObjectStorage.ts#L41