Cloud Storage Uploads for Contact Avatar Images & Message Attachments

[steffh]

## User Story As an Airy developer trying to be able to keep access to image and media files with expiring urls, I want to be able to specify the cloud storage provider of my choice (e.g. AWS S3, Google Cloud Storage, Azure Storage, local folder on my server) so that the Airy instance can upload the relevant files I receive on my Airy instance asynchronously to the configured cloud storage provider and replace the relevant urls, so that I will stay in control of these files and keep access to them

---

## Acceptance Criteria

• GIVEN A cloud storage provider is specified in my Airy configuration files / environment variables including all necessary credentials to access and write to a bucket/folder specified for the relevant cloud storage provider,
• WHEN a new message arrives via a webhook for any source supporting contact / user profile pictures and for a validly connected channel from an unknown contact in a new conversation that includes an image url for a profile picture / avatar image of such contact; or
• WHEN an asynchronous user profile lookup request (e.g. for Facebook https://github.com/airyhq/airy/issues/482) in response to a new message from an unknown contact in a new conversation contains an image url for a profile picture / avatar image of such contact,
• THEN an asynchronous upload request is being started for the specified user profile picture to the configured cloud storage provider,
• THEN in case such upload request succeeds, the avatar image url of the contact for the relevant conversation should be updated to the url of the uploaded avatar image, or
• THEN in case such upload request fails, the avatar image url of the contact for the relevant conversation should be updated to a generic avatar image,
• THEN an event should be sent to all connected clients via the web socket informing them about the changed avatar image,

## Acceptance Criteria

• GIVEN A cloud storage provider is specified in my Airy configuration files / environment variables including all necessary credentials to access and write to a bucket/folder specified for the relevant cloud storage provider,
• WHEN a new message arrives via a webhook for any source supporting media attachments and for a validly connected channel including a message attachment specified as an image or media file (including video),
• THEN an asynchronous upload request is being started for the specified message attachment to the configured cloud storage provider,
• THEN in case such upload request succeeds, the relevant attachment url of the message should be updated, or
• THEN in case such upload request fails, the upload request should be retried after a while, and in case it continues to fail the message attachment url should be kept as it was in the relevant message (despite it will expire)

## Dev Notes

Sources that currently support contact / user profile pictures by way of ingestion of new messages: facebook, google

Suggested cloud storage providers to support initially for this feature:

AWS S3, Google Cloud Storage, Azure Storage, local folder on server, none (=keep expiring urls)

Please feel free to choose a sub selection of these cloud storage providers for the initial version of this feature, depending on the complexity of the integration.

## Definition of Done

• [] Code review done
• [] Acceptance criteria passed
• [] Documentation done
• [] Internal rollout communication done

[chrismatix]

@steffh While reviewing and updating our current solution I found this bit in the Facebook Instagram messenger documentation.

In addition, you must comply all other technical requirements set forth in the technical documentation when using the Instagram Messaging API. In particular, Instagram Messaging API leverages CDN URLs which allow you to retrieve rich media content shared by users. The CDN URL returned via webhooks, and the Conversation API, is privacy-aware. This means, the CDN URL will not return the media when the content is deleted or expired. You must not download, retain, or otherwise store on your system the media content sent or made accessible by any user via the API (or enable any third party to do so) and you, or any third party, must not do anything to circumvent expiration and/or removal of any link to such media content, without our prior permission. Instead, if your app requires continued access to the media made available via the IG Messaging API, you must only store the privacy aware CDN URL in your system and use that to render the media made accessible via the API.

It seems that we have been doing this wrong sofar and I was wondering if for compliance reasons we should review this ticket. What do you think?

[AitorAlgorta]

Just to publicly answer this: The CDN problem is just for IG, so we shouldn't store anything from them, but we can store stuff from other sources. So if our solutions is source specific, and we don't store IG data, we are fine.

[chrismatix]

Revisiting this, since it has been quite some time. @steffh we already have this functionality, but only for AWS. Do you consider this ticket done if we also implement "Google Cloud Storage, Azure Storage, local folder on my server"?

The actual data manipulation is source-specific (and already implemented) as we discussed way back then.

[steffh]

Ancient ticket here.

Yes, the idea was to offer other cloud storage providers next to AWS as well.

Whether it is more efficient to have one media-resolver component with configuration options for AWS S3, Google Cloud Storage, Azure Storage, local folder, etc. or rather different connectors like a AWS S3 Connector that are fully independent from each other, you would have to decide.