✨ AppFlow for ingesting data from SharePoint

[gwionap]

Describe the feature request.

We would like to have an AppFlow setup for ingesting data from SaaS applications. The current use case is for ingesting data from SharePoint online to reduce the need for users to manually upload data from SharePoint into S3.

Describe the context.

OPG as well as a number parts of MoJ are migrating to MoJo and so a number of files useful for analysis (including unstructured data) will be migrated there. There is already a feature request for setting up DataSync to handle legacy fileshares (which will be needed in the medium term as VBA/Macros are disabled on sharepoint and so some key files are still on DOM1) but we would like to make sure we have a long term solution for data held on SharePoint.

Value / Purpose

• This will allow OPG to continue to work on ambitions around the modernising lasting power of attorney programme.
• Many analysts work with data which they take from SharePoint / Teams sites and so this would be of wider value in reducing the need to upload manually

User Types

data engineers, data scientists, data analysts

[simon-pope]

Request forum: Request and use cases understood. @bagg3rs to investigate if there are any issues with authentication (user account vs service account). Following this a spike will be created to cover providing Appflow and ingesting data from sharepoint in to S3.
If a service account is required, then a request will need to be raised with Technical Services