Provide data consolidation process - Githubissues

tomkerkhove / gdpr-with-azure

Scenarios on how you can be GDPR compliant by using Azure services

MIT License

6 stars 2 forks source link

Provide data consolidation process #11

Open tomkerkhove opened 6 years ago

tomkerkhove commented 6 years ago

Provide data consolidation process that is represented by an Azure Data Factory pipeline.

Process will:

[x] Get all stored user information from our SQL database
[x] Process StackExchange data set stored in Azure Data Lake Store, via U-SQL script ran by Azure Data Lake Analytics
[ ] Consolidate data from above provides in Azure Storage and compress it (#20 )
[ ] Publish event on Event Grid with following information: (#19)
- Type: DataConsolidationCompleted
- Data: EmailAddress, First Name, Last Name

Input parameters

Email address of customer

Endpoint in API Management

Verb: Post
Uri: /users/{email}/data-consolidation/archive

tomkerkhove commented 6 years ago

Given this is not available in Data Factory yet, the trigger will go through API Management

tomkerkhove commented 6 years ago

This is done with known limitations:

Output from ADLA is not copied to Blob storage yet
No event grid event since this is not there yet