Add Single Tech Sample for More Synapse Integration Testing

[kottofy]

Type of PR

• New content

Purpose

Adds a new single tech sample with additional Synapse integation tests and also can deploy to AzureUSGovernment cloud.

Does this introduce a breaking change? If yes, details on what can break

Author pre-publish checklist

• [x] Added test to prove my fix is effective or new feature works
• [x] No PII in logs
• [x] Made corresponding changes to the documentation

Validation steps

Follow steps in README

[adrian-gonzalez]

WHOOOOOO

[promisinganuj]

Hi @kottofy @mmclende, is this PR ready for DTD review. If not, please covert it to draft.

[devlace]

I ran into this error while trying to deploy the solution. Looks like it is missing the Trigger definition.

[mmclende]

@devlace We pushed some updates. Can you take a look when you get a chance?

[devlace]

I've tried deploying the sample and still running into an error. Looks like its a missing pipeline definition?

[mmclende]

I've tried deploying the sample and still running into an error. Looks like its a missing pipeline definition?

There's an issue where if there's no folder it is giving that issue. I had added READMEs to populate the folder on pull and just missed adding the readme to the last git oops. I pushed the change, can you try again?

[thurstonchen]

Dear colleagues,

I've tried deploying the sample and running into below error when starting Synapse pipeline trigger. (BTW, the previous issue has been fixed, thanks! : )

~~The debugging logs of relevant az synapse trigger start command shows there's RBAC issue, looks like we need to assign the Contributor role of related storage account to relevant SP, according to this document? Thanks for your help to take a look!~~

{
  "code": "BadRequest",
  "message": "The client 'xxx' with object id 'xxx' does not have authorization to perform action 'Microsoft.EventGrid/eventSubscriptions/write' over scope '/subscriptions/xxx/resourceGroups/mdwdops-mdwtc21-dev-rg/providers/Microsoft.Storage/storageAccounts/mdwdopsstdevmdwtc21/providers/Microsoft.EventGrid/eventSubscriptions/xxx' or the scope is invalid. If access was recently granted, please refresh your credentials.",
  "target": "T_Stor_mdwdops-mdwtc21-dev",
  "details": null,
  "error": null
}

UPDATED on Apr. 6th

• Checked the above trigger starting issue again today and noticed the root cause is not around RBAC (because my user account used by az commands has been Owner of relevant storage account).

  Instead it's subscription Resource Provider issue, since my subscription was created recently and both of Microsoft.EventGrid and Microsoft.DataFactory resource providers are not registered yet, after the registration, the deploy.sh re-run looks good, all resources are deployed properly and pipeline trigger has started OK too.

  Therefore maybe we could consider either to add that resource provider registration step as prerequisites to relevant README, or include registration logic to deploy scripts? So that the deploy.sh could work well for fresh new subscription as well : )

  https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-event-trigger?tabs=data-factory#role-based-access-control

• Also tried running the integration tests, it works well as expected! Just one minor point, after executed the clean_up.sh script, I notice relevant app registration is still there, which was created by az ad sp create-for-rbac command (in deploy_infrastructure.sh), since the az ad sp delete command only delete relevant SP; maybe we use az ad app delete command instead to delete both app registration and SP at once?

Best, DataOps V-team

[sweanan]

hi @kottofy Just a small comment to update the documents for the below:

1. when trying to run these integration test noticed that the Service principal created would require to have Storage Blob Data Contributor along with the owner role permissions.

2. Also from the Synapse workspace we need to provide the [RBAC role assignments] (https://learn.microsoft.com/en-us/azure/synapse-analytics/security/how-to-manage-synapse-rbac-role-assignments) to this service principal

[devlace]

I tried deploying this and looks good to me! Thanks @thurstonchen and @sweanan for trying this out as well. @kottofy if you can do the quick update to the docs as Sweanan suggested, I'm happy to merge this.