Missing sif-demo-dev-activityPipelineSM State Machine in Spoke Account

[sathish0804]

The sif-demo-dev-activityPipelineSM state machine is not visible in the AWS Step Functions console of the spoke account. As part of the workflow, this state machine should have been triggered after the successful execution of the sdf-demo-seeder state machine. These are the state machines created in spoke account

[johnrotach]

Hello. The name sif-demo-dev-activityPipelineSM is made up of: sif-<tenant id>-<environment>-activityPipelineSM. It looks like in your case you used gb-tenant as the tenant ID and dev as the environment when deploying SIF so the step function is: sif-gb-tenant-dev-activityPipelineSM. I'll make a note to update the documentation to more clearly call out that the name is dependent on the parameters given to the SIF deployment.

[sathish0804]

I am not able to see a successful execution in the sif-demo-dev-activityPipelineSM state machine. Upon checking the Step Functions in the spoke account, the pipeline execution doesn't complete successfully. I expect to see a successful execution of the pipeline that registers the pipeline and metric output in DataZone.

[johnrotach]

From this screenshot it doesn't look like the step function started. Is that true? Just double checking to see if there were any errors. Did the CloudFormation stacks in the spoke account successfully deploy? Did the sdf-demo-seeder state machine succeed?

[sathish0804]

Step Function Not Triggered: You’re right—based on the screenshot, it appears that the sif-demo-dev-activityPipelineSM state machine did not start. I checked and it seems like the trigger may not have been initiated from the previous step. sdf-demo-seeder screenshots below

These are all stacks deployed in Spoke account

[sathish0804]

Hi, I hope you're doing well! I wanted to kindly follow up on this issue. Could you please take a look and provide any feedback or insights when you have the chance? Your response would be greatly appreciated. Thank you for your time!

[johnrotach]

Can you check the logs for the Lambda functions associated with the tasks in the Step Function? They are:

• sdf-spoke-workflow-triggerPipeline
• sdf-spoke-workflow-checkPipeline

Check for errors or any other messages in the logs.

[sathish0804]

sdf-spoke-workflow-checkPipeline logs:

sdf-spoke-workflow-triggerPipeline logs:

[johnrotach]

Hmmm. It seems as if maybe the pipelines weren't actually seeded. Let's try to validate that first. For that we'll have to do a list pipelines call in your SIF instance. If you have done anything with the SIF API then go do a list pipelines call and see if the pipelines were seeded. If you have not, the instructions are below:

You will be doing a GET call to the /pipelines path in the pipelines module API.

The swagger doc for this API can be found here. The base URL endpoint for the pipelines API can be found in an SSM parameter named: /sif/<tenant id>/<envrionment>/pipelines/apiUrl. See here to obtain an auth token which will be used in Bearer Token auth for invoking the API. Reply back if you have additional questions on doing this call.

If the pipelines are present we will need to debug why they weren't started. If there are no pipelines we will need to revisit deploying the seeding stack.

[sathish0804]

While doing a GET call to the /pipelines, iam getting "message": "Unauthorized"

[johnrotach]

This may just be the request headers. Can you verify those first? They should include:

• Authorization: Bearer <token> - looks like you should be good with this one from your screenshot above
• Accept-Version: 1.0.0 - this is how the API is versioned
• x-groupcontextid: / - this is what group context in SIF you want to use in the call - / is the root level - let's try that first so we can see all pipelines across all groups (see also the note below on including children)

Also, include the query string parameter includeChildGroups like this: https://<your base URL>/pipelines?includeChildGroups=true This will tell the list call to include pipelines from all the child groups under root (/) which should be all pipelines in SIF.

[sathish0804]

[johnrotach]

Can you confirm that the user you used when you generated your bearer token is the same user you put in in the ADMIN_EMAIL parameter when you deployed SIF. This would have been done as part of the tenant deploy in step 3. ix. in the instructions here.

[sathish0804]

[johnrotach]

Ok, this looks good. It seems the SIF pipelines have been seeded. Can you try to run the SDF workflow deployment again? It is the one calling out the SdfHubWorkflowStack.

[sathish0804]

I run this command npm run cdk -- deploy -c hubAccountId=<HUB_ACCOUNT_ID> \ -c spokeAccountId=<SPOKE_ACCOUNT_ID> -c domainId=<DATAZONE_DOMAIN_ID> \ -c domainName=<DATAZONE_DOMAIN_NAME> -c projectId=<DATAZONE_PROJECT_ID> \ -c athenaEnvironmentId=<DATAZONE_DATA_LAKE_ENV_ID> -c redshiftEnvironmentId=<DATAZONE_DATA_WAREHOUSE_ENV_ID> \ -c roleArn=<SERVICE_ROLE_ARN> -c environment=<SIF_ENV> -c tenantId=<SIF_TENANT_ID> \ -c sifAdminEmailAddress=<SIF_ADMIN_EMAIL> -c sdfAdminUserId=<IAM_IDENTITY_CENTER_USERNAME> \ —require-approval never —concurrency=10 SdfSpokeWorkflowStack

The sif-demo-dev-activityPipelineSM state machine is not visible in the AWS Step Functions console of the spoke account. As part of the workflow, this state machine should have been triggered after the successful execution of the sdf-demo-seeder state machine.