[QUERY] Synapse - How to get pipeline output?

[Gbertaz]

Library name and version

Azure.Analytics.Synapse.Spark 1.0.0-preview.8

Query/Question

Hello!

I have a Notebook which returns a string and I am running it in a Pipeline. Ho can I pass the Notebook's output to the pipeline and read that value in code?

I created the same flow in Synapse workspace and it works fine. I have a Notebook activity and a Set variable activity as you can see in the images. Please find the following code example I am using. I get the following error:

Service request failed. 
Status: 400 (Bad Request)
Content: { 
 "code": "BadRequest",
 "message": "ErrorCode=InvalidTemplate, ErrorMessage=cannot reference action 'act-CheckDatabase'. Action 'act-CheckDatabase' must either be in 'runAfter' path, or be a Trigger
 "target": "pipeline/..../runid/....",
 "details": null,
 "error": null
}

The part of the error ...must either be in 'runAfter' path... sounds like the Set variable activity must be set as dependent on the Notebook activity with condition Succeeded. But I am not able to set this dependency on code. Property DependsOn in Activity class of SDK is read-only and I don't see any set method.

public async Task<string> RunNotebook(NotebookSerializationModel notebook)
{
 var pipelineName = $"ppl-{notebook.Name!}";
 string activityName = $"act-{notebook.Name}";
 var client = await apiService.CreatePipelineClient();
 var notebookRef = new SynapseNotebookReference(NotebookReferenceType.NotebookReference, notebook.Name);
 var notebookResource = new SynapseNotebookActivity(activityName, notebookRef);

 // Notebook parameters
 foreach(var parameter in notebook.Parameters!)
 {
 notebookResource.Parameters.Add(parameter);
 }

 var pipelineResource = new PipelineResource();
 pipelineResource.Activities.Add(notebookResource);
 pipelineResource.Variables.Add("NotebookOutput", new VariableSpecification(VariableType.String));

 SetVariableActivity setActivity = new SetVariableActivity("NotebookOutput");
 setActivity.Name = "NotebookOutput";
 setActivity.Value = $"@activity('{activityName}').output.status.Output.result.exitValue";

 pipelineResource.Activities.Add(setActivity);

 var pipelineOperation = await client.StartCreateOrUpdatePipelineAsync(pipelineName, pipelineResource);
 Response<PipelineResource> createdPipeline = await pipelineOperation.WaitForCompletionAsync();
 var run = await client.CreatePipelineRunAsync(pipelineName);

 return run.Value.RunId;
}

Thank you!

Environment

No response

[ghost]

Thank you for your feedback. This has been routed to the support team for assistance.

[github-actions[bot]]

Thank you for your feedback. This has been routed to the support team for assistance.

[Gbertaz]

I found the way how to set activity dependency. Here is the working code:

ActivityDependency dependency = new ActivityDependency(notebookActivityName, new List<DependencyCondition>() { DependencyCondition.Succeeded });
SetVariableActivity setActivity = new SetVariableActivity(notebook.SetVariableActivityName);
setActivity.VariableName = "NotebookOutput";
setActivity.Value = $"@activity('{notebookActivityName}').output.status.Output.result.exitValue";
setActivity.DependsOn.Add(dependency);

pipelineResource.Activities.Add(setActivity);

[navba-MSFT]

@Gbertaz Thanks for getting back and providing an update. If you need any further assistance on this issue in future, please feel free to reopen this thread. We would be happy to help.