Action crashing when pipeline not folder present

[tomasvanpottelbergh]

When trying out this action on a test branch where I only added a notebook, the action crashes with the following error:

Stdout: Unexpected error during export template command: {"stack":"Error: ENOENT: no such file or directory, scandir 'synapse/pipeline'","message":"ENOENT: no such file or directory, scandir 'synapse/pipeline'","errno":-2,"code":"ENOENT","syscall":"scandir","path":"synapse/pipeline"}

Stderr: /home/runner/work/Anvil/Anvil/downloads/main.js:336518
          throw ex;
          ^

[Error: ENOENT: no such file or directory, scandir 'synapse/pipeline'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'scandir',
  path: 'synapse/pipeline'
}

I believe the action should ignore this if the folder doesn't exist.

[lordozb]

If you are using validate or validate and deploy operation, it expects the files to be in the respective folders as generated in the collaboration branch for Synapse Workspace. The directory structure should be same.

[tomasvanpottelbergh]

I understand, but this folder is not present when one does not create any pipelines, but only notebooks for example. Maybe this is not very common in production, but still I would not expect the action to crash.

[lordozb]

Does the notebook template has any reference to pipelines?

[tomasvanpottelbergh]

No, I don't think that's possible? The notebooks can be standalone without any reference to other resources.

[lordozb]

Yes, thats correct. But I was asking whether the template you added has any reference to any pipeline? If you are facing the issue, I request you to raise a support ticket on azure portal and please provide the pipeline logs and notebook template.

[tomasvanpottelbergh]

This is the dummy notebook I added via the Studio, which is written to the synapse/notebook folder:

{
    "name": "Test notebook",
    "properties": {
        "nbformat": 4,
        "nbformat_minor": 2,
        "sessionProperties": {
            "driverMemory": "28g",
            "driverCores": 4,
            "executorMemory": "28g",
            "executorCores": 4,
            "numExecutors": 2,
            "conf": {
                "spark.dynamicAllocation.enabled": "false",
                "spark.dynamicAllocation.minExecutors": "2",
                "spark.dynamicAllocation.maxExecutors": "2",
                "spark.autotune.trackingId": "b0e5eb16-044e-4a04-a59d-2f52dc3a8377"
            }
        },
        "metadata": {
            "saveOutput": true,
            "enableDebugMode": false,
            "kernelspec": {
                "name": "synapse_pyspark",
                "display_name": "python"
            },
            "language_info": {
                "name": "python"
            },
            "sessionKeepAliveTimeout": 30
        },
        "cells": [
            {
                "cell_type": "code",
                "source": [
                    "# Test notebook"
                ],
                "execution_count": null
            }
        ]
    }
}

Now as long as I don't create a Pipeline in the Synapse Studio, the synapse/pipeline folder is not created. Since the action crashes, I don't see the generated template. I don't really understand how this is not an issue in this action. It is this action that combines the JSON files in the different folders into a single template file, no?

[lordozb]

    "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "workspaceName": {
            "type": "string",
            "metadata": "Workspace name"
        }
    },
    "variables": {
        "workspaceId": "[concat('Microsoft.Synapse/workspaces/', parameters('workspaceName'))]"
    },
    "resources": [
        {
            "name": "[concat(parameters('workspaceName'), '/testnotebook')]",
            "type": "Microsoft.Synapse/workspaces/notebooks",
            "apiVersion": "2019-06-01-preview",
            "properties": {
                "nbformat": 4,
                "nbformat_minor": 2,
                "sessionProperties": {
                    "driverMemory": "28g",
                    "driverCores": 4,
                    "executorMemory": "28g",
                    "executorCores": 4,
                    "numExecutors": 2,
                    "conf": {
                        "spark.dynamicAllocation.enabled": "false",
                        "spark.dynamicAllocation.minExecutors": "2",
                        "spark.dynamicAllocation.maxExecutors": "2",
                        "spark.autotune.trackingId": "b0e5eb16-044e-4a04-a59d-2f52dc3a8377"
                    }
                },
                "metadata": {
                    "saveOutput": true,
                    "enableDebugMode": false,
                    "kernelspec": {
                        "name": "synapse_pyspark",
                        "display_name": "python"
                    },
                    "language_info": {
                        "name": "python"
                    },
                    "sessionKeepAliveTimeout": 30
                },
                "cells": [
                    {
                        "cell_type": "code",
                        "source": [
                            "# Test notebook"
                        ],
                        "outputs": [],
                        "execution_count": null
                    }
                ]
            },
            "dependsOn": []
        }
    ]
}

I took the template you shared and ran it. Seems to be working fine. For further help please raise a support ticket on Azure portal. The team will be able to provide further assistance.

[tomasvanpottelbergh]

If I manually create a pipeline folder it might indeed work, but the problem is that Synapse doesn't create this by default.

[lordozb]

I didn't create any pipeline folder. Just notebook folder with this template in it.

[tomasvanpottelbergh]

Strange, because that doesn't work for me with the latest version of the action. But if you are not able to reproduce it, feel free to close it as it's not a major issue.

[lordozb]

I request you to please raise a support ticket on Azure portal. The team will look into it.