[CustomScript] extension should allow for a script payload

[rcarmo]

Given that uploading dependencies to an added storage account may be unfeasible (or just too much overhead) when using CLI scripting, this extension should support the use case of inlining a shell script as part of an ARM template - something like the example below would go a very long way towards matching this competing feature and allowing for easy migration.

        "settings": {
          "inlineScript": "#!/bin/bash\nsudo apt-get -y update\nsudo apt-get install nginx"
        }

[boumenot]

It does.

 {
            "type": "Microsoft.Compute/virtualMachines/extensions",
            "name": "[concat(variables('vmName'), '/', variables('extensionName'))]",
            "apiVersion": "[variables('apiVersion')]",
            "location": "[resourceGroup().location]",
            "dependsOn": [
                "[concat('Microsoft.Compute/virtualMachines/', variables('vmName'))]"
            ],
            "properties": {
                "publisher": "Microsoft.OSTCExtensions",
                "type": "CustomScriptForLinux.Test",
                "typeHandlerVersion": "1.0",
                "autoUpgradeMinorVersion": true,
                "settings": {
                    "fileUris": [],
                    "commandToExecute": "echo 'Hello World!'"
                }
            }
        }

[rcarmo]

Well, if you check #215, you’ll see that isn’t working. Also, there is no clear support for multi-line scripts, or scripts that run with an alternate shell. Looking at the code, the commandToExecute field is treated like a single command line, and not as a multi-line script, even down to the subprocess invocation.

[boumenot]

I have posted an example showing how to execute multiple commands.

[rcarmo]

I would like to re-open the issue since that example does not cover the need to run a complete bash shell script for conditional provisioning of machines like the competing feature I mentioned - running two commands in a row is just the minimal case that didn't originally work for me, and the example does not cover a true use case.

[boumenot]

Our stance has been to use a Storage Account if you would like execute many commands, or multiline command. I agree this is more work for the user. I also think you are starting to graduate beyond script execution if you have configuration changes you would like to make. Where's the inflection point where custom script is duplicating work that is better done by others.

cc: @ahmetalpbalkan - what are your thoughts on this

[ahmetb]

@rcarmo Azure supports startup scripts through this script or cloud-init. If you specify CustomData field on your Virtual Machine, the script file/contents you provided will be executed on the boot. Also as @boumenot said, you can link to an external script file URL (could be hosted on Azure Storage privately, too).

[rcarmo]

Yes, I know. But having to:

1) Create a storage account 2) Invoke the storage API to obtain the keys 3) Implement the REST calls (with the specific signing required by the Storage API) or add a dependency to the SDK 4) Create a container 5) PUT the blob there 6) Set the blob privileges accordingly (and yes, I know I can skip this) 7) Add a URL to the template to reference the key

Is an inordinate amount of work.

On 04 Aug 2016, at 21:51 , Christopher Boumenot notifications@github.com wrote:

Our stance has been to use a Storage Account if you would like execute many commands, or multiline command. I agree this is more work for the user. I also think you are starting to graduate beyond script execution if you have configuration changes you would like to make.

cc: @ahmetalpbalkan https://github.com/ahmetalpbalkan - what are your thoughts on this

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Azure/azure-linux-extensions/issues/216#issuecomment-237679660, or mute the thread https://github.com/notifications/unsubscribe-auth/AAX96-XUnW-4-MXCI6SwcyObNCfpHCEaks5qclDQgaJpZM4JcAXC.

[ahmetb]

@rcarmo That's fair. Have you looked at CustomData field assuming your scripts are setup scripts you want to be executed after the VM boots for the first time?

[rcarmo]

@ahmetalpbalkan last time I looked at CustomData there was some kind of limitation/quirk regarding either the format/escaping/newlines of the script. This was back when ARM was in preview, so I forget exactly which. But I was planning to do more than first-boot init, so having the ability to redeploy an extension to run scripts on the machines without having to SSH in (which is going to be tricky because there might not be direct connectivity to them, even if I orchestrate from inside the subscription) would be really nice.

Otherwise I'd just use Ansible :) (before you mention it, I've looked at the DSC for Linux stuff already)

Still, I will test with CustomData so that I can at least finish the base template. I just wish the example was a bit more complex, though. I fear having to wrestle with escaping multi-line scripts all over again...

[ahmetb]

@rcarmo CustomData is just a string field, so multi-line strings has to be base64 encoded or escaped properly with \ns in ARM templates at some point. We have tricks to pass JSON objects as CustomData easily (lmk if you need to see it).

Do you know if AWS etc APIs let you pass this multiline string without escaping? tools such as Azure CLI, docker-machine etc let you pass the file as CustomData field without worrying about escaping etc.

[rcarmo]

@ahmetalpbalkan I used to use boto with AWS, and this example shows you how it could be used. It was a straight-up string, that was placed in the filesystem, made executable, and run with whatever shell was designated on the hasbang. I passed Python scripts to it.

No encoding, and IIRC it was POSTed directly to the REST API. No easy way to check that now :)

I'm curious as to how CustomData figures out whether the string is base64 encoded or not, though - and I can't find much documentation on it. I'd rather pass a simple \n escaped string for legibility (and it's easier to pack from an existing script), but am unsure as to any limits.

(I'm going to see if I can figure out #215 now, will test CustomDataright after I can get deployments going again)

[ahmetb]

I used to use boto with AWS, and this example shows you how it could be used. It was a straight-up string, that was placed in the filesystem, made executable, and run with whatever shell was designated on the hasbang. I passed Python scripts to it.

So in this case, boto (which is a Python SDK library) accepts a string for user-data field. This is the same for Python (or any other language) SDK for Azure as well. The SDKs handle escaping of multiple lines while constructing of the actual REST API request.

I'm curious as to how CustomData figures out whether the string is base64 encoded or not, though - and I can't find much documentation on it.

I believe It does figure out automatically. You're right the docs at https://msdn.microsoft.com/en-us/library/azure/mt163591.aspx should make it clear we allow non-base64decoded and properly escaped values as well.

[rcarmo]

Actually, boto's run_instances invokes get_object, which in turn uses make_request through a connection pool that through some indirection required by six actually invokes the Python standard library client, so I'm definitely sure that the string is just sent as a POST field without any intermediate encoding other than the required by HTTP...

(told you I used it. we contributed some patches to it on my previous job)

Thanks for the CustomData link, will check that out ASAP even if it doesn't full cover my needs.

[rcarmo]

@ahmetalpbalkan: Just had an idea that would save me a lot of time: do you happen to know if CustomData works with scale sets?

[ahmetb]

@rcarmo I was wrong about my earlier statement saying customData does accept non-base64 strings as well. It only accepts base64 strings and the docs are correct.

If you care about readability in your ARM templates you can specify a myScript in "variables" or "parameters" section and in customData field you can use:

"customData": "[base64(variables('myScript'))]"

so it essentially is the same thing as if customData accepted non-base64 strings.

[ahmetb]

Just had an idea that would save me a lot of time: do you happen to know if CustomData works with scale sets?

I am not sure but it should as it's part of VM configuration (just like SSH keys).

[rcarmo]

@ahmetalpbalkan Well, I'm going to know for sure in a few hours (just after a little test). Thanks for the base64pointer - I knew the function existed (and I can always encode when generating my template programmatically), but it's nice to have a complete example for other folk who might stumble upon this thread.

[ahmetb]

@rcarmo your template will probably fail with error:

Custom data in OSProfile must be in Base64 encoding and with a maximum length of 87380 characters.

[boumenot]

If we get you up and running the way you want I'll submit a template to the repository to share our learnings with other. (Learnings in addition to this thread.)

[rcarmo]

@ahmetalpbalkan: Meanwhile, I can confirm I've successfully used CustomData to prime a scale set. I still need to update the machines afterwards (the provisioning I have to do requires unique IDs), but it goes a long way.

[pedrams1]

Is it possible to improve "commandToExecute" to accept an array, rather than having a to rely on a long string, separated by the "&&" operator?

This can be achieved in AWS UserData like this:

"UserData": { "Fn::Base64": { "Fn::Join": [ "",[ "echo Macro\n", "echo Polo\n", ] } }

The reason this is good is it allows you to keep your bootstrap code together with your ARM template. This keeps it transportable which is great because you can deploy your ARM template to different environments without needing anything other than the template itself.

We're doing this right now with a very long string separated with the && operator but it's hard to manage.

CustomData is another option but we prefer the Custom Script Extension as it:

a) Works on all flavours of Linux available in the marketplace (cloud-init isn't installed on RHEL by default) b) Rolls up the results of scripts as part of the deployment. This means if our bootstrap fails we can fail the deployment and see stderr from the script bubbled right back up to the Azure CLI / Azure PowerShell cmdlets. This is especially powerful as all of our builds are automated through our CD processes and this allows us to report on these failures in our telemetry (Splunk).

[ahmetb]

@pedrams1 That would be a breaking change so we're not planning to take it anytime soon. If we ever make it an array, it will not be an array of commands but rather a single command split appropriately to its arguments (such as `["echo", "Marco Polo"]).

This is not exactly what you're asking but essentially what you are looking for is definitely the runcmd feature of cloud-init. To give you a bit more context:

Works on all flavours of Linux available in the marketplace (cloud-init isn't installed on RHEL by default)

RHEL on Azure will be getting cloud-init soon. The code for this is in review at the cloud-init repository and will soon be getting into the release pipeline. This will hopefully enable cloud-init on RedHat Atomic, Fedora and CentOS images as well soon. We're also actively looking to ship cloud-init on more distros. CoreOS already has its own version of cloud-init.

This means if our bootstrap fails we can fail the deployment and see stderr from the script bubbled right back up to the Azure CLI / Azure PowerShell cmdlets

If you're making ARM template deployment and if the Custom Script extension is part of it, if it fails, the template deployment should be failing as well (and the error should bubble up back to the Azure CLI/PS and Portal etc).

[pedrams1]

Hi @ahmetalpbalkan

Thanks for getting back to me. Absolutely agreed on the the cloud-init front - runcmd is precisely what we're looking for. Unfortunately RHEL is one of our target distributions and the lack of cloud-init support out of the box on RHEL means we're locked into using the Custom Script Extension.

Unfortunately, we do still have the same multi-line issue with the CustomData property in the ARM template. Because the base64 template function doesn't support an array as input, so we're back to having our entire script in one big long line.

If you're making ARM template deployment and if the Custom Script extension is part of it, if it fails, the template deployment should be failing as well (and the error should bubble up back to the Azure CLI/PS and Portal etc)

I think we're talking about the same thing here. We absolutely love this feature! We've gotten pretty advanced with bootstrapping our instances and the native support for dragging stderr from the target machine all the way back to the deployment of the ARM template is a killer feature! :)

[kjhosein]

Had the same issue using v 1.x of the CustomScriptForLinux extension, but discovered that there is a version 2.0.x of the CustomScript that has a script option that accepts a base64-encoded file and executes it with /bin/sh: https://github.com/Azure/custom-script-extension-linux

Note that the name and publisher/namespace of the extension has changed (I don't know why) as per: https://github.com/Azure/custom-script-extension-linux/blob/master/misc/manifest.xml

My basic tests have worked as expected. HTH!