Support for YAML

[tomkerkhove]

After using Kubernetes for a while I have to say that I find YAML declarations a lot nicer than JSON.

Are you planning on supporting this?

[ahelland]

I'm split on this. I love how YAML reads; it's better than JSON if you want to skim the definitions.

On the other hand I hate how YAML is super-fuzzy about the number of spaces, and being strict about whether you use space or tabs for indentation.

[mkosieradzki]

@ahelland

You are aware that YAML is a superset of JSON and every JSON is a valid YAML, right?

http://yaml.org/spec/1.2/spec.html#id2759572

[ahelland]

Yes, I have no problem with that part. It sort of aligns with what I'm saying: YAML is focused on reading, JSON is focused on writing. That's a simplification of course, but you get my point :) I have messed up k8s configs where I didn't pay proper attention to these details. (Sloppiness can break JSON too though.)

I would be entirely happy with both being supported as long as documentation is coherent, and you don't see samples that mix YAML and JSON in confusing ways.

[tomkerkhove]

Support for both JSON & YAML feel like the sweet spot to me, gives people the opportunity to choose instead of enforcing one over the other.

[CESARDELATORRE]

YAML feels a lot simpler and cleaner, less noise than a JSON file which is usually a lot larger. I would really prefer to work with YAML as long as the support for SF-Mesh is exactly the same than when using JSON, no less support and everything being released at the same time for both formats, not later for YAML... ;)

[vipul-modi]

The Visual Studio tooling will use YAML for authoring, however we are also split in terms of being able to pick a particular format. You can see some examples here: https://github.com/Azure/service-fabric-mesh-preview-pr/blob/master/samples/src/quickstart/Sbz/VotingData.yaml

Current thinking is to use YAML with support for structural overlay for overrides and then convert everything to JSON when sending it over the wire. There will be standalone tools as well as CLI will have the inbuilt conversion.

We would really appriciate feedback on this approach.

[tomkerkhove]

@VipulM-MSFT Sounds good to me!

[NickDarvey]

+1 for YAML because of being able to include comments in the file.

[bacongobbler]

Please avoid YAML as a standard if you can. If the SIG Apps user survey (slides 9 and 26 in particular) is an indicator of anything, most users dislike interacting with YAML as a data format. Comments are nice, sure, but there's a whole lot going wrong with YAML's syntax and the ecosystem surrounding YAML, namely that there's likely no single parser that handles every case in YAML's 23,000+ word spec (and in many edge cases, a lot of them are straight up wrong).

As for proposing alternative solutions, I've written up a mini-rant on why Azure/Draft uses TOML as the standard data format here: https://github.com/Azure/draft/blob/master/docs/reference/dep-006.md#rationale

I'd highly suggest a more lightweight data format than YAML. TOML fit our needs quite nicely and has a good tooling ecosystem built around it. It has simple markup syntax, comments and - above all else - accomplishes 100% of the tasks users require from a human-readable data format.

Other suggested alternatives are:

• INI
• XML

INI's an interesting one: it allows comments, it's human-readable and provides a simple data format, but the tooling ecosystem could use some improvement. TOML outshines INI in this regard, IMO.

XML on the other hand falls to the same issue JSON has: it's a data format primarily for machines, not humans.

I know YAML is a format that Kubernetes supports, but IMO we'd all be much better off if YAML kicked the metaphorical bucket in favour of something else.

Sorry for the rant; just needed to get that off my chest. :heart:

[bacongobbler]

Current thinking is to use YAML with support for structural overlay for overrides and then convert everything to JSON when sending it over the wire.

Just out of curiosity, shouldn't it be the other way around? YAML is a superset of JSON. If you convert YAML into JSON - in theory - some objects (anchors, for example) will not translate into JSON.

Oh, and one extra data point I'll touch upon here: we found that by rewriting some of our parsers to accept JSON instead of yaml, we experienced a 4x-6x performance and memory improvement overall, simply because the JSON parsers are much more memory-efficient due to the simplified syntax over what YAML parsers need to implement. Some of our findings can be found at https://github.com/mattfarina/yamlbench

[tomkerkhove]

Personally I really really prefer maintaining declarations in YAML over JSON as they are more human readible, but I'm not an expert on performance but I'm not really sure if this really matters here? We will not provide pages and pages of YAML, I think

[piizei]

I see that the problem with the ARM JSON is the verbosity. I really prefer writing Terraform over ARM a lot because of the density (there are other benefits too...). K8S YAML is nightmare to modify but delight to read and Terraforms condensed HJSON like syntax is nice compromise between these 2.

My vote would go for either radically condensed JSON or YAML

[tomkerkhove]

@VipulM-MSFT That idea sounds good, is this supported already?

[mikkelhegn]

I broke the link @VipulM-MSFT posted above (sorry) - you can see an example here: https://github.com/Azure/service-fabric-mesh-preview-pr/blob/master/samples/src/votingapp/windows/VotingApp/VotingWeb/Service%20Resources/service.yaml

[tomkerkhove]

Thanks for the update @MikkelHegn.

Does that mean we can use yaml declarations with the following command already? az mesh deployment create --resource-group foo --template-uri <link>

[mikkelhegn]

At this point only VS does the YAML to JSON and ARM JSON convention as par tof local and remote deploy. We will add that functionality into the CLIs soon.

[tomkerkhove]

Circling back to this - In #134 @VipulM-MSFT mentioned CLI support for YAML is coming. I presume this will mean that the above scenario will be supported and I can deploy my YAML templates via az mesh deployment create?

[vipul-modi]

Yes that is the idea.

[mattquinn]

      Yes that is the idea.

My understanding is that this is now part of the CLI toolkit using the --input-yaml-files parameter.

Is that correct? Is it still a work in progress?

I'm using Azure CLI 2.0.52 service fabric mesh installed, but am getting lots of parsing errors from really basic YAML files. They work with the Visual Studio SDK to publish to Azure, but point the az mesh deployment create command at them and there's all manner of issues.

I have a really empty app.yaml:

application: schemaVersion: 1.0.0-preview2 name: MQSFMesh101

And then a couple of services within the solution that have service.yaml files as below:

`application: schemaVersion: 1.0.0-preview2 name: MQSFMesh101 properties: services:

• name: WebFrontEnd properties: description: WebFrontEnd description. osType: Windows codePackages:
  • name: WebFrontEnd image: webfrontend:dev endpoints:
    • name: WebFrontEndListener port: 20005 environmentVariables:
    • name: ASPNETCORE_URLS value: http://+:20005
    • name: MQSFMesh101_BackendServiceHostName value: backendservice
    • name: MQSFMesh101_BackendServicePort value: 20006 resources: requests: cpu: "[parameters('WebFrontEnd_cpu')]" memoryInGB: 1 replicaCount: 1 networkRefs:
  • name: MQSFMesh101Network endpointRefs:
    • name: WebFrontEndListener`

As I say, the Visual Studio tooling takes these and publishes without issue.

Run a command such as:

az mesh deployment create --resource-group RG_ServiceFabricMeshDec12Demo --input-yaml-files .\app.yaml,.\gateway.yaml,.\network.yaml,'..\..\BackendService\Service Resources\service.yaml','..\..\WebFrontApp\Service Resources\service.yaml' --parameters ..\Environments\Cloud\parameters.yaml

from the relevant path and it throws all manger of syntax errors when trying to parse the yaml (all files have been yamllint'd to verify they're valid YAML.

'list' object has no attribute 'get' Traceback (most recent call last): File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\knack\knack\cli.py", line 206, in invoke File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\azure-cli-core\azure\cli\core\commands\__init__.py", line 346, in execute File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\six\six.py", line 693, in reraise File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\azure-cli-core\azure\cli\core\commands\__init__.py", line 320, in execute File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\azure-cli-core\azure\cli\core\commands\__init__.py", line 169, in __call__ File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-e5ce5s7r\azure-cli-core\azure\cli\core\__init__.py", line 440, in default_command_handler File "C:\Users\***\.azure\cliextensions\mesh\azext_mesh\custom.py", line 283, in deploy_arm_template input_yaml_files, deployment_name, parameters, mode, no_wait=no_wait) File "C:\Users\***\.azure\cliextensions\mesh\azext_mesh\custom.py", line 249, in _deploy_arm_template_core SFMergeUtility.sf_merge_utility(file_path_list, "SF_SBZ_RP_JSON", parameters=parameters, output_dir=None, prefix=prefix) File "C:\Users\***\.azure\cliextensions\mesh\sfmergeutility\sf_merge_utility.py", line 48, in sf_merge_utility merged_resource_documents = SFMergeUtility.load_and_merge_partial_documents(input_list) File "C:\Users\***\.azure\cliextensions\mesh\sfmergeutility\sf_merge_utility.py", line 207, in load_and_merge_partial_documents name = my_dict.get(kind).get(Constants.PrimaryPropertyName) AttributeError: 'list' object has no attribute 'get'

Am I missing something here? I cannot find any documentation on what elements should/must exist in the YAML file to make the deployment work - or am I totally misinterpreting what the --input-yaml-files parameter is for?

My understanding is you can use this to have it generate the ARM template JSON so you have only to maintain once definition of your services in YAML, rather than YAML for Docker/SFMesh and an ARM replica for deployment to Azure. Am I reading that right? yaml-files.zip

NOTE: My objective here is to hack out the CLI commands to build and deploy the application so that I can then automate it in Azure DevOps, but struggling to get it to deploy locally from the CLI first at the moment.

[anantshankar17]

Hi @mattquinn, your usage is correct and that is how it should be.

I looked at your yamls, and it seems the app.yaml and the service-frontend.yaml have the BOM characters at the beginning. Removing those by changing the file encoding to UTF-8 (without BOM), the same deployment command works for me. Refer to this https://stackoverflow.com/questions/3255993/how-do-i-remove-%C3%AF-from-the-beginning-of-a-file. Can you also give that a try and let us know ? Also, how did you create these yamls ? Were these created on a Linux system ? On which platform are you using the "az mesh" command, (windows/linux) ?

[mattquinn]

Thank you @anantshankar17 - life saver. I even looked at the encoding having seen the strange representation of whitespace chars in the error message, but somehow managed to think I'd looked at / fixed that.

You were indeed right - Save without encoding / UTF-8 Codepage 65001 and BOOM! works like a charm.

[anantshankar17]

@mattquinn good to know. Closing this thread.