Bug: Apostrophes are dropped when converting back to YAML

[piotrminkina]

Hello,

First of all, thank you for the great tool that is yq.

I would like to report a problem I noticed in version 3.0.2, well values of type string whose value is yes or no, when converted back to YAML have apostrophes dropped, which will cause the YAML parser to convert such values to type boolean. See the following examples.

My environment:

$ jq --version
jq-master-v20210212-7320-g4be84a0f82
$ yq --version
yq 3.0.2
$ python3 --version
Python 3.9.5
$ cat /etc/alpine-release 
3.14.0

Example file example.yaml:

---
graphana_dashboard: 'yes'

Actual command result:

$ yq --yaml-output '.' example.yaml 
graphana_dashboard: yes

Expected result of the command:

$ yq --yaml-output '.' example.yaml 
grafana_dashboard: 'yes'

For the example in version 2.14.0, which I have used so far, the result of the command is as expected. I also checked what the document looks like in JSON format, after conversion by yq 3.0.2, well it is as expected, which would mean that the problem occurs when converting from JSON to YAML again.

$ yq '.' example.yaml 
{
  "grafana_dashboard": "yes"
}

Regards Piotr Minkina

[kislyuk]

Thanks. Yes, starting with version 3, yq uses the YAML 1.2 grammar, where bare yes literals are not interpreted as booleans. So this is intended behavior.

Unfortunately, the YAML 1.2 grammar is not yet available as part of the PyYAML package. I'm planning to publish a package that configures PyYAML to use the YAML 1.2 grammar.

[piotrminkina]

Unfortunately, this is currently a problem, for example, when operating on Kubernetes manifests. See a sample manifest file:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    sidecar.istio.io/inject: "false"
    annnotation.dashboard: "yes"
  labels:
    label.dashboard: "no"
  name: test-pod
  namespace: default
spec:
  containers:
  - name: test-pod
    image: alpine
    env:
    - name: ENV_DASHBOARD
      value: "yes"
    args:
      - cat
    stdin: true

This is the correct Kubernetes manifest, so applying it to the cluster succeeds:

$ cat pod.yaml | kubectl apply -f .                 
pod/test-pod created

Unfortunately, if I wanted to change something in this object, using the yq tool, it is impossible:

$ yq --yaml-output '.' pod.yaml | kubectl apply -f -
error: unable to decode "STDIN": resource.metadataOnlyObject.ObjectMeta: v1.ObjectMeta.Annotations: ReadString: expects " or n, but found t, error found in #10 byte of ...|shboard":true,"sidec|..., bigger context ...|etadata":{"annotations":{"annnotation.dashboard":true,"sidecar.istio.io/inject":"false"},"labels":{"|...

As I understand, currently the solution is to use yq version 2.x?

Regards Piotr Minkina

[kislyuk]

Thanks for the feedback. I've been thinking about this and I'm going to release an update to v3 which will use yaml 1.2 as the input grammar, but yaml 1.1 as the output grammar by default. This will preserve the quotes in the example that you gave.

Please understand that this is a tricky backwards compatibility situation. If you take the github actions yaml for example, it uses the bare on: key frequently, implying the 1.2 grammar. Any yq invocations on that yaml should keep the output formatting as close to the original as possible, but I agree with you that the safest/most backwards compatible output is to quote the ambiguous values - resulting in "on": and other quoting that may appear inconsistent.

[piotrminkina]

Thank you for this comment. I wish you good luck in the further development of this tool.

Regards Piotr Minkina

[kislyuk]

Released in v3.1.0, please test.