search

kislyuk / yq

Command-line YAML, XML, TOML processor - jq wrapper for YAML/XML/TOML documents
https://kislyuk.github.io/yq/
Apache License 2.0
2.57k stars 82 forks source link

Question about --yaml-output behavior (possible bug) #104

Open nemchik opened 3 years ago

nemchik commented 3 years ago

I am trying to parse the following yml:

services:
  watchtower:
    image: containrrr/watchtower
    hostname: ${DOCKERHOSTNAME}
    labels:
      com.dockstarter.appinfo.deprecated: 'false'
      com.dockstarter.appinfo.description: Automatically update running Docker containers
      com.dockstarter.appinfo.nicename: Watchtower
      com.dockstarter.appvars.watchtower_cleanup: 'true'
      com.dockstarter.appvars.watchtower_enabled: 'true'
      com.dockstarter.appvars.watchtower_include_stopped: 'false'
      com.dockstarter.appvars.watchtower_monitor_only: 'false'
      com.dockstarter.appvars.watchtower_network_mode: ''
      com.dockstarter.appvars.watchtower_notification_template: '{{range .}}{{.Message}}{{println}}{{end}}'
      com.dockstarter.appvars.watchtower_notification_url: ''
      com.dockstarter.appvars.watchtower_notifications_level: info
      com.dockstarter.appvars.watchtower_notifications: shoutrrr
      com.dockstarter.appvars.watchtower_repo_pass: ''
      com.dockstarter.appvars.watchtower_repo_user: ''
      com.dockstarter.appvars.watchtower_schedule: 0 0 4 * * *
      com.dockstarter.appvars.watchtower_timeout: 10s
    network_mode: ${WATCHTOWER_NETWORK_MODE}
    container_name: watchtower
    environment:
      - REPO_PASS=${WATCHTOWER_REPO_PASS}
      - REPO_USER=${WATCHTOWER_REPO_USER}
      - TZ=${TZ}
      - WATCHTOWER_CLEANUP=${WATCHTOWER_CLEANUP}
      - WATCHTOWER_INCLUDE_STOPPED=${WATCHTOWER_INCLUDE_STOPPED}
      - WATCHTOWER_MONITOR_ONLY=${WATCHTOWER_MONITOR_ONLY}
      - WATCHTOWER_NOTIFICATION_TEMPLATE=${WATCHTOWER_NOTIFICATION_TEMPLATE}
      - WATCHTOWER_NOTIFICATION_URL=${WATCHTOWER_NOTIFICATION_URL}
      - WATCHTOWER_NOTIFICATIONS_LEVEL=${WATCHTOWER_NOTIFICATIONS_LEVEL}
      - WATCHTOWER_NOTIFICATIONS=${WATCHTOWER_NOTIFICATIONS}
      - WATCHTOWER_SCHEDULE=${WATCHTOWER_SCHEDULE}
      - WATCHTOWER_TIMEOUT=${WATCHTOWER_TIMEOUT}
    logging:
      driver: json-file
      options:
        max-file: ${DOCKERLOGGING_MAXFILE}
        max-size: ${DOCKERLOGGING_MAXSIZE}
    restart: unless-stopped
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock
      - ${DOCKERSHAREDDIR}:/shared
      - ${DOCKERSTORAGEDIR}:/storage
version: '3.4'

With the following command:

yq -y -r '.services.watchtower.labels["com.dockstarter.appvars.watchtower_cleanup"]' watchtower.yml

I get the following output:

true
...

With the following command:

yq -r '.services.watchtower.labels["com.dockstarter.appvars.watchtower_cleanup"]' watchtower.yml

I get the following output:

true

In my use case I am actually passing in the filter as a variable, and in most cases I want the value of a given key, but in some cases I want the yml children of a parent, like so:

With the following command:

yq -r '.services.watchtower.labels' watchtower.yml

I get the following output:

com.dockstarter.appinfo.deprecated: 'false'
com.dockstarter.appinfo.description: Automatically update running Docker containers
com.dockstarter.appinfo.nicename: Watchtower
com.dockstarter.appvars.watchtower_cleanup: 'true'
com.dockstarter.appvars.watchtower_enabled: 'true'
com.dockstarter.appvars.watchtower_include_stopped: 'false'
com.dockstarter.appvars.watchtower_monitor_only: 'false'
com.dockstarter.appvars.watchtower_network_mode: ''
com.dockstarter.appvars.watchtower_notification_template: '{{range .}}{{.Message}}{{println}}{{end}}'
com.dockstarter.appvars.watchtower_notification_url: ''
com.dockstarter.appvars.watchtower_notifications_level: info
com.dockstarter.appvars.watchtower_notifications: shoutrrr
com.dockstarter.appvars.watchtower_repo_pass: ''
com.dockstarter.appvars.watchtower_repo_user: ''
com.dockstarter.appvars.watchtower_schedule: 0 0 4 * * *
com.dockstarter.appvars.watchtower_timeout: 10s

And this is exactly what I want. If I don't use -y I of course get a JSON object that looks like this:

{
"com.dockstarter.appinfo.deprecated": "false",
"com.dockstarter.appinfo.description": "Automatically update running Docker containers",
"com.dockstarter.appinfo.nicename": "Watchtower",
"com.dockstarter.appvars.watchtower_cleanup": "true",
"com.dockstarter.appvars.watchtower_enabled": "true",
"com.dockstarter.appvars.watchtower_include_stopped": "false",
"com.dockstarter.appvars.watchtower_monitor_only": "false",
"com.dockstarter.appvars.watchtower_network_mode": "",
"com.dockstarter.appvars.watchtower_notification_template": "{{range .}}{{.Message}}{{println}}{{end}}",
"com.dockstarter.appvars.watchtower_notification_url": "",
"com.dockstarter.appvars.watchtower_notifications_level": "info",
"com.dockstarter.appvars.watchtower_notifications": "shoutrrr",
"com.dockstarter.appvars.watchtower_repo_pass": "",
"com.dockstarter.appvars.watchtower_repo_user": "",
"com.dockstarter.appvars.watchtower_schedule": "0 0 4 * * *",
"com.dockstarter.appvars.watchtower_timeout": "10s"
}

But in this case I do NOT want the top and bottom curly braces or the quotes around the keys.

So in conclusion I am wondering why the ... is included at the end of a single value output when using the -y flag. Is that a bug, or is there a more specific way I should be trying to get what I want?

I searched and the only possibly related issue I found was https://github.com/kislyuk/yq/issues/93 but that looks like they are intentionally expecting the ... because they included it in their original file. In my situation ... is not included in the original file, and thus I would not expect it to be included in the output.

kislyuk commented 3 years ago

This is not a bug in yq, but a peculiarity in the PyYAML parser's behavior (which yq depends on). Dumping a single scalar node in PyYAML will always produce a document end marker (...) and there is no way to disable this behavior. A fix for this issue has recently been submitted upstream (https://github.com/yaml/pyyaml/issues/379, https://github.com/yaml/pyyaml/pull/438) and once it is merged and released, yq will no longer emit ... in this situation.

nemchik commented 3 years ago

Thanks! I'll subscribe to notifications for those.

thetumper commented 3 years ago

Same issue. Looks like yq actually changed this behavior for https://github.com/kislyuk/yq/issues/93. Given that, I was trying to use sed to just strip off the explicit_end line, but as I'm a rank beginner with sed and average with bash, I couldn't make it work. Anyone have a working solution to use yq to query for a single scalar value and remove the unwanted explicit_end line folllowing the desired value?

kislyuk commented 3 years ago

The default behavior was not changed for #93. To query for a single scalar value without emitting the document end marker, just don't use -y, or use head -n -1 to cut off the last line with the marker.