kislyuk / yq

Command-line YAML, XML, TOML processor - jq wrapper for YAML/XML/TOML documents
https://kislyuk.github.io/yq/
Apache License 2.0
2.57k stars 82 forks source link

Add support for input_format="json" #121

Closed shawwn closed 3 years ago

shawwn commented 3 years ago

This PR adds support for yq.yq(input_format="json", output_format="toml").

In my case, I added this was because I wanted to turn a GitHub REST API response (json) into toml, and discovered there seems to be no way to parse json as input.

For me, it was super useful to be able to turn arbitrary json trees into toml files.

The tests seem to pass. If I missed anything, let me know.

shawwn commented 3 years ago

In addition, I've added support for --input-format and --json-input/-j options.

Now you can do everything with yq, if you wanted to. It feels lovely to use.

$ echo '{"bar": [{"foo": [1,2,3]}, {"baz": 99}]}' | yq . --input-format json --output-format toml
[[bar]]
foo = [ 1, 2, 3,]

[[bar]]
baz = 99
gerardolima commented 3 years ago

On README, we can read the following:

Because YAML treats JSON as a dialect of YAML, you can use yq to convert JSON to YAML: yq -y . < in.json > out.yml.

Is this really necessary?

kislyuk commented 3 years ago

Thanks for your interest in yq. This PR will not be merged because JSON input is already handled as a dialect of YAML.

kislyuk commented 3 years ago

P.S. If your aim is to convert JSON to TOML, you can do so with yq --toml-output . input.json.

shawwn commented 3 years ago

No no, my goal was the other way around! This PR lets you convert toml to json. You can't do that with current yq:

$ curl -fsSL "https://pypi.org/pypi/walrus/json" | yq -t | yq -t
yq: Error running jq: ScannerError: mapping values are not allowed here
  in "<stdin>", line 23, column 38

It's highly useful to be able to parse toml. In fact, the first thing I wanted to do with this was to parse Poetry pyproject.toml files. I've been using my local yq for so long that I assumed this PR got merged in the meantime, because it's been that handy:

image

You're right; I didn't explain that at all. Sorry. Could you reconsider merging this PR?

shawwn commented 3 years ago

Oh, I see. yq installs a wrapper called tomlq, and you're supposed to use that instead when the input type is toml.

I guess that's okay... It would've been nice to do everything with yq for simplicity, since this PR would help it support any combination of input formats and output formats, but I guess you're right about this PR being unnecessary.

shawwn commented 3 years ago

Regarding yq.yq, I see what you mean now -- I was confused that you can't call it with input_format="json", but as you say, you don't need to; input_format="yaml" works fine for json input, which was surprising.

Ok, I'll just use tomlq and xq and yq binaries on the command line then, depending on the file type I'm working with. I guess it's no harder than having an --input-format option, even if it's a bit strange to use different binaries.

Thanks!