json decoding does not seem to properly respect character encoding

oboukili commented 5 years ago

Hello, I'm using sops 3.2.0 with gcp kms. I noticed that decrypting using json input type does not respect (decode) the keys and values' text encoding.

It is easy to reproduce for the '&' character (https://www.fileformat.info/info/unicode/char/0026/index.htm):

'testing1' file contents

{
  "key&?abcd":"value?p1=true&p2=false"
}

Encrypt as a json input type file and a binary input file

sops --encrypt --input-type=json --output-type=json --gcp-kms projects/someproject/locations/global/keyRings/sops/cryptoKeys/sops-key testing1 > testing1.json.enc  

sops --encrypt --input-type=raw --output-type=raw --gcp-kms projects/someproject/locations/global/keyRings/sops/cryptoKeys/sops-key testing1 > testing1.raw.enc

Decrypt the raw encrypted version (ok)

sops -d --input-type=raw --output-type=raw testing1.raw.enc

{
  "key&?abcd":"value?p1=true&p2=false"
}

Decrypt the json encrypted version (not ok)

sops -d --input-type=json --output-type=json testing1.json.enc

{
    "key\u0026?abcd": "value?p1=true\u0026p2=false"
}

autrilla commented 5 years ago

Semantically, they are the same:

autrilla@iron:/Users/autrilla/Projects/go/src/go.mozilla.org/sops:master# cat test.json
{
  "key&?abcd":"value?p1=true&p2=false"
}
autrilla@iron:/Users/autrilla/Projects/go/src/go.mozilla.org/sops:master# sops -e test.json > test.enc.json
autrilla@iron:/Users/autrilla/Projects/go/src/go.mozilla.org/sops:master# sops -d test.enc.json | jq
{
  "key&?abcd": "value?p1=true&p2=false"
}

We could turn it off through https://godoc.org/encoding/json#Encoder.SetEscapeHTML, but I'm not really sure we should. There could be people relying on this.

oboukili commented 5 years ago

@autrilla thanks for your answer, how about an optional flag that we could enable through cli parameters or environment variable?

autrilla commented 5 years ago

I'd be okay with that. Although I think making this breaking change would be fine, since we really don't make any guarantees about the style of the outputs in any of our formats.

xakraz commented 4 years ago

Hi guys !

What is the status of this issue ? Thx !

ifraixedes commented 3 years ago

I'd like to know if this feature would require a lot of work. I think it shouldn't be I'm thinking blind because I haven't checked the source code.

If it wouldn't require a lot of work, I may think to work on it, however, I wouldn't commit to any time to finish it.

Moskovych commented 3 years ago

Is there any progress? Somebody is working on it? Might be it is included in some milestone for next releases?

ifraixedes commented 3 years ago

@Moskovych I was waiting for a reply from the owning team to spot some time to work on it because I wanted to make sure that if I spend time on it it isn't going to overlap the work with someone else and the changes will be fine to be merged.

Moskovych commented 3 years ago

@ifraixedes , ok, but I've already started working on it: https://github.com/mozilla/sops/pull/887. As discussed in https://github.com/mozilla/sops/issues/881 it will be "breaking" (without semantic) changes and without the flag, as is. Just waiting for the tests fix, and I'll proceed with adding tests, to cover that case.

ifraixedes commented 3 years ago

@Moskovych oh good, sorry for not responding earlier, normally I do in less than 2 days, but I was on holiday for 3 weeks and mostly away from my workstation.

vborcea commented 1 year ago

Hello, is there any progress on this?

kvishweshwar commented 6 months ago

Any updates?

can we use single quote/ literals (around password string) at the time of encryption and remove that single quote at the time of decryption? does this approach cause an issue with other functionalities?

felixfontein commented 6 months ago

@kvishweshwar are you talking about modifying the content? Or the encoding as JSON (or other formats)?

getsops / sops

json decoding does not seem to properly respect character encoding #464