Specify all the secret format we support

[AnomalRoil]

Summary

We supported more and more secret formats before recently introducing One secret format to rule them all, namely the MIME one.

We need to make sure that we document and specify all the formats that we are officially supporting.

We need to document the migration path/what happens when we start mixing the formats. When e.g. we add a new key-value to a plain text secret.

Conversion and conditions for it should be explicated.

In particular we need to document what happens if the setting mime is set to false.

And how we can switch from one format to another (maybe add a re-encrypt command that would make sure all secrets are in the right format and encrypted for the right recipients and all? This might also be useful to help people re-encrypt the secrets that used to be for just a few recipients now that this was dropped.)

[dominikschulz]

Yes, indeed. We need to make sure all formats are properly documented. I will try to write something down.

What is your concern with mime being set to false? This mode is indeed a bit hacky, but what's the point you're trying to make there?

Switching to the new format is easy, we have a command for that: gopass fsck --decrypt (this should likely be documented, I guess). But I did never test the other direction (i.e. gopass config mime false; gopass fsck --decrypt).

[AnomalRoil]

What if I edit a secret in Mime format with mime being set to false? Should it revert to another format? If so, which one?

What if I insert a key-value to a secret which is not in Mime format when mime is not set?

And so on.

Nice, I didn't know about gopass fsck --decrypt actually xD

[dominikschulz]

The implementation is very naive: https://github.com/gopasspw/gopass/blob/master/pkg/gopass/secret/mime.go#L93

If you edit a secret with mime set to false this method is used to write the output in the pass-compatible KV format (which we should certainly specify). The same applies if you insert a key.

The default now is always the new native MIME format. The mime = false setting just provides an escape hatch. It was never meant to be a fully featured and supported use case, just a compromise for users who really need it.

[dominikschulz]

This issue is much bigger than I anticipated, e.g. #1594, #1600, #1601, #1602, #1607, #1612.

When working on the MIME format I tried to come up with a format that has a well defined machine readable syntax and feels familiar to the average user (read: myself).

The problem is: Not everyone is familiar with text-based network protocols like SMTP, HTTP or MIME.

And apparently there are endless ways to populate your secrets - where I only ever use the (new) Key-Value format:

• Some people only want a dictonary-style key value mapping, i.e. map[string]string
• Others expect to be able to use the same key multiple times, map[string][]string
• Some people want to use advanced YAML features
• Others are confused by how YAML interprets unquoted numbers (e.g. foo: 0123 -> foo: 83 [octal!])
• While some people just want their secrets to be left alone entirely (e.g. #1612)
• Or they want to use the Key-Value features but still preserve their custom secrets format (i.e. parsing only for querying. I'm not sure how these users would expect mutating operations to work)

The idea with the MIME format was to enforce a uniform container layout and then allow using any kind of content inside nested multipart MIME messages. But then we'd have to teach our users to write multipart MIME headers. That's not going to work.

Actually I'm not entirely sure how to proceed on this one.

I have some ideas to mitigate the situation, but I'll need more input to come up with a good long term solution.

• I'll need to refactor the secrets implementation to make it more flexible and explicitly handle some corner cases which are currently unhandled. These are dormant bugs that might be noticed anytime. Then we could introduce options to disable secrets parsing in parts or even entirely.
• We need to define how the currently implemented formats are supposed to behave and try to get input how they should behave.

[OJFord]

What if I edit a secret in Mime format with mime being set to false? Should it revert to another format? If so, which one?

Why not just treat it as a dumb file, as pass does? (Or, I guess you could say that's text/plain?)

The problem is: Not everyone is familiar with text-based network protocols like SMTP, HTTP or MIME.

I'm relatively familiar, but we must be using (go)pass extremely differently!

[marco-m]

@dominikschulz

The idea with the MIME format was to enforce a uniform container layout and then allow using any kind of content inside nested multipart MIME messages. But then we'd have to teach our users to write multipart MIME headers. That's not going to work.

Actually I'm not entirely sure how to proceed on this one.

I think I understand the appeal of MIME for its flexibility, but I am concerned about the cognitive load and confusion that is showing up.

I would like to understand one thing: why, in general, do we think that there is a need for "structured data" (K/V, YAML, MIME) ? Why the simple approach of treating everything "raw" or plain text is not good enough ? Why not simply having one gopass entry per item instead of the "structure" ? If this has already been explained/discussed, please excuse me :-), where could I find the discussion ?

[dominikschulz]

So MIME is gone now :sweat_smile:

Why do we need structured data? Well, at some point we introduced getters for getting keys from secrets and slowly expanded on that. I like that feature and apparently other do like it as well.

Do we need it? I think that depends who you ask. I'd say yes.

[AnomalRoil]

On my side I'm using key-values (and sometimes YAML secrets) mostly to store my credentials:

s3cr3tpassword
url: website.com
username: Anomalroil

And then I'm exploiting it with a Rofi prompt to easily auto-type my website's credentials whenever I need them with a key-binding thanks to this: https://github.com/gopasspw/gopass/blob/master/docs/setup.md#dmenu--rofi-support Notice how the username key if called here:

gopass ls --flat | dmenu | xargs --no-run-if-empty -- bash -c 'gopass show -f $0 username' | head -n 1 | xdotool type --clearmodifiers --file -

So I'm also in favour of keeping the parsing at least for the KV type. But since the YAML type isn't too expansive and isn't adding too much complexity, if you ask me, I guess we could keep the 3 formats:

• plain
• key-value (we need to fix #1576 and support key-valueS, plural, at some point. I'm on it.)
• YAML (we've increase the documentation already in the latest release: https://github.com/gopasspw/gopass/blob/master/docs/commands/show.md#parsing-and-secrets)

[dominikschulz]

I believe that's what we have today and that we my understanding of what we wanted to have already.

So, yes. Let's settle on supporting: plain, key-value and yaml, where key-value would be my most and yaml my least favourite one.

[dominikschulz]

So we've settled on key-value (default), yaml (for gopass backwards compatibility) as well as plain (for pass compatibility).

They all have some documentation in the code, but should move/copy that to markdown.