MatrixAI / Polykey-CLI

Polykey CLI - Open Source Decentralized Secret Sharing System for Zero Trust Workflows
https://polykey.com
GNU General Public License v3.0
5 stars 3 forks source link

Incorporate Unix Commands to Secrets Subcommand - `pk secrets *` #32

Open CMCDragonkai opened 2 years ago

CMCDragonkai commented 2 years ago

Specification

Standard unix commands include things like cp, mv and ls.

This allows users to interact with secret vaults as if they were real filesystems. (And they sort of are).

Imagine (IN ORDER OF PRIORITY):

# these 2 might be merged into 1 command just `write` is sufficient
pk secrets read vault1:/a/b
pk secrets write v1:/abc

pk secrets cat vault1:/a/b
pk secrets touch vault1:/a/b
pk secrets ls vault1
pk secrets mkdir vault1:/c
pk secrets rm vault1:/a/b
pk secrets mv vault1:/a/b vault1:/a/c
pk secrets cp vault1:/a/b vault1:/a/c
pk secrets ln vault1:/a/b vault1:/d/e

# this one is special
pk secrets env

# this one is also special
pk secrets ed vault1:/a/b

# these are less important
pk secrets head vault1:/a/b
pk secrets tail vault1:/a/b
pk secrets find vault1
pk secrets grep vault1:/a/b
pk secrets sed vault1:/a/b

I believe some of these commands were already implemented before in the old PK codebase. You have things like pk secrets create and pk secrets delete.

»» ~/Projects/js-polykey/src/bin/secrets
 ♖ tree .                                                                                                          (client-refactoring) pts/5 14:47:05
.
├── create.ts
├── delete.ts
├── dir.ts
├── edit.ts
├── env.ts
├── get.ts
├── index.ts
├── list.ts
├── mkdir.ts
├── rename.ts
└── update.ts

0 directories, 11 files

I believe there's a problem with doing this. We are reinventing the wheel, and we'll never cover all the commands that Unix already has.

This has the benefit of reusing context that developers already know and remember when interacting with a Unix shell.

But how do we do this without having to rewrite all the code? Luckily it seems someone has already done this.

See: https://github.com/shelljs/shelljs

It has implemented all the major Unix shell commands in raw JS.

The only problem that their command source code currently directly imports the native Node fs:

https://github.com/shelljs/shelljs/blob/79ae14d30d7ce4064de05d41c7889885326b6754/src/ls.js#L2

If we would want to use the shelljs library, we would need to globally mock the fs as described here: https://github.com/shelljs/shelljs/issues/747#issuecomment-316301410

However that may be dangerous if that leaks into other places of the FS.

There is another alternative: https://github.com/dthree/cash. The difference between the 2 are: https://github.com/dthree/cash#doesnt-shelljs-do-this However I think it's even less integratable compared to shelljs.

So it seems we would need to "extract" the command code from shelljs and place it into PK directly and thus enable us to change the fs object to our Vault EFS.

Note that we do not need all shell commands, just the major ones that relate to files, and doesn't change cwd context since we don't use that. Nothing that changes permissions is relevant to us. Process control is also not relevant.

One major difference is that our commands will have to traverse both vault filesystems and the real filesystem. For example pk secrets mv vault1:/a ./a which has to take a file from a vault to the real fs. The functionality to achieve this will also end up being used to do commands between vaults. Because the efs.mv won't work between EFS instances anyway.

Additional context

Tasks

  1. ...
  2. ...
  3. ...
CMCDragonkai commented 2 years ago

With regards to pk secrets cat and "writing a file" to a secret vault. While the cat command can be used to read out a secret, what's more difficult is putting back a secret that can be composed with other commands. Here I propose a null-command or a variant of pk secrets that opens a readable file descriptor.

Here are some examples:

# using process redirection, as pk secrets v1:/a would need to take a FD
echo 'somesecret' > >(pk secrets v1:/a)
printf 'somesecret\n' > >(pk secrets v1:/a)
echo 'somesecret' | pk secrets v1:/a

# echo 'somesecret' > pk secrets v1:/a # <- this will not work as pk secrets is a comand, not a filepath

# compare with shell commands like:
echo 'somesecret' > >(cat > abc)
echo 'somesecret' | cat > abc

Note that pk secrets v1:/a has no subcommand under secrets. It would need to open a file descriptor by attempting to read from STDIN. This takes over from the shell command cat > abc which runs cat without a file path argument which causes it to read from STDIN. However is not usable in PK since we aren't running process redirection operators > and < from inside the PK command. That is PK is not a shell. Therefore we run something different:

# starts reading from STDIN immediately
pk secrets v1:/a

As well as the ability to use cat subcommand to output to STDOUT.

# read v1:/abc and pipe it to cat, pretty simple
pk secrets cat v1:/abc | cat

At this point, one might argue that one should enable pk secrets v1:/a to be a writable file descriptor as well. That is both STDIN and STDOUT may be usable.

cat <(pk secrets v1:/abc)

Which would take over from the cat subcommand.

However I'm not sure I really like the usage of pk secrets v1:/abc as a universal file descriptor. It's nice in one way, but it seems such a "default" command to be relatively hard to use and hard to learn.

Consider that we also need another command that one can just use to put a secret in without the need to wire up other commands.

pk secrets put v1:/abc 'abc'
pk secrets write v1:/abc 'abc'
pk secrets print v1:/abc 'abc'

And as we have talked about before, we need 3 ways:

I propose a special command for this:

# this will read from STDIN, and write to v1:/abc (like reading into a secret)
pk secrets read v1:/abc

# pipe into a file descriptor
echo 'secretdata' | pk secrets read v1:/abc
pk secrets read v1:/abc <<<'secretdata'
echo 'secretdata' > >(pk secrets read v1:/abc)

So then read is the opposite of cat.

But still one more command that allows one to write a secret in easily. Here are our 3 candidates.

pk secrets put v1:/abc 'abc'
pk secrets write v1:/abc 'abc'
pk secrets print v1:/abc 'abc'

From these 3, I quite like pk secrets write. It's the opposite of read, it actually can function very similarly if pk secrets write v1:/abc was allowed. That is with no arguments, it acquires from the STDIN. In that sense, there's no need for pk secrets read at all then as it is just confusing. So write is probably the correct keyword.

CMCDragonkai commented 2 years ago

Some commands here due to usage over GRPC will involve client streaming or even duplex streaming. Deadlines will be important here. And if a command gets cut in the middle, the data written to EFS should be removed. That might be good idea to per-vault lock to do this. Until of course we can create a "vault"/efs snapshot. The beginnings of a vault/efs snapshot is already done in https://github.com/MatrixAI/js-encryptedfs/issues/49 but it would need to be exposed to the PK vaults system. Perhaps that can help with enabling COW for the vaults MatrixAI/Polykey#172 MatrixAI/Polykey#180.

CMCDragonkai commented 2 years ago

Now that MatrixAI/Polykey#266 has fixed up vaults related commands, the next step is a proper review of all the secret commands, and hopefully finish up our pk secrets env PR MatrixAI/Polykey#265 so it can be merged.

CMCDragonkai commented 3 months ago

@CryptoTotalWar this is relevant for your review of PK CLI on Mac.

CryptoTotalWar commented 1 month ago

Proposal for Streamlining Secrets Creation Using Unix Piping Paradigms

Concept:

Leverage Unix-like command paradigms to streamline the process of creating secrets directly into the vault, bypassing the need to first create a file on the local filesystem. This would enhance the functionality of Polykey by allowing direct input of secrets via standard input (stdin), aligning with Unix principles of simplicity and flexibility.

Details:

Instead of the current method where a file must exist beforehand, we could introduce functionality where users can pipe data directly into a Polykey secret path. This approach avoids unnecessary steps and strengthens security by not requiring sensitive data to touch the filesystem.

Suggested Command Enhancement Example:

echo 'your_secret_data' | polykey secrets write vaultName:/path/to/secret

This command would take the output from echo and directly write it to the specified secret path in the vault. It simplifies the workflow, especially in scripting and automation scenarios, where secrets need to be dynamically generated and stored.

Benefits

I believe this enhancement would significantly improve the user experience by making secret management more seamless and integrated with standard command line operations. It encourages best practices in handling sensitive data.