Add remaining commands

[prestwich]

Step 1: determining which commands need to be added. :)

[Miskerest]

Some commands are there, but are not implemented or completely functional. Right now, major commands I can think off of the top of my head:

• File encryption
• Sharding (implemented, but hash160 calculations need work)
• Adding shards to a frame
• Deleting the local file (?) after upload

All of these functions are used within file_upload, so without them file upload will be at a roadblock. I think for now we can hold off on file encryption, and focus on getting upload/download done. We can implement encryption later.

Bridge is down right now, but do we know if file_download works?

[prestwich]

I wouldn't delete the local file after uploading.

As far as I can tell, file_download works. I tested it in the wild with a file I had uploaded with the Node CLI.

Speaking of encryption and decryption, the way that is done client side recently changed. The Node library now uses the bucketID and filename with a client-side passphrase to generate a deterministic key. This means users can easily re-generate keys on different devices using the passphrase. We will need to replicate this functionality.

The user's private key should be stored in ~/.storj/id_ecdsa.

When we store keys, they should should go in ~/.storj/key.ring/{file_ID}.

[prestwich]

Next steps, imho:

Document how the node client implements the following:

• [ ] Sharding
• [ ] Audit tree generation
• [ ] Storing and loading private keys
• [ ] Generating file keys deterministically, and storing them locally

By documenting those, we can replicate the functionality exactly, which is very important.

I would highly recommend we right tests using real output from the Node libraries, to be sure we exactly replicate functionality.

Resources:

Sharding: https://github.com/Storj/core/blob/master/lib/file-handling/file-demuxer.js

Audits https://github.com/Storj/core/blob/master/lib/audit-tools/audit-stream.js https://github.com/bookchin/mtree/blob/master/lib/tree.js

Keypairs: https://github.com/Storj/core/blob/master/lib/crypto-tools/keypair.js

File keys: https://github.com/Storj/core/blob/master/lib/crypto-tools/keyring.js

[prestwich]

Audits

1. Generate n challenges
2. Create a list of inputs primed with the challenge data.
3. Fill the challenge list with rmd160(sha256('')) until its length is a power of two
4. Update the primed ResponseInputs with the data
5. Generate the tree from the inputs

Things to note The data is hashed before entering the merkle tree. These pre-images are hashed again to make the leaves. leaf = hash(hash(challenge + data)). See flow in next section.

Flow for an input

input = crypto.createHash('sha256').update(challenge)  # L140
input.update(bytes.toString('hex'))   # L76
input = utils.rmd160sha256(utils.rmd160(input.digest('hex'))) # L121

Current implementation https://github.com/Storj/storj-python-sdk/blob/master/storj/model.py#L265-L275

Recommendation

1. Break out a new MerkleTree class. Use MerkleTree objects for shard.tree.
2. Implement the getPublicRecord and getPrivateRecord functions found here.

[RichardLitt]

👋 Hey! Thanks for this contribution. Apologies for the delay in responding!

We've decided to rearchitect Storj, so that we can scale better. You can read more about this decision here. This means that we are entirely focused on v3 at the moment, in the storj/storj repository. Our white paper for v3 is coming very, very soon - follow along on the blog and in our Rocketchat.

As this repository is part of the v2 network, we're no longer maintaining this repository. I am going to close this for now. If you have any questions, I encourage you to jump on Rocketchat and ask them there. Thanks!