Redesign ciphertext file layout

[overheadhunter]

Due to lessons learned I propose the following ciphertext file layout:

.
├─ d
│  ├─ BZ
│  │  └─ R4VZSS5PEF7TU3PMFIMON5GJRNBDWA
│  │     ├─ 5TyvCyF255sRtfrIv__83ucADQ==.c9r  # regular file
│  │     ├─ FHTa55bH_sUfVDbEb0gTL9hZ8nho.c9r  # irregular file...
│  │     │  └─ dir.c9r  # ...which is a directory
│  │     ├─ gLeOGMCN358_UBf2Qk9cWCQl.c9r  # irregular file...
│  │     │  └─ symlink.c9r  # ...which is a symlink
│  │     ├─ IjTsXtReTy6bAAuxzLPV9T0k2vg=.c9s  # shortened name...
│  │     │  ├─ contents.c9r  # ...which is a regular file
│  │     │  └─ name.c9s  # ...mapping to this full name
│  │     ├─ q2nx5XeNCenHyQvkFD4mxYNrWpQ=.c9s  # shortened name...
│  │     │  ├─ dir.c9r  # ...which is a directory
│  │     │  └─ name.c9s  # ...mapping to this full name
│  │     ├─ u_JJCJE-T4IH-EBYASUp1u3p7mA=.c9s  # shortened name...
│  │     │  ├─ name.c9s  # ...mapping to this full name
│  │     │  └─ symlink.c9r  # ...which is a symlink
│  │     └─ ...
│  └─ FC
│     └─ ZKZRLZUODUUYTYA4457CSBPZXB5A77
│        └─ ...
├─ vault.cryptomator  # renamed masterkey file to stop people from asking if it is save to sync ;)
└─ vault.cryptomator.DFD9B248.bkup

• filenames only consist of the SIV-encrypted cleartext name + file extension (e.g. .c9r standing for "cryptomator")
• Any irregular file is actually a directory, which contains further files depending on the type
  • We can now atomically create a ciphertext file, which will fail, if a symlink or directory with the same name already exists, as all map to the same exact ciphertext name.
  • The contents of this directory can contain any structured data, e.g. JSON-formatted, which is needed to fulfil the purpose of the file
  • Only irregular files can create a L2 conflict, which can be resolved within this directory
• c9r files (!) contain the file contents encrypted using GCM (96 bit IV, 128 bit tag), which is now also hardware-accelerated since Java 9.

[buddydvd]

Elegant solution.

Just curious— why store metadata files in the JSON format? Giving them a .json filename extension implies they are user-editable. I see a few issues:

• You can't binary-compare files for equivalence in your conflict resolution logic
• User-edits can introduce variations BOM marker, CR/LF EOL markers, UTF-8/16, and etc
• Slower parsing speed

[overheadhunter]

JSON is just an example, maybe it isn‘t even necessary. For example directories merely constist of a unique ID, which can be any binary pseudorandom content.

We can even extend the directory layout in further versions, e.g. to add xattr stored in separate files.

[admdly]

Glad to hear that the directory structure / architecture is being reconsidered. Based on my experience, the current version is far too error prone and easily corrupted. Perhaps this is due to client issues (Cyberduck/Mountain Duck), or some other sync issue, but it is important to remember that most cloud sync services are far from reliable.

Personally, I'd like to see a focus on robustness and fault tolerance in any redesign. In my case, the whole thing falls over as soon as deep directory structures or many small files are involved. Data integrity is extremely important, and I include metadata and directory structure in this, and it's hard to trust Cryptomator in its current form.

The suggested redesign looks like an improvement, but I believe there is potential for a more robust design better suited to the nature of cloud storage. I don't have any answers or suggestions at the moment, but I'd love to contribute some suggestions once I've had time to test some ideas out.

[admdly]

Thinking about this, briefly, I would like to see each file be 'standalone' to avoid any corruption issues. It shouldn't be too difficult to include the file name and relative path in the file header; thus if the directory structure is ever mangled, there is a way to full recovery. As it is, each file appears to need re-encrypting with every move, so this isn't adding much/if any overhead.

[tobihagemann]

As it is, each file appears to need re-encrypting with every move, so this isn't adding much/if any overhead.

This is not true. The file content is neither dependent on the filename nor on the directory structure.

We decided against this because it would make things reeeaaally complex and actually more error-prone. Some of the discussion is here: https://github.com/cryptomator/cryptomator/issues/119

[admdly]

As it is, each file appears to need re-encrypting with every move, so this isn't adding much/if any overhead.

This is not true. The file content is neither dependent on the filename nor on the directory structure.

We decided against this because it would make things reeeaaally complex and actually more error-prone. Some of the discussion is here: cryptomator/cryptomator#119

This is news to me; I may be incorrect, but it seems that the vaults easily forget their structure and require restoring. I'm curious why including the filename and path in the file header would be a bad thing.

[tobihagemann]

I would say the main argument is that it wouldn't work with the mobile apps. It's not practical if a rename/move would mean that you have to fully download and upload the file. You cannot "just update" the file header using the cloud storage APIs.

[admdly]

I would say the main argument is that it wouldn't work with the mobile apps. It's not practical if a rename/move would mean that you have to fully download and upload the file. You cannot "just update" the file header using the cloud storage APIs.

Fair play, I'd agree. But doesn't the current method link file to parent directory? I'd fully support remote moves, but I don't think that's the current way things work.

[tobihagemann]

Currently, the filename is dependent on the parent directory (or the parent directory ID to be more exact), see https://cryptomator.org/security/architecture/#nameEncryption. We did that because of https://github.com/cryptomator/cryptomator/issues/128.

[admdly]

Currently, the filename is dependent on the parent directory (or the parent directory ID to be more exact), see https://cryptomator.org/security/architecture/#nameEncryption. We did that because of cryptomator/cryptomator#128.

Then what is the harm in linking a file to a directory / path? The current scheme suggests a filename needs to be recalculated if the directory is change.

My mistake, I can see what you mean now. It is currently possible to move / rename a file without downloading the file itself; simply by moving the file or renaming it. I can understand why this would be preferable and why including this in the file header itself wouldn't be very practical.

[admdly]

I'm not sure, if any, progress has been made toward this. But I still feel that Cryptomator is too fragile for every day use. I'm curious if there has been any format changes toward robustness?

[overheadhunter]

@admdly All of the directory layout changes have been implemented in 1.9.0. The only thing that didn't change is the file content encryption, which is still CTR+HMAC rather than GCM.

Since other security researchers and pentesters testify our cryptographic design an exceptional level of robustness, I'm curious what makes you come to this highly scientific conclusion that the layout is fragile. Since I'm pretty sure you are aware of the fact that this is a platform for developers and not a Q&A forum for trolls, I guess you have some valuable input to add to improve the format for CryptoFS 2.0.

[admdly]

Since other security researchers and pentesters testify our cryptographic design an exceptional level of robustness, I'm curious what makes you come to this highly scientific conclusion that the layout is fragile. Since I'm pretty sure you are aware of the fact that this is a platform for developers and not a Q&A forum for trolls, I guess you have some valuable input to add to improve the format for CryptoFS 2.0.

@overheadhunter I apologise if my comment came across as trolling; this is not my intention. Perhaps fragile was the wrong word to use.

I understand that the cryptographic design has been vetted and deemed secure, and my comment does not refer to the security of the design in any way. I based my comment on experience using Cryptomator, which unfortunately has resulted in vault corruption in many cases. This typically is evidenced by files going missing from view (but still being in the underlying folder structure), and loss of directory structure.

I have been considering different ways of ensuring greater robustness to such issues, though I'm not sure if they would fit in with the current design. Therefore, I was querying if any changes had been made before investigating this further and making suggestions.

[overheadhunter]

I based my comment on experience using Cryptomator, which unfortunately has resulted in vault corruption in many cases.

So far you did not explain what issue you actually have. Others did so and therefore we were able to reproduce and fix the issue. For example some people were experiencing disappearing files when using Google Drive and renaming a file from a very long name to a shorter one.

People created a bug report and describing exactly what sequence of actions caused the problem, what their setup looked like and especially what third party tools were involved.

You, on the other hand, vaguely commented about some generic fragility on an issue that is about redesigning the cryptographic mapping between ciphertext and plaintext files.

I hope you understand, that this is not exactly helpful. Neither for this issue, nor does it in any way fix your (still undefined) problems. Sadly, GitHub doesn't allow extraction of comments into separate issues. But I'd kindly ask you to keep topics clean. If you're not yet able to formulate a concrete bug report, our forum might be the better place to narrow down any problems in a discussion.

[kojid0]

Hey there, coming from Boxcryptor and switching to Cryptormator. Nice work so far!

I want to ask if you could add a cryptormator-icon for .c9r files so that new users will immediately see that this file belongs to cryptormator? It might prevent non-tech users to think it's something "malicious" hence prevent accidental deletion. Just a thought to make UX more intuitive :)

[overheadhunter]

Hey @jwsp1, thanks for your suggestions, please create a feature request for this on the main repo.

[overheadhunter]

Closing this in favour of #64 and #94.