supranational / supra_seal

Apache License 2.0
32 stars 21 forks source link

PC1/PC2: being able to disable TreeR building #27

Open vmx opened 1 year ago

vmx commented 1 year ago

When looking at the PC2 API, it looks like you cannot run PC2 without building TreeR. It would be great to have an option that only generates the TreeC and the sector key. The sector key is what currently is called the sealed-file.

Background on why this would be nice. When doing data sectors (as opposed to CC sectors), it could be that the data files are not accessible from the machine that runs the PC2. Therefore, you would run the PC1 process as you currently do and disable the TreeC building. You would then use the sealed-file as sector key and move that to where the data file is. You'd the run the encoding step and TreeC building there, the existing tools/tree_r.cpp to do that.

I know that building TreeC is just a small fraction of the work done during PC1/PC2, but not doing things that are not needed are always a win.

This request is not urgent, but it will eventually be needed. I though I open the issue early on, so that you can schedule it into your work stream easily.

simonatsn commented 1 year ago

Currently tree-c and tree-r are constructed concurrently since for CC sectors and when the data is present on the sealing machine they utilize the same data. This is important in those cases since the data movement is a significant part of the cost.

In PC2 tree-c is much more expensive than tree-r so in the scenario where the data is remote the potential savings from skipping tree-r is minimal.

We'll add this one to the list for future updates.

vmx commented 1 year ago

Currently tree-c and tree-r are constructed concurrently

Oh that's interesting. I though they would be built sequentially. But thinking about it, it makes sense (given the on disk data layout). Then indeed the TreeR costs are pretty small. I guess you would have a little more memory on the GPU available to do the TreeC, but that probably doesn't really matter.

vmx commented 1 year ago

On a side-channel we agreed on adding a option to not write the TreeR tree files to disk.