search

nowittynamesleft / merkle-tree-models

Using zero knowledge proofs to prove model history/evolution in a tree
1 stars 0 forks source link

Make slerp work #9

Open nowittynamesleft opened 2 months ago

nowittynamesleft commented 2 months ago

the SLERP merge function is not currently able to be zk proven, was unable to calibrate before. It's been replaced with weighted average merging for now.

We need this. It might be some data type problem, or it could potentially be a problem of having operations like sin and acos which may not be able to be included in a zk proof.

nowittynamesleft commented 2 months ago

So looks like it's mainly a memory issue, since doing it for really tiny models works. Tried several smaller versions, seems to scale linearly with the number of parameters. In order to get the original 64x256x10 model working (19k params), it would need to be run with ~600GB memory according to this trend. image

However, it'd be much better if there's a way to avoid having to use so much memory. Should make sure that there's no other way that would reduce the memory requirement.

nowittynamesleft commented 2 months ago

tbh, the memory requirements need to be reduced if this is to be actually used by people. so we need to optimize the memory usage by whatever way we can, otherwise model merging with slerp would be pretty much a non-starter