search

in-toto / go-witness

Go implementation of witness
Apache License 2.0
22 stars 18 forks source link

[Enhancement]: Improving the speed of the material and product attestor #360

Open matglas opened 3 days ago

matglas commented 3 days ago

Describe the solution you'd like:

When hashing 10000's of files small and big for both material an product it takes time. Currently the hashing is done in serial while parallel hashing thru threads would speed it up.

User value:

The amount of time spend 'extra' to do the attestation in a pipeline goes down.

Expected behavior:

Material and product hashing should be fast and have little impact on the overall pipeline that a witness execution is part of.

Proposed solution:

Implement threading and a worker queue to process files at the speed of the amount of processors available.

Anything else you would like to add:

I have done a test earlier that showed a 10x speed increase.

Testing changes required:

TBD

Documentation changes required:

TBD

adityasaky commented 1 day ago

I have done a test earlier that showed a 10x speed increase.

I thought this seemed familiar! Do you have a link handy to that?

matglas commented 21 hours ago

Yes I do. https://github.com/matglas/go-hash-walk