oll3 / bita

Differential file synchronization over http
https://crates.io/crates/bita
MIT License
265 stars 10 forks source link

Create create_archive function + associated types #20

Closed rminderhoud closed 2 years ago

rminderhoud commented 2 years ago

Adds a new module called api. This module is designed to mimick the high level of functionality of bita but make to make it available programmatically using bitar.

Currently, only one method is implemented, which is create_archive. This is the same logic used from the bita bita compress command with some logic cleaned up and some changes to make it suitable as a library function.

This change moves tempfile from dev dependency to a hard dependency in bitar. This is needed to write archive data to a temporary file before prepending the header. Additionally the num_cpus dependency was added to set the default buffer count when chunking.

oll3 commented 2 years ago

Nice one, I like it! Think this make sense and simplifies usage of the library in archive creating applications.

Will have a closer look when I have more time but some initial thoughts here anyway...

So far the "compress" feature flag has been used to remove code and dependencies when an application is only in need of the cloning part of bitar. This mainly to decrease build time and possibly binary size for applications which only need to clone. Since my primary usage of bitar is in software update of embedded systems (devices which may be limited both in memory, CPU performance and network bandwidth) I want to keep binary bloat minimal.

With that in mind I'm thinking that the compress feature flag could possibly be renamed to something like 'create archive' flag. And then have all code related to archive creation living under it, like most of this PR's code. Both to better reflect its intent and to keep build time and binary size down. Make sense?

Also thinking that some kind of test could be in place to validate that this api creates valid archives.

rminderhoud commented 2 years ago

Ah, I misunderstood that feature flag. I'll change it so the entire thing is behind the feature flag. I think the name "compress" is fine, I just misunderstood its usage. Perhaps can simply be documented in the README?

For testing, there's two options I think: 1) Commit a set of known good archives (e.g. one per hashing algorithm + compressed/nocompressed) and compare output 2) Just check the header of the output file?

My preference is (1) if you would be able to provide the files then I can write the test. Alternatively, I can just generate a few on my end if you don't care what the original one is.

rminderhoud commented 2 years ago

@oll3 I've fixed up the usage of the compress flag. Now the whole module won't be included by default unless the flag is set. Additionally I adjusted the dependencies I pulled in to only be included when the feature flag is set. Finally, I added some tests for the various compression scenarios.