mirage / ocaml-tar

Pure OCaml library to read and write tar files
ISC License
54 stars 34 forks source link

Implement allocate and set_partial #105

Closed reynir closed 1 year ago

hannesm commented 1 year ago

there are some CI test failures... would you mind to look into them?

hannesm commented 1 year ago

reviewed and implemented rename pairing with @reynir (a4b619b). good to go (though tests would be nice)

reynir commented 1 year ago

Besides the added tests I tested as well with a version of opam-mirror that exercises allocate, set_partial and rename. I tested on unix target with the default sector size (I believe it is 512), and hvt with sector sizes 512, 2048 and 4096.

What opam mirror does

The unikernel checks out the git repository https://github.com/ocaml/opam-repository/ and fetches all archives (save for some known bad links) and save them to a tar filesystem. If the size is known ahead of time (http server responds with Content-Length: len) we call allocate with the size and a path inside /pending/... and then incrementally write the contents as we receive them using set_partial. Finally, once the file has been download we either commit by renaming the file to its sha256 sum or if the checksum doesn't match with the expected we rename it to /to-delete/....

About 14.5k archives are downloaded. Of those, about half of them have a Content-Length header in the http response. That is, about half exercise allocate, set_partial and rename. The remaining just use set.

The corrupt archives in unix are worrying, but I suspect they are not related to the tar-mirage changes.

Unless any concerns are raised I intend to merge this PR later today.