search

ipld / go-car

A content addressible archive utility
Other
148 stars 44 forks source link

Progress tracker for index creation #351

Open aschmahmann opened 1 year ago

aschmahmann commented 1 year ago

Index creation can be very expensive for large CARs and it'd be nice to be able to track (and in the CLI emit) progress as it's being generated.

If https://github.com/ipld/go-car/blob/e48b3f16cfc2014e2e56dae952d04021ac66a61b/v2/index_gen.go#L43 took a context I'd probably recommend attaching the tracker there (and probably also allowing the caller to bail via context cancellation). We could potentially put it into the Options struct, but I don't know if that fits the model of what that struct is aiming for.

Any suggestions on acceptable API modifications (or new function introductions) here?

willscott commented 1 year ago

I think somewhere else we tracked progress by knowing the total length of the reader and tracking how much had been read so far.

Can you do it that way?

BigLep commented 1 year ago

During IPLD triage: do you need more input here @aschmahmann ? Do you think you'll proceed with this change?