This PR adds support for migrating IPFS blocks from an input file list. It also introduces both an offset and a limit that can be used to paginate migrations, whether from a directory or from a file list.
This is needed for being able to apply incremental ZFS snapshots without having to re-migrate the entire blockstore. The list of added blocks can be extracted from the incremental snapshot and input to the migration so that only new blocks are migrated. This will dramatically reduce the amount of time needed for migrating new blocks added after the first snapshot was taken.
Note that paginating over a directory might have unpredictable results if it is receiving live updates.
This PR adds support for migrating IPFS blocks from an input file list. It also introduces both an
offsetand alimitthat can be used to paginate migrations, whether from a directory or from a file list.This is needed for being able to apply incremental ZFS snapshots without having to re-migrate the entire blockstore. The list of added blocks can be extracted from the incremental snapshot and input to the migration so that only new blocks are migrated. This will dramatically reduce the amount of time needed for migrating new blocks added after the first snapshot was taken.
Note that paginating over a directory might have unpredictable results if it is receiving live updates.