harvard-lil / perma

Indelible links
417 stars 71 forks source link

add additional ways to select sample warcs #3506

Closed teovin closed 5 months ago

teovin commented 5 months ago

Add the below features so we can mix up the sample selection:

Sample usage:

docker compose exec web invoke dev.benchmark-wacz-conversion --benchmark-log='perma/wacz_experiment/benchmark.csv' --batch-range=3000:4000 docker compose exec web invoke dev.benchmark-wacz-conversion --benchmark-log='perma/wacz_experiment/benchmark.csv' --prefix=A2 docker compose exec web invoke dev.benchmark-wacz-conversion --benchmark-log='perma/wacz_experiment/benchmark.csv' --big-warcs