Open Shrinks99 opened 7 months ago
Renamed slightly to avoid confusion with https://github.com/webrecorder/browsertrix-cloud/issues/890
Most major clients don't support mutable torrents. Similarly while I got desktop webtorrent to work with them it's very slow and it cannot work in browsers due to a lack of dht. There's also a variation on mutable torrents where a newly downloaded torrent file can "update" on top of an older one. This doesn't do automatic updates but I believe it's more widely supported.
I was asked to grab some large crawls out of our account since we hit our quota. I ended up writing an adhoc Python utility to do this, but along the way a few things occurred to me:
orgs/{org_id}/crawls/{crawl_id}/replay.json
the best place to find the list of WACZ files?Hi Ed, this is really cool! Thanks for sharing :)
- it would be nice for the script to have some options to limit what was copied
- maybe lean on rclone to be able to more than just s3?
rclone is very flexible! I think you'd need the credentials of the Browsertrix S3 bucket if you wanted to copy files directly from there though, which we likely won't want to share out for our production app. We use rclone internally in Browsertrix in the background jobs to replicate content from the primary s3 bucket to backups/replica locations. Still worth thinking through the possibilities though!
- is it feasible to build a better browsertrix client from the openapi doc?
That's a great idea, Python and/or Node Browsertrix API client libraries could be very useful for use cases like this! Definitely worth looking into to what degree we could use the OpenAPI docs to help with that.
- is
orgs/{org_id}/crawls/{crawl_id}/replay.json
the best place to find the list of WACZ files?
That is generally the best endpoint if you already have the crawl ids or have gathered them from other crawl endpoints such as orgs/{org_id}/crawls
. One note, if you use orgs/{org_id}/all-crawls/{crawl_id}/replay.json
, then the same endpoint will work for both crawls and uploads as long as you have the right id for the crawl/upload, whereas orgs/{org_id}/crawls/{crawl_id}/replay.json
or orgs/{org_id}/uploads/{crawl_id}/replay.json
will only work for that archived item type. Similarly, orgs/{org_id}/all-crawls
will return both crawls and uploads in the same paginated list, and you can tell which is why by the type
field value.
Thanks @tw4l! I thought maybe rclone could be used programmatically to pull the set of signed URLs instead of a bucket, and then it could write to the many endpoints it already supports?
https://rclone.org/commands/rclone_copyurl/
But it wasn't a fully formed thought.
Thanks @tw4l! I thought maybe rclone could be used programmatically to pull the set of signed URLs instead of a bucket, and then it could write to the many endpoints it already supports?
Ah, that's a good call! I misunderstood but makes a lot of sense to me :)
Thanks for sharing this @edsu. I think #578 will essentially provide built-in support for this, we hope to get to it soon. The idea would be to allow users' to switch primary and possibly secondary storage to a bucket of their choosing, which would use rclone internally to copy all of the data. There's still a few things to figure out with that, but should generally be doable. I think that will probably cover your specific use. Yeah, perhaps worth thinking how it could be made more flexible - we were thinking of this as a one-time switch, rather than periodic backups with filtering, eg. bring-your-own bucket instead of using ours.
Context
Our service generates data but right now we require users to click a lot of stuff to download it all. Given the ease of mass data creation we should facilitate a similar level of ease of use when it comes to mass data export.
Possible methods of downloading data