webrecorder / browsertrix

Browsertrix is the hosted, high-fidelity, browser-based crawling service from Webrecorder designed to make web archiving easier and more accessible for all!
https://webrecorder.net/browsertrix
GNU Affero General Public License v3.0
200 stars 33 forks source link

Mass org file export #1647

Open Shrinks99 opened 7 months ago

Shrinks99 commented 7 months ago

Context

Our service generates data but right now we require users to click a lot of stuff to download it all. Given the ease of mass data creation we should facilitate a similar level of ease of use when it comes to mass data export.

Possible methods of downloading data

tw4l commented 7 months ago

Renamed slightly to avoid confusion with https://github.com/webrecorder/browsertrix-cloud/issues/890

RangerMauve commented 7 months ago

Most major clients don't support mutable torrents. Similarly while I got desktop webtorrent to work with them it's very slow and it cannot work in browsers due to a lack of dht. There's also a variation on mutable torrents where a newly downloaded torrent file can "update" on top of an older one. This doesn't do automatic updates but I believe it's more widely supported.

edsu commented 3 months ago

I was asked to grab some large crawls out of our account since we hit our quota. I ended up writing an adhoc Python utility to do this, but along the way a few things occurred to me:

tw4l commented 3 months ago

Hi Ed, this is really cool! Thanks for sharing :)

  • it would be nice for the script to have some options to limit what was copied
  • maybe lean on rclone to be able to more than just s3?

rclone is very flexible! I think you'd need the credentials of the Browsertrix S3 bucket if you wanted to copy files directly from there though, which we likely won't want to share out for our production app. We use rclone internally in Browsertrix in the background jobs to replicate content from the primary s3 bucket to backups/replica locations. Still worth thinking through the possibilities though!

  • is it feasible to build a better browsertrix client from the openapi doc?

That's a great idea, Python and/or Node Browsertrix API client libraries could be very useful for use cases like this! Definitely worth looking into to what degree we could use the OpenAPI docs to help with that.

  • is orgs/{org_id}/crawls/{crawl_id}/replay.json the best place to find the list of WACZ files?

That is generally the best endpoint if you already have the crawl ids or have gathered them from other crawl endpoints such as orgs/{org_id}/crawls. One note, if you use orgs/{org_id}/all-crawls/{crawl_id}/replay.json, then the same endpoint will work for both crawls and uploads as long as you have the right id for the crawl/upload, whereas orgs/{org_id}/crawls/{crawl_id}/replay.json or orgs/{org_id}/uploads/{crawl_id}/replay.json will only work for that archived item type. Similarly, orgs/{org_id}/all-crawls will return both crawls and uploads in the same paginated list, and you can tell which is why by the type field value.

edsu commented 3 months ago

Thanks @tw4l! I thought maybe rclone could be used programmatically to pull the set of signed URLs instead of a bucket, and then it could write to the many endpoints it already supports?

https://rclone.org/commands/rclone_copyurl/

But it wasn't a fully formed thought.

tw4l commented 3 months ago

Thanks @tw4l! I thought maybe rclone could be used programmatically to pull the set of signed URLs instead of a bucket, and then it could write to the many endpoints it already supports?

Ah, that's a good call! I misunderstood but makes a lot of sense to me :)

ikreymer commented 3 months ago

Thanks for sharing this @edsu. I think #578 will essentially provide built-in support for this, we hope to get to it soon. The idea would be to allow users' to switch primary and possibly secondary storage to a bucket of their choosing, which would use rclone internally to copy all of the data. There's still a few things to figure out with that, but should generally be doable. I think that will probably cover your specific use. Yeah, perhaps worth thinking how it could be made more flexible - we were thinking of this as a one-time switch, rather than periodic backups with filtering, eg. bring-your-own bucket instead of using ours.