create playbook to install dependencies and create a web archive for a site

[rlskoeser]

• setup deploy user
• install docker
• pull docker image for browsertrix: docker pull webrecorder/browsertrix-crawler
• create browsertrix yaml crawl config file
• (stop playbook here) - create a bash script that launches crawl
  • start the crawl
  • push resulting collection files to a github repo

questions:

• On my machine, crawling Derrida's Margins took ~4 hours; we can't leave the playbook running that long. How can we kick off the crawl and save the results when it's done?
• Looks like docker on Linux is normally run as root. Should we:
  • run docker commands with sudo, for simplicity?
  • Follow post-install steps to run docker as deploy user? https://docs.docker.com/engine/install/linux-postinstall/
  • Follow instructions to run docker as non-root user? https://docs.docker.com/engine/security/rootless/

revisions needed

• 293

• 294

[kayiwa]

run ansible task inside a tmux session

https://stackoverflow.com/questions/68925949/running-server-in-tmux-through-ansible

[rlskoeser]

ideally the script that runs the crawl and sends to github should also generate the index file and commit it, but that would require setting up python and installing pywb

~maybe we don't need this, since we're not planning to make the deployed version searchable and can use autoindexing for dev~

update: we do need this, because wayback can't find the pages to serve without an index.

[rlskoeser]

going to close this and open issues for remaining tasks, because the basic deploy & crawl script are working