Configure custom driver for browsertrix for visualization and deep zoom

[kmcelwee]

Use the --driver flag or customize run.sh to point the crawl to a custom JS file (inspired by the default: https://github.com/webrecorder/browsertrix-crawler/blob/main/defaultDriver.js). We need a custom driver that can interact with the visualization page and deep zoom.

Notes from Ilya

• Recommended debugging: a single-page crawl with a combination of screencasting and logging via console.log
• Driver example: https://github.com/webrecorder/dashboard-custom-drivers/tree/main/driver
• You can use any of the puppeteer apis, like page.hover() to automate the browser within a single page and cause additional resources to be crawled.
• To add additional pages to the crawl queue, you can use the "crawler.queueInScopeUrls" command and do something like:

module.exports = async ({data, page, crawler}) => {
  await crawler.loadPage(page, data);

  ...
  const moreUrls = ["https://example.com/a", "https://example.com/b", ...];
  crawler.queueInScopeUrls(data.seedId, moreUrls, data.depth);
};

---

Remaining Todos

• [x] Overwrite /app/defaultDriver.js with the custom driver or copy it to the server and point to it using the --driver flag
• ~Add the list of info.json links as seeds in the yaml (or create a seed file?) List of links: iiif-info-seed-file.txt~

[kmcelwee]

We have it working in puppeteer: https://gist.github.com/kmcelwee/cdbb6d2b4a5c2d9ac234d6de5db4716c

[kmcelwee]

When running custom driver locally, we can use docker volumes to override defaulDriver.js with our driver.

docker run -v $PWD/custom-driver.js:/app/defaultDriver.js -v $PWD/crawls:/crawls/ -it webrecorder/browsertrix-crawler crawl --url https://derridas-margins.princeton.edu/library/abraham-oeuvres-completes-1966/gallery/front-cover/ --limit 2 --generateWACZ --text --collection deep-zoom

[kmcelwee]

It will be easier to simply provide the extra info.json as a seedlist to the scrape instead of including it in the custom driver.

[kmcelwee]

Addressed by https://github.com/Princeton-CDH/cdh-ansible/pull/108

[rlskoeser]

@kmcelwee I think these are the relevant parts of ansible browsertrix role:

copy file — maybe just change the destination filename? https://github.com/Princeton-CDH/cdh-ansible/blob/main/roles/browsertrix/tasks/main.yml#L32-L37

crawl script — adjust command line argument https://github.com/Princeton-CDH/cdh-ansible/blob/main/roles/browsertrix/templates/crawl.sh.j2#L7