Backend crawl log streaming API: Stream logs while crawl is running

webrecorder / browsertrix

Browsertrix is the hosted, high-fidelity, browser-based crawling service from Webrecorder designed to make web archiving easier and more accessible for all!

https://webrecorder.net/browsertrix

GNU Affero General Public License v3.0

201 stars 35 forks source link

Backend crawl log streaming API: Stream logs while crawl is running #670

Open tw4l opened 1 year ago

tw4l commented 1 year ago

Sub-task of #796

Options:

Stream from k8s crawler pods stdout - see https://github.com/webrecorder/browsertrix-cloud/tree/issue-670-stream-logs-running-crawl for one implementation
Stream from files in logs/ directories in k8s crawler pods
Stream from fluentd?

tw4l commented 1 year ago

Connected to https://github.com/webrecorder/browsertrix-cloud/issues/738, which will be an intermediary step to streaming full logs from running crawls