thomasdondorf / puppeteer-cluster

Puppeteer Pool, run a cluster of instances in parallel
MIT License
3.24k stars 310 forks source link

Expose stats via prometheus #517

Closed abhisheksurve45 closed 1 year ago

abhisheksurve45 commented 1 year ago

Hi, is it possible to expose the stats via prometheus

image

The monitor option just shows on console

teammakdi commented 1 year ago
    const puppeteer = addExtra(vanillaPuppeteer)

    const cluster = await Cluster.launch()
    const clusterSize = cluster.jobQueue.size()
    clusterSizeMetric.set(clusterSize)

    const availableWorkers = cluster.workersAvail.length
    const busyWorkers = cluster.workersBusy.length
    const totalWorkers = cluster.workers.length
    crawlerWorkerQueueMetric.labels('available').set(availableWorkers)
    crawlerWorkerQueueMetric.labels('busy').set(busyWorkers)
    crawlerWorkerQueueMetric.labels('total').set(totalWorkers)

    const cpuUsage = cluster.systemMonitor.cpuUsage
    const memory = cluster.systemMonitor.memoryUsage
    crawlerClusterTelemetryMetric.labels('cpu_usage').set(cpuUsage)
    crawlerClusterTelemetryMetric.labels('memory_usage').set(memory)
jwgo commented 1 year ago

https://www.npmjs.com/package/@devgo/puppeteer-cluster?activeTab=readme#clustergetmonitorobject

I've released it with what's in the PR and a few features I need, so check it out and do with it what you will.