search

thomasdondorf / puppeteer-cluster

Puppeteer Pool, run a cluster of instances in parallel
MIT License
3.2k stars 307 forks source link

enhance batch output for cluster runs #266

Open blaskovicz opened 4 years ago

blaskovicz commented 4 years ago

For non-interactive (eg: batch) environments where puppeteer-cluster is used, it would be nice to enhance the display of statistics to be more friendly:

Example output:

[K== Start:     2020-03-26 20:01:45.859
== Now:       2020-03-26 20:01:50.859 (running for 5.0 seconds)
== Progress:  0 / 306 (0.00%), errors: 0 (0.00%)
== Remaining: unknown (@ 0 pages/second)
== Sys. load: 7.3% CPU / 94.3% memory
== Workers:   4
   #0 WORK UNKNOWN TARGET
   #1 WORK UNKNOWN TARGET
   #2 WORK UNKNOWN TARGET
   #3 WORK UNKNOWN TARGET

Ideally, there would be 3 enhancements made (listed in order of request priority):

1) ability to aggregate display output (for example, when tasks change state or on some interval) 1) ability to set custom work target descriptions (eg: for when the data passed to a task is a non-string (code here); 1) ability to detect non-interactive terminal and avoid outputting terminal escape sequences

thomasdondorf commented 4 years ago

I guess it's better to just implement a API to read the statistics (see #8)? Then people can develop their own code to log the status to console.

I'll leave it open for discussion.

BrightSoul commented 3 years ago

About item 2 (work target descriptions): the worker can update the url property of the complex object you passed as data, and it will display as the worker description. It doesn't need to be an actual url at all. It just needs to be a string, as you can see here in the source. https://github.com/thomasdondorf/puppeteer-cluster/blob/fbd9a9f8f9c2c889e145b6af8c2bd3fa28ec4bf6/src/Job.ts#L31-L42

It's kind of hacky but... well, it works for me. Displaying descriptions is essential in my case because each of my workers performs a long-running, multiple step task. And here's the output.

monitor

You can even use ANSI color escape codes. 

data.url = '\033[33m Doing step 1 \033[39m';