search

admc / wd

A node.js client for webdriver/selenium 2.
Other
1.53k stars 402 forks source link

Support streaming of jsonWire call responses #450

Closed lennym closed 5 years ago

lennym commented 7 years ago

Some calls to jsonWire endpoints - in particular log - can potentially return large volumes of data, which is not ideally suited to being aggregated in memory and returned in a callback, and is much better consumed by the client as a stream. So add an option to _jsonWireCall to support building custom methods which can consume the response as a stream.

The context for this is a tool I am working on (https://npmjs/timeliner) which exports the performance logs from chrome devtools when running a page in chromedriver to export various performance metrics. However the performance tracing logs can be quite large and so for long running pages frequently cause the process to crash due to memory limits when consuming the logs.

By streaming the logs I can filter out non-pertinent entries without aggregating the entire log into memory and so no longer hit memory limits. See also (https://github.com/lennym/timeliner/commit/0f1f2524499d7f523246ed00003580c9ed7f62db)

lennym commented 7 years ago

I'm seeing a lot of Error: no user specified. connecting to saucelabs in the travis build. It doesn't seem related to this PR.

sebv commented 7 years ago

I see what you want to do, but this seem too hacky, would be better to define proper interface protected by tests to do what you want to do.

lennym commented 7 years ago

No problem. I don't necessarily disagree at all about the hackiness.

If I get some time for another run at it I'll see what I can do, but it may not happen as I have a "working" (read: "monkey patched") solution already.

admc commented 6 years ago

Did we wind up with a good solution for this?