Chromium on a different machine in the network

[gmile]

First of all - amazing work!

It never occurred to me that using Chromium without firing up Puppeteer to render PDFs was possible. Knew about DevTools Protocol, but never connected the dots that one can actually implement that protocol to talk Chromium directly. Makes perfect sense in hindsight!

A question: would it be possible, if not currently then in principle, to talk to a Chromium that resides on an another computer on the network? E.g. a setup where Chromium would essentially represent a microservice? Or is talking via JsonRPC limited & can be done only to a local instance of Chromium?

[MaeIsBad]

It is possible to have chrome run on a separate machine on the network in principle, but it's not implemented currently.

The chrome devtools protocol can run over the network instead of using a pipe if you set the --remote-debugging-port=9222 and --remote-debugging-address options, but currently chromic_pdf only supports the pipe method

https://github.com/bitcrowd/chromic_pdf/blob/7eac5185ed794b7cd80df7703db01009b9988509/lib/chromic_pdf/pdf/chrome_runner.ex#L11

[maltoe]

Exactly ^^ AFAIK historically Chrome did not even support true remote debugging (i.e. 9222 was bound to 127.0.0.1, likely for security reasons), --remote-debugging-address was introduced later. Adding support to chromic_pdf would be feasible in theory, though I personally think building a small Elixir service yourself to encapsulate it is much preferable. What makes you think about this option, if I may ask?

[gmile]

What makes you think about this option, if I may ask?

With chromium-as-a-service available on the network, we'd essentially be trading performance of having chromium available as a sibling process in the same OS for:

• having a smaller, simpler Docker image,

• ability to scale chromium service independently of the our main app (it so happens that scaling our main app is not easy due to some architecture decisions).

Another advantage of this is, because there are examples of dockerising chromium (for instance, this), it should be relatively straightforward to deploy it with a debugger port exposed, without the need of having a think HTTP wrapper around it. As new versions of chromium come out, we could then update just this service without restarting our main application.

[maltoe]

This one looks more targeted at your use-case.

Not sure if I'm going to find the time to work on this, but FWIW I tried scaffolding a gen_tcp based implementation of ChromeRunner in 249-chromium-on-a-different-machine-in-the-network . Luckily there's an undocumented DI mechanism in place, so you can just do

ChromicPDF.start_link(chrome_runner: ChromicPDF.ChromeTCP)

to try it out. Could not get it working in the short time I had unfortunately, socket is always immediately closed with tcp_closed.

[edit] After switch to websockets, it's ChromicPDF.ChromeWS

[maltoe]

Took a closer look into the docs and it's actually a HTTP endpoint where instrumentation uses a websocket connection. Updated the branch with a working (but dirty) proof-of-concept (with nextools/chromium). But this will be more work than expected, also requires quite a bit of refactoring in Connection to make it pretty.

TBH I'm not 100% convinced that the setup you're aiming for is worth adding this complexity? Especially the exchange of the thin Elixir wrapper + HTTP API (that we can design and control) with a yet-to-implement network interface predetermined by Chromium... My guess would be that you'll be facing just a different set of pains on this path.

However, if you insist, feel free to adopt the PoC branch. Or possibly wait for me / someone else to pick this up, but not sure when that would be.

Hope this helps :wave:

[gmile]

@maltoe thanks for looking into this! I will definitely try find the time this week to check out the nextools/chromium branch :+1:

But this will be more work than expected, also requires quite a bit of refactoring in Connection to make it pretty.

My knowledge about chromic_pdf architecture is quite limited rn, could you elaborate what do you in mind for refactoring the Connection to make it pretty? Do you mean like refactoring it in a way that separates handling of messages coming from WS and from port?

[maltoe]

@gmile Branch is called 249-chromium-on-a-different-machine-in-the-network, nextools/chromium is the docker image I've used.

Refactoring: ChromeRunner should encapsulate the local chrome/chromium process spawned via a Port, but some parts of it have leaked into Connection. E.g., the entire port message handling is located there as well as the message "tokenization" (which isn't needed at all for ws connections as messages will arrives in frames). At first glance I'd say that ChromeRunner/ChromeWS should be the active processes, ChromeRunner take care of tokenization and port event handling; and Connection instead be passive and only do the JsonRPC transcoding and delegate to its connection "strategy" (-> probably rename ChromeRunner behaviour nested in Connection).

Other thoughts on implementation:

• HTTP client / Websocket client dependency should be kept optional
• Since the external chrome process remains when you shutdown chromic_pdf, we may need to have a way to recover existing targets to avoid accumulating unused ones.

[maltoe]

@gmile any updates on this?

[maltoe]

This working for us in one of our projects.