search

nteract / papermill

📚 Parameterize, execute, and analyze notebooks
http://papermill.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
5.98k stars 429 forks source link

Enterprise Gateway Support #602

Open j-raghavan opened 3 years ago

j-raghavan commented 3 years ago

Hello,

Wanting to find out if papermill supports Enterprise Gateway? I am looking for a solution where the kernels are installed in the Enterprise Gateway and Notebook resides on the Jupyter server. Can papermill be used in this configuration? Has anyone tried it? any pointers would be greatly appreciated.

Regards, Jayasimha

MSeal commented 3 years ago

Sorry I missed this question when it was first raised.

I haven't tried making Enterprise Gateway and papermill operate. I believe it was possible but might require a little bit of configuration magic. @kevin-bates might know more specifically on this.

kevin-bates commented 3 years ago

Thanks for the ping @MSeal and hello @j-raghavan.

It is possible to run a notebook via Papermill while targeting an EG server and this is exactly what the Elyra server extension does when its hosting server is configured with a gateway and a local pipeline is executed.

The key pieces of this are the GatewayKernelManager and GatewayKernelClient classes that reside in jupyter_server. These classes essentially proxy their methods to the gateway which proxies the methods to the actual KernelManager and KernelClient classes communicating with the kernel directly.

We also use a custom papermill engine primarily so we can prepare some kernel-specific arguments.

If your papermill invocation is not from a jupyter server extension, then the current packaging of the gateway kernel classes may be somewhat problematic in that it would require the jupyter_server package and configuration of the gateway information (which, from within a server extension you get for free from the underlying server). This issue may provide some help as papermill is essentially an nbclient application.