search

javascriptdata / dnotebook

Dnotebook is a Jupyter-like library for javaScript environment. It allows you to create and share pages that contain live code, text and visualizations.
https://dnotebook.jsdata.org/getting-started
MIT License
146 stars 10 forks source link

Epic Roadmap to new version of Dnotebook #10

Open risenW opened 2 years ago

risenW commented 2 years ago

We decided to rewrite Dnotebook to be more like Jupyter notebooks, where we can write and run code that interacts directly with the user's local OS in a safe and secure way. This overarching goal here is that we want users to have a native experience when using Dnotebook.

The following is a list of proposed features for the new release: Target release date: March 2022

risenW commented 2 years ago

Some musings on design architecture so far:

Update on the server side

The first week, I started by asking the question, how do I make Dnotebook work like Jupyter, meaning bash command works so I can run yarn add package in the browser, execute it server-side in Nodejs, and get real-time output logs in the browser.

I decided to go with the express HTTP stream. Nodejs has great support for streams, so I would basically start a server, send commands via HTTP request, open a stream response, capture logs from stdout and stderr, and continuously write back a response to the UI. This is basically done at the moment. see https://github.com/javascriptdata/dnotebook/blob/ref/typescript/src/server/src/routes/code_routes.ts

Now for the next task, I wanted to execute JS code. Executing JS code one time and returning the result is pretty straightforward, we can use an eval for that. But what I needed is a way to execute each cell in the same context. I.e All cells from the notebook will share the same variables-the typical notebook behavior So what do I do here?

Well, interestingly Nodejs has the VM API, which you can use to compile and execute code in a given context. So basically what I did was start a server, initialize a context, then send chunks of code to be executed in that context. Because the server keeps running between execution, the context is kept alive! This was implemented here: https://github.com/javascriptdata/dnotebook/blob/ref/typescript/src/server/src/runner/nodejs.ts

So with the core functionality done, I decided to add some more features.

First, I added an inline transpiler, meaning you can write any flavor of JS, and mid-way before it gets executed, it is transpiled using Babel to a compatible version that can run on the VM. See implementation here: https://github.com/javascriptdata/dnotebook/blob/f8a65fdca5961854e4c4ae864cf0ec2ac69e71e6/src/server/src/runner/nodejs.ts#L39

So what next? Well, a lot of things still has to be done on the server-side, some of the important issues I’m yet to figure out is:

  1. How to track the state of cells and know when they finish execution? This is quite tricky given the async nature of JS

  2. Managing multiple child processes-Kernels in Jupyter notebooks-so I can easily start and stop them from the UI

  3. Security of code process

Update on the UI

At the moment, it supports full editing of Markdown, JS, HTML, bash, and JSON. That means, syntax highlighting, code snippets, autocomplete, and execution all work at the moment. I used AceEditor for this. See implementation https://github.com/javascriptdata/dnotebook/blob/ref/typescript/src/frontend/components/CellEditor/index.tsx

So what’s next here? Well, the next major task we want to tackle in the UI is:

I will keep updating this comment as we add new features and encounter weird issues. This will be used to create a detailed architecture document before release.