Jupyterlite and Pyodide integration

[miohtama]

This is the epic issue for integrating the Python-based backtest execution with our website. It is based on the proof of concept built by @joemarshall.

• Uses Python libraries thru Pyodide
• Trading Strategy Python library [has been modified to run using Pyodide interpreter]() (uses features like async and JavaScript side download request for datasets)
• All same features you can do in standalone Python should be available on the website

UI

• Integrates with existing trading pair browser
• We use Svelte component integration
  • BacktestWizard.svelte - generate the skeleton of Python notebook to get started based on user choices (trading pair, candle time frame, strategy type)
  • Backtest.svelte - incorporate backtesting functionality (buttons, UI framing)
  • Notebook.svelte - low-level Jupyter Notebook integration component
• Comes with a wizard to generate a skeleton backtest Python notebook (Choose pair, strategy type and some parameters)
• Notebook is integrated into the website through a component (svelte-python-notebook) - to be researched what this means in practice
• A custom notebook limited UI to make it easy for the user to focus on the task on the hand - has only buttons: Play, Resume, Add cell, etc. No file browser or menus.
• A button to download the backtest script and run it locally on the user computers if more computational power is needed

Spec

TODO

• What inputs the main website needs to pass to BacktestWizard
• What inputs the main website needs to pass to Backtest (the generated Python skeleton script) -How Notebook.svelte will look like

Milestones

• [ ] Specification of interfaces
• [ ] Create UI wireframes based on existing Jupyter notebook implementations
• [ ] Create a barebone Svelte-wrapped Notebook component
• [ ] Add play, resume, etc. editor controls
• [ ] Create a Backtest that runs with a single Python notebook
• [ ] Create a BacktestWizard that allows generating Python notebooks based on user choices
• [ ] Integrate Backtest and BacktestWizard to the main website so that they are accessible from the trading pair view

Out of scope

For the future

• Integrate the current trade executor interface for position/trade analysis
• Saving backtest code and results locally in the web browser
• Share backtest code and results easily

[joemarshall]

One thought on the notebook interface - There's 2 options here I think 1) The interface is a stripped down full notebook interface, in that you can add text / code cells etc. 2) The interface gives you just a set of text cells with instructions and a fixed set of code cells in between, but with no ability to edit the text. This is even more stripped down, and would allow the text to be styled in with the main site content really nicely, but obviously does limit how much users can mess with it.

From a tech side it doesn't make that much difference - there needs to be a per-page python kernel whatever. Number 2 would just reduce the amount of UI on the page.

[kenkunz]

Frontend integration Proof-of-Concept

Based on Joe's guidance, I was able to get a basic PoC working in frontend

• copied the build artifacts for notebook and pyodide into /static
• wrapped with a lightweight Notebook.svelte component similar to Joe's example
• loaded Notebook.svelte component into a test page and successfully loaded the test.ipynb notebook

I ran into some styling issues where some of the jupyterlab CSS applies global styles to the page. I spent some time trying to scope the CSS to the <jupyter-notebook> component using shadow DOM. I experimented with this for a while and got it sort-of working, but it caused other issues. For now, it looks like there are some manageable work-arounds we can use to fix the global page CSS issues.

Joe's feedback on global CSS issue:

I also tried shadow dom briefly, I think jupyterlab makes some assumptions it shouldn't about the Dom. I think because jupyter haven't really focused on this usage, it is slightly brittle. I think a good longer term plan would be try to get jupyter folks to support jupyterlite as an embeddable webcomponent. I think this is a nice proof of concept that you can do it mostly by plugging together existing node packages and without having to build your own jupyterlite/lab tree. I suspect that may be a non starter until jupyterlite becomes an official project though.

test.ipynb running in frontend

[kenkunz]

MVP / v1 requirements

Now that we've proven we can load a a JupyterLite notebook in frontend, the next step is to integrate this into our trading-pair workflow.

Based on discussion with Mikko, here is a basic v1 user flow:

• user navigates to a trading-pair page
• in place of the current "Historical Data" button, user now sees "Create Automated Trading Strategy" button
• clicking this button takes user to a new page:
  • page includes the Single trading pair EMA crossover strategy notebook example from our documentation
  • the exchange_slug and trading_pair values in the notebook reflect the exchange and pair that the user was viewing
  • user should be able to press ▶ play and run this notebook
  • user should be able to edit the notebook (changes are auto-saved)

Questions

1. If user navigates to a different pair and clicks the same button, does this load the same notebook or a new one?

A: it generates a new notebook based on the original template, with the new exchange and trading pair values

2. If the user navigates back to a pair they have previously opened / saved a notebook for, what happens?

A: preferably, it opens their modified / saved notebook for that pair; this is "nice-to-have" and can be addressed in v2.

Eventually, we'll need to add a nav option to the site where users can get back to "your backtests / your strategies"

For now, the most important thing is that you can successfully run a notebook that used the trading-strategy-client to conduct technical analysis based on data from our APIs.