Replace NodeJS backend in SeqTubeMap with Django

[subwaystation]

I need to get familiar with Django and a first nice exercise to do this would be to port the current SeqTubeMap Node.js backend to Django.

[josiahseaman]

Interesting. How complex is the SeqTubeMap backend? I was expecting we might treat SequenceTubeMap as an external service and talk through json for renders. But I hadn't gotten far into figuring out how tightly coupled the components were. I fully endorse familiarizing and integrating, but also know it's okay to have components stand alone as long as there's clearly defined interfaces and the translation speed is fast enough.

[subwaystation]

In his MoMI-G, @6br had his backend seperate from the frontend. We are currently discussing, what would be the best way to go.

[6br]

Because MoMI-G provides not only SequenceTubeMap but also other view modules such as Circos, Table, Linear Genome Browser, ...., and so on, I decided to separate between backend and frontend implementation. So, MoMI-G is one of an alternative backend implementation for SequenceTubeMap. I think that the modular approach is easy for development and debugging if we would like to provide a rich user-interface with features of webpack, react, typescript,..., among others.

[subwaystation]

I agree, that an isolated development of both parts of the implementation should be easier to do and easier to debug. Testing both together is then straightforward, too? Furthermore, we would have to define perfect interfaces between backend and frontend, because it is highly likely, that they will be implemented by different people, right? @6br @6br just mentioned to me, that our modular approach makes it difficult to deploy our tools. This might be a caveat for biomedical users. How about we provide

• a docker image including both backend and frontend working out of the box
• a bioconda package including both backend and frontend working out of the box

That should suffice most biomedical users?

[6br]

Yes, recent advancement of container technology mitigated the burden of deployment and development. If we can provide a container including both backend and frontend working out of the box, then testing both becomes straightforward. Also, I agree with the second part that we need to define an API between backend and frontend. At last, if we provide a docker image and/or a bioconda package, then that should suffice for most users; or another choice is that we provide desktop apps like Electron, but that might be too much.

[subwaystation]

Summarizing, both @6br and I favour a separated front-end and back-end. I see the following options:

back-end:

• STM (Node.js)
• MoMi-G (Rust)
• vgbrowser (Python)

As we already have started our back-end using Django, it is kind of obvious, we select the 3rd option. This would mean, I will have to extend our current implementation so that it is able to serve an appropriate JSON, which is ready for visualization. The details of this implementation might vary dependent on the front-end we chose:

front-end:

• STM (React)
• MoMI-G (React, but lot's of features, we won't need for now)
• vgbrowser (does not exist yet)

As both STM and MoMI-G have different tubemap.js, I am not sure, which way to go here. The MoMI-G solution might be too overblown to get something running until the hackathon. The development speed with STM's front-end could be faster. @6br What do you think?

[6br]

In backend, the backend of STM is not so complicated. So it can be ported to Python (or Rust) if we start from the GFA-json. I agree with the python implementation considering that we need to do a lot of trials and errors and Django supports WebSockets. Also, we should work with the server on top of xg https://github.com/graph-genome/vgbrowser/issues/5#issuecomment-521614662. I am happy if I can discuss and work on this part in the hackathon together.

In frontend, there is a list of the difference between STM and MoMI-G tubemap.

• A selected chromosome path is by default drawn as a straight horizontal line regardless of the path order in the input.
• Genes, annotations, chromosomes, and variants can have different colors for easily distinguishing them.
• We implemented another scale mode suitable for visualizing large SVs.
• We added bigWig/bigBed support.
• We inserted a shaded gap between distant genomic intervals.

For example, I can try to port the latest STM into MoMI-G STM in the hackathon. But some features are tightly coupled with the MoMI-G backend; they cannot work without MoMI-G backend.

[subwaystation]

The fastest way to get a demo for the hackathon is to take the front-end code of STM and extend our back-end accordingly, effectively presenting the current IVG user interface. In addition, the front-end code will be extended so that it can handle our block structure and people can switch between zoom levels.

All other additional features we want to have are optional or can be discussed and implemented at the hackathon. Would that be alright with everyone?

If so, I would volunteer for the job ;)

[josiahseaman]

For the bulk of it, I think you need to implement a view in Django that returns a gfaJSON of the whole graph. The sequence of events goes something like this:

• Django Database has a graph called "test"
• User loads a page with STM and URL /vgbrowser/test/
• STM requests the JSON for 'test'
• Graph.views.get_graph fetches a GraphGenome(name='test') and "renders" it to JSON
• STM read JSON and produces graphic

Simple Django JSON responses The nice thing is they're basically just dictionaries. Let me know if you have any questions.

[subwaystation]

I think I get the whole workflow.

What I am missing is where vg steps in? How do we calculate the haplotype blocks from a given .vg file? We could even use vg view in order to directly produce STM ready JSON files from a given GFA.

Or is your approach aiming at testing our current setup only?

[subwaystation]

Current issue status:

• [X] NodeJS + React Tutorials.
• [X] Split SequenceTubeMap into server and client. --> https://github.com/subwaystation/sequenceTubeMap/tree/split
• [ ] Django + React Tutorials.
• [ ] Replace NodeJS backend with Django backend.
• [ ] Bring @josiahseaman's backend and mine together.