Integrating Django with Frontend JavaScript Frameworks

[jeancochrane]

Integrating Django with Frontend JavaScript Frameworks

Background

Our R&D work on Gatsby (#7) has given us a taste of the power of contemporary JavaScript frameworks like React. Through their native support for ES6, their stateful component-oriented APIs, and their modern developer tooling, contemporary frontend frameworks make JavaScript development more fun while simultaneously expanding the horizon of possibility for complex user interfaces.

As we noted in our recommendation of adoption for Gatsby, however, these frameworks have a steep learning curve, and if we want to use them more extensively we need to adopt them incrementally. Most immediately, we need a way to integrate frontend frameworks like React into our standard Django stack in a way that allows us to continue to leverage as much of our Django expertise as possible while we get acquainted with the new paradigms offered by contemporary JavaScript frameworks.

Proposal

I propose to research approaches to integrating contemporary JavaScript frameworks with our standard Django app architecture. My goal will be to produce a clear path forward whereby we can use a frontend framework like React for views that require particularly complex interactivity, while falling back to standard Django views and templates for simpler views like List and Detail pages.

In sum, my focus will be on developing what many developers call a "hybrid" approach: one where we can isolate our use of the frontend framework only to specific views, instead of following the more common pattern of using Django exclusively as a data layer API while delegating all user-facing logic (like templating and routing) to the frontend framework.

Deliverables

This R&D project will proceed in two phases: research and development.

In the research phase, I plan to read articles and solicit advise from other developers about hybrid approaches to integrating Django and frontend frameworks. While my main focus will be on React, I expect I may open up my research to Vue.js as well, since it follows a similar conceptual paradigm as React and is advertised as being optimized for hybrid apps and incremental adoption.

In the development phase, I plan to produce a sample project that implements the most promising hybrid approach as identified in the research phase. Once this sample project has been approved by the R&D team, I plan to adapt it into a template that we can use for a future client app.

Timeline

I expect this R&D project to take somewhere between one to three months (two to six R&D days). The main reason for my uncertainty is that I don't yet have a good sense of how much prior work has gone into hybrid approaches like this one: if clear best practices already exist, this R&D project may be as simple as adapting an existing project based on a blog post; but if (as I suspect) there hasn't been much reusable work on this kind of approach, it will take longer to forge a new path.

[beamalsky]

I saw a version of this implemented in a NICAR 2020 session I was pretty taken with. Here's my summary from https://github.com/datamade/ops/issues/642:

How to build a live data driven application that never crashes. Loved this one. Tyler Fisher demoed a stack he uses at News Catalyst: Django backend with REST framework and signals, data as serialized JSON, and a React frontend. He wasn't sure what the limits of this approach are data-wise, since you wouldn't want to do it with enormous JSON files, but for moderate amounts it leads to a fast and stable application.

Here's the demo repo he used in the session: https://github.com/tylerfisher/nicar20

Not our exact needs necessarily, but worth looking into when this gets picked up!

[hancush]

This article offers several examples of patterns for achieving this. It's a few years old, but maybe a good starting point.

[hancush]

Also intrigued by (but don't love the API for) this Django plugin.

[jeancochrane]

I made some good progress last week setting up a sample app based on the blog post Hannah linked above. I'm keeping track of my progress in this repo: https://github.com/datamade/django-react-templates

Overall the approach works great for using client-side React components in a Django template. The downside is that it only supports client-side apps, which means markup is not pre-rendered, so search indexes won't see the markup and there will be a lag on the client side when they navigate to a page while JS paints the components on the screen.

I think this approach could work well for apps that have one or two interactive components. As an example, https://edi.erikson.edu/ mostly works fine as a simple MVC app, but the map is highly interactive and could greatly benefit from being a React component. I could imagine it being an acceptable tradeoff to have the map load slower than the rest of the page and not be indexed by search engines, but to be able to leverage the expressive power of React to design it.

However, I don't think this approach will achieve our goal of making React useable on every DataMade project. For that, I think we need a solution that makes use of server-side rendering to ensure that all responses have pre-rendered markup.

So far there aren't any great solutions for server-side React rendering with Django, as confirmed by the author of the post above. The best supported solution, python-react, requires a sidecar Node server to do the rendering. However, I think we could do a much better job if we wrote a custom Django template backend for React. This would require more custom code up front, but potentially could be a high-value open source library.

In order to have a fully-functioning React template backend for Django, we'll need to at least solve the following problems:

• Run the server-side render function in a node subprocess in a way that is fast and resilient
• Figure out a forms API
• Ensure automatic HTML escaping
• Figure out integration with Django template tags like url and static
• Wagtail compatibility
• Translations (could be covered by a templatetag integration)

I'm going to spend some R&D time exploring this idea further and trying to get a handle on just how complex it would be.

[hancush]

@jeancochrane TIL Google indexes both raw and dynamically rendered HTML documents, provided content is rendered in a reasonable amount of time (generally thought to be five seconds or less). So, it doesn't seem like client-side components are automatically excluded from search indexing (though, of course, that places higher importance on optimizing load speed and thoughtfully loading important content first).

Thinking more about this, and about how painful even Jinja and Django has been on Payroll, I'm also interested on hearing more on why implementing and maintaining a custom template engine is preferable to running a Node server alongside an app. This seems pretty similar to running a database or Redis instance alongside an app, i.e., relatively trivial compared to a custom code solution.

[jeancochrane]

So, it doesn't seem like client-side components are automatically excluded from search indexing

Great to know that we'd be covered on search indexes 👍I do think the loading lag remains an equally serious problem but we at least have more control over how we manage it.

I'm also interested on hearing more on why implementing and maintaining a custom template engine is preferable to running a Node server alongside an app.

There are a few reasons I'm skeptical of python-react, the library that implements the separate server approach:

• Two network calls (request and response) feels like an unnecessary amount of overhead for a render function
• On Heroku, it necessarily doubles our app hosting costs from $25/hr to $50/hr
• It fully separates React rendering from Django rendering, meaning e.g. templatetag integrations are not possible
• The library hasn't been updated in two years

All this being said I think there's a good chance that the subprocess approach is not feasible (python-react says it used to run on a subprocess and switched to a separate server because they found the approach too brittle). Either way I'm expecting that for due diligence purposes I'll have to give python-react a try and include it in my recommendation of adoption for whatever approach winds up being best.

[jeancochrane]

I made some good progress here today. Currently working on server-side rendering in a separate branch.

So far the main downside of the subprocess approach that I can see is that it's slow to render compared to the Django template backend (about three seconds to render a simple component). Turns out this is actually the whole reason that python-react switched to a separate server process. I'm actually not convinced it's that bad since we can put a cache in front of it, and the performance is basically the same as django-compressor since they're doing the same thing under the hood (running a node subprocess to compile the code with browserify/babel). But I want to spend more time looking into whether we can speed this up at all, since if we can it will also bring benefits to our work with django-compressor.

Right now I'm thinking through what it would mean to integrate with Django template tags. The approach I'm pursuing right now is a two-step render, first passing the template through the default Django template renderer to render out template tags, and then passing it into the Node process to compile the React code. This works great for single-file components, but I ran into a wall when I realized that it won't apply the Django renderer to any imported components since the Django template renderer doesn't understand JavaScript imports. One way we could work through this would be to extend the parser to maybe use the JavaScript lexer and traverse import/require trees the same way it does with extends. This would sort of be like implementing our own bundler and would probably be hard.

Another approach might be to bundle the templates before we parse them with templatetags; this would probably be much easier to implement but I expect would slow things down even further, since bundling is expensive. We could also implement custom functions for Django template tags and avoid the integration altogether, although the amount of duplicative code involved in that effort seems prohibitive.

So, some fun progress, but we're nowhere near using this in production yet. I'm going to think a little bit more about these problems and try to make a game plan.

[jeancochrane]

Here's a quick summary of where I'm at so far. At this point I've tried out three approaches:

• Hybrid
• React as a template backend
• Gatsby frontend and Django API backend

Hybrid approach

Gist: Pass Django context into a React app at runtime, and use a library like Django Compressor to bundle and distribute the React app as a static JS file.

Advantages:

• Least amount of change in our current practice for the most amount of React
• Only requires tools we already use, including Django Compressor
• Makes React an optional add-on, otherwise fully supports using Django templates

Disadvantages:

• Bundling times are long, which causes a problem for page load absent a cache
• Lack of easy templatetag integration in React code (all logic must be pushed up into the view)
• No server-side rendering, so React apps load slowly and require a loading animation
• Won't necessarily accomplish the goal of spreading React knowledge if React is clearly a second-class citizen (maybe the full support of Django template language is a downside)

React-as-template backend

Gist: Swap out the Django templating language and write a Django template backend to use React instead.

Advantages:

• The most sensible level of abstraction (React as a templating layer for Django)
• Fits in the most with canonical Django patterns
• Full integration with Django template tags is possible
• Supports server-side rendering

Disadvantages:

• Render time is quite slow compared to Python
• Speeding up the render to a reasonable speed requires a sidecar Node server
• Requires maintaining a custom library that monkey patches the Django templating system

Gatsby frontend and Django API backend

Gist: Keep the frontend and backend as two separate apps; use Gatsby deployed on Netlify for the frontend app, and communicate with a Django API hosted on Heroku (this is what we did for LISC CNDA, although we deployed on EC2 because Heroku was not yet approved)

Advantages:

• Maintain consistency between our static and dynamic app stacks
• Gatsby is opinionated about React setup and distribution, and makes good optimizations
  • E.g. no webpack config; minification, source maps, code splitting out of the box
• Full separation of concerns between frontend and backend logic
• Gatsby builds in server-side rendering by default, since it's a static site generator

Disadvantages:

• Biggest break from our current patterns of development
• Requires much more React knowledge up front
• Frontend and backend have to be deployed on separate services (Netlify vs Heroku)
• Requires more active R&D to make sure we have good React options for common tasks
  • E.g. forms and CMS

[hancush]

@jeancochrane Re: load speed, I'm not 100% sure this if this fits any of your use cases, but I learned this week that I can leverage the Django cache to cache compiled inline JavaScript.

Like, if I have a Django template with some compress blocks containing inline JavaScript, which are recompiled every time you load the page –

{% compress js %}
<script type="text/javascript" type="module">
    // do some javascript
</script>
{% endcompress %}

– you can cache the entire page, including the compiled JavaScript, in the Django cache. We may be able to use template fragment caching or django-adv-cache-tag to cache compiled JavaScript more precisely.

I think django-compressor will cache compiled modules for you.

[jeancochrane]

Yeah, we definitely want to stick a cache in front of Django Compressor no matter how we use it. My concern is that the request that populates the cache will still be quite expensive, and there are inevitably some views in certain types of projects that can't be meaningfully cached at all using Django's caching system. Development less important, but I've also noticed it slows down development a lot when you need to wait 2-3 seconds on every page reload.

[hancush]

Hm, good point. Would offline compression be of any help?

[jeancochrane]

Yeah! I would expect that would do a lot to improve the performance of the cache-populating request in cases where we can cache.

[jeancochrane]

We had a great tea time convo about this today. It sounds to me like one clear next step is to formalize the hybrid approach and adopt it for cases where we can accept some amount of load time for the components and where React isn't the core of the application. Simultaneously we should continue R&D into areas where we have no clear React analog for things we can do in Django: complex forms and CMSes seem like the top of the list right now.

[hancush]

A couple of articles on Webpack as an alternative to browserify / babel / django-compressor-toolkit:

• https://eldarion.com/blog/2018/10/09/setup-guide-django-vue-webpack/
• https://www.valentinog.com/blog/webpack-django/