Use variables in JS / CSS and caching thereof

[JuroOravec]

Background

I feel I've addressed most of the items that were blocking me for the the UI component library (there's about 5-6 MRs I've got half-ready in the pipeline), and kinda the bigger pieces that are still remaining for v1 is the rendering of CSS / JS dependencies and then documentation. So I've started looking into the CSS / JS rendering.

IMO before I get back to some of the ideas we discussed before, regarding the dependency middleware and such (https://github.com/EmilStenstrom/django-components/issues/478), it first make sense to add desired CSS- / JS-related features to django_components , and only then think about how to send the CSS / JS over to the client.

So one thing I'd like to have is to use the variables from get_context_data() in CSS / JS files. Let's break it down.

How static files are loaded

When it comes to static files, so those that are served by the server AS IS, those are managed by Django's staticfiles app. My experience was that the files are "collected" / generated using collectstatic, which physically puts them in a certain directory. And then, when the server starts, it serves the files from that directory.

Now, if we want to treat CSS / JS as Django templates, we cannot pre-process them with collectstatic. Instead, what would be generated would depend on the components inputs. Same as it is with the HTML template.

How CSS / JS is inserted into the HTML

Both CSS and JS can be "injected" into an HTML file it two ways:

• Inlined, meaning that I embed the full definition within the HTML, e.g.

  <div>
      <!-- CSS -->
      <style>
          .my-class {
              background: red;
          }
      </style>
  
      <!-- JS -->
      <script>
          console.log("hello world!")
      </script>
  </div>

• Linked , meaning that I only declare the URL where the CSS / JS can be found, and the browser fetches it

  <div>
      <!-- CSS -->
      <link media="all" rel="stylesheet" href="https://example.com/assets/light-efd2f2257c96.css">
  
      <!-- JS -->
      <script defer="defer" type="application/javascript" src="https://example.com/assets/wp-runtime-cc0837173e45.js"></script>
  </div>

Static CSS / JS is linked. This is done by the ComponentDependencyMiddleware, which works like this:

1. When we render a component, we prepend the formatter HTML string with <!-- _RENDERED {name} -->
2. When the HTML response is being sent, the middleware looks at the response HTML, looks up all <!-- _RENDERED {name} --> comments within the HTML, and retrieves corresponding component classes from the registry.
   • TODO - How would this be handled if there was multiple registries?
3. For each component class, it gets the JS and CSS associated with the class
   • If the JS / CSS is inlined, it renders it inlined, e.g. <style> ... </style>
   • If the JS / CSS is file name, it is treated as a path to a static file
     • NOTE: The file name is first attempted to be resolved relatively to the python file that defines the component, then relatively to STATICFILES_DIRS.
     • In this case the JS / CSS is rendered as linked, e.g. `

Considerations

• We should be able to serve both static CSS / JS and dynamic (generated at component render time)
• All CSS / JS files and snippets should have a unique URL, so we can support the use case for HTMX fragments, where we would send info on used CSS / JS files in the HTTP header along with the rendered fragment in the body.
• It'd be great to cache the dynamically-generated CSS / JS files.

Suggestion

One approach that could fulfill the above is using Django's cache plus defining an endpoint / view that returns the cached file.

It would work like this:

1. When I define a component, I could specify an attribute like ~dyn_js~ where I could write an inlined JS script, same as we currently do with the js attribute.

   • NOTE: Actually reuse the js attribute, and only define js_vars

2. ~At render time, at the same time as we're rendering the HTML template, we'd also treat the JS file as a Template, and render it.~

   • NOTE: Instead, we'd pre-process the JS file by wrapping it in function

3. We'd put the result string into Django's cache, using the component inputs to generate a hash key.

4. Then there would also be an endpoint defined by django_components, that the user would have to register at installation.

   The endpoint could be something like:

   /components/cache/table.a7df5a67e.css

   • /components/ would scope all django_component's endpoints
   • /cache/<compname>.<hash>.<suffix> would be a path to fetch the cached item.

5. And the same way as we currently insert <!-- _RENDERED {name} --> into the rendered template, we would instead insert <!-- _RENDERED {name}.{js_hash}.js --> or <!-- _RENDERED {name}.{css_hash}.css -->.

6. Thus, once this HTML would get to ComponentDependencyMiddleware, it would parse these comments, and:

   • For static files, it would behave the same as now, and point at the static file's URL
   • For the dynamic, hashed, files, it would point to /components/cache/{name}.{js_hash}.js (or .css)

Declaring used variables

But to be able to effective cache the JS / CSS files, one way could be to declare which variables are intended to be used in JS / CSS, e.g. like so:

class MyComp(Component):
    def get_context_data(self, name, address, city):
        return {"name": name, "address": address}

    js_vars = ["name", "address"]
    js = """
        ...
    """

So in the example above, only name and address context variables would be available for use in the JS script. And it also means that if either of the two would change, then this would result in a miss in the cache, and a new JS script would have to be rendered.

Cool thing about using something like js_vars is that we could say that the default is [], and then we wouldn't need js_dyn attribute, and instead we could just use the js attribute we already use.

So current behavior, which is static JS script, would be the same as

class MyComp(Component):
    ...
    js_vars = []
    js = """
        ...
    """

Passing declared variables to JS

Don't know if others share this sentiment with me, but IMO formatting JS script with Python / Django is weird and hard to read.

So maybe a better way would be to pass those declared variables to JS.

So if my original JS script was this:

console.log("hello world")

Then what we'd actually send to the browser is:

(() => {
    const $component = JSON.parse('{ "name": "John", "address": "BTS" }');

    console.log("hello world")
})()

So:

1. Wrap the JS in self-invoking function, so variables are not defined globally.
2. Define $component, which is an object of the js_vars keys and their values piped through JSON de/serialization

So in the declared JS file, one could use e.g. $component.name:

console.log(`hello ${ $component.name }`)

Reusing JS script with different inputs

This maybe raises a question - use JS vars or Django vars? At first I was inclined to allow both - thinking that maybe users may have some complex filters or tags that could be difficult to rewrite in JS, and likewise some data manipulation could be easier in JS than in Django template.

However, I am also considering the option of using only JS vars. So people could still use Django filters and tags in the JS template, but would not be able to access any variables. The reason is that this would work nicely with the client-side JS / CSS manager, as discussed in https://github.com/EmilStenstrom/django-components/issues/478. Because then we'd need to send any JS script to the browser only once. And if there were multiple component instances using the same script, they would just invoke that given script with different arguments.

How will the JS script know which HTML template it is associated with?

I'd say that Django / Django components are loosely coupled with HTML. Meaning that while do have e.g. the html_attrs tag or separate HTML, JS, and CSS, django components could still be theoretically used to render anything. Markdown, LaTeX, other code, etc...

This isn't bad on it's own, but it means that rendering HTML with it has its limits.

For example, because the rendered HTML is passed around as string, some components could have an HTML template that defines multiple root elements. Or none. Or would not render closing tags, because they would expect the closing tag to be supplied outside the component. Crazy, but it could happen.

The point is, we can't reliably tell if there's any valid HTML in the template. And so we cannot mark the generated string in any way that would make it possible for the JS / CSS to know what which HTML it is scoped to.

It would be possible to parse the generated string, but I'm concerned that that could give unnecessary overhead to each request.

It would be maybe possible for some system where the user would opt-in, signalling that the component's template IS a valid HTML. ...But then we'd still need to parse the HTML string to find the root element

Since Django's templating is completely independent from HTML, there would always have to be 2 parsing rounds (django -> HTML and HTML -> DOM) if we wanted scoped CSS / JS.

But I guess if we ever did go in this direction, it would still be compatible with the approach of wrapping the JS script in a function, and calling it with different inputs. E.g. then we could allow users to access the root component(s) under $els

What about CSS?

This is already quite long post, so won't go into CSS. But much of the same applies as does for JS.

---

⚠️ UPDATE ⚠️: I'm now actually NOT in favour of treating the JS as a template, so the only way to pass data from Python component to JS would be via the exposed variables. Because otherwise, it would make sense to allow django syntax also in standalone JS files. But including Django syntax in JS or CSS would mess up the linters.

[EmilStenstrom]

I think this discussion would be easier if there was a concrete problem that you are trying to solve, that you made clear. It's always so hard to design for "problems that people might have", because if you get the problem wrong, you will get the solution wrong too.

Some things to consider:

• Django's cache is not enabled by default, so this would add stuff to the default setup
• Wrapping the script in an anonymous function might break it, because variables might get hidden that were depended on
• Passing context from python to JS means the objects needs to be JSON serializable, which some python objects (namely dates) are not.
• What if you want other variables/calculations in the js files compared to the template? I think that's likely.

This above that I agree with:

• JS and CSS should not be rendered by Django. I don't think that's something you need often.
• I think passing variables to JS makes sense
• I think passing variables to CSS makes sense too

Given the above, I'm suggesting the API be something like:

class MyComp(Component):
    def get_context_data(self, name, address, city):
        return {"name": name, "address": address}

    def get_context_data_js(self, name, address, city):
        return {"name": name, "address": address}

    def get_context_data_css(self, *args):
        return self.get_context_data(*args)

    js = ""
    css = ""

[JuroOravec]

I think this discussion would be easier if there was a concrete problem that you are trying to solve, that you made clear. It's always so hard to design for "problems that people might have", because if you get the problem wrong, you will get the solution wrong too.

Yeah, agree. I feel I will have to do a full end-to-end proof of concept (that includes also the client-side dependency manager) to understand thing a bit better and expand on this. Because there are some constraints, e.g. so far it looks that, to support this feature, we would users to ALWAYS use the middleware (or call an equivalent function on the rendered content).

---

I think passing variables to CSS makes sense too

This is one thing that I was thinking about, but don't think there's a non-breaking solution, because to make it work reliably, the CSS variable would have to be scoped under the top-level element of the HTML template. And for that we'd need to parse the HTML with beautifulsoup or similar to insert a class or other identifier to that top-level element.

Maybe this will all work out fine, but I am yet to run benchmarks to see how much overhead it would introduce.

We would also need to place constraints on how people write HTML templates - each template would need to be a valid HTML.

IMO this is probably a step in a good direction. But overall it will be a big update.

Given the above, I'm suggesting the API be something like:

class MyComp(Component):
    def get_context_data(self, name, address, city):
        return {"name": name, "address": address}

    def get_context_data_js(self, name, address, city):
        return {"name": name, "address": address}

    def get_context_data_css(self, *args):
        return self.get_context_data(*args)

    js = ""
    css = ""

I like on my proposed approach that 1. the function arguments have to be defined only once, and 2. that we don't have to re-compute get_context_data if there's shared data.

On the other hand, I think having dedicated functions like these will make it clear to people that js and css data need to be JSON-serializable.

Also, something which I haven't shared yet, but is relevant here, is that I also wrote a bit of validation logic for typed components. So when someone specifies the generics like:

Args = Tuple[int, str]

class Kwargs(TypedDict):
    key: NotRequired(str)

class MyComp(Component[Args, Kwargs, ...]):
    ...

Then the args, kwargs, slots, and data returned from get_context_data is type-checked against the declared types.

Now, the relevant point is that the validation will make it possible for people to use the types for declaring function signature of get_context_data. So one could then leverage this to avoid having to duplicately define types and function signatures for all 3 get_context_datas, people could instead use self.input, so like this:

class MyComp(Component):
    def get_context_data(self, *args, **kwargs):
        return {"name": self.input.kwargs["name"], "address": self.input.kwargs["address"]}

    def get_context_data_js(self, *args, **kwargs):
        return {"name": self.input.kwargs["name"], "address": self.input.kwargs["address"]}

    def get_context_data_css(self, *args, **kwargs):
        return self.get_context_data(*args, **kwargs)

    js = ""
    css = ""

So that would address my point 1) "the function arguments have to be defined only once".

[JuroOravec]

Update on this:

This is one thing that I was thinking about, but don't think there's a non-breaking solution, because to make it work reliably, the CSS variable would have to be scoped under the top-level element of the HTML template. And for that we'd need to parse the HTML with beautifulsoup or similar to insert a class or other identifier to that top-level element.

Maybe this will all work out fine, but I am yet to run benchmarks to see how much overhead it would introduce.

Just came across this blog post, where selectolax came out to be much, much faster, taking only about 0.02 seconds to parse the page. I haven't run the script myself, but looking at the webpage they scraped, it currently has a size of 750 kB

So that's about about 0.026 seconds per MB of HTML. Assuming it might be same to convert it back to HTML, then the cost of modifying the outgoing HTML might about 0.050 seconds/MB.

In exchange, we'd gain:

1. Being able to access root element(s) from within the JS script
2. Using CSS variables (they would be scoped under the root element(s))

Sounds like a good deal to me!

[EmilStenstrom]

@JuroOravec I'm not following, why would we need to parse the HTML?

This is one thing that I was thinking about, but don't think there's a non-breaking solution, because to make it work reliably, the CSS variable would have to be scoped under the top-level element of the HTML template. And for that we'd need to parse the HTML with beautifulsoup or similar to insert a class or other identifier to that top-level element.

[JuroOravec]

Yeah, so imagine that we want to use variables from Python in component's CSS:

.my-class {
    background: var(--my-var)
}

If there was multiple components with perhaps diffferent values of my-var, then this would generate multiple CSS files, e.g.

.my-class {
    background: red;
}

.my-class {
    background: blue;
}

But because of how CSS works, these two definitions would conflict, and in the end ALL instance of the component would end up having the same value.

So one way to avoid conflict is what e.g. Vue does with scoped css.

In Vue, when you write

<style scoped>
.example {
  color: red;
}
</style>

<template>
  <div class="example">hi</div>
</template>

then it renders

<style>
.example[data-v-f3f3eg9] {
  color: red;
}
</style>

<template>
  <div class="example" data-v-f3f3eg9>hi</div>
</template>

So it does two things:

1. Adds a unique identifiers to the top-level HTML element(s) in the template, specifically the data-v-f3f3eg9.
2. Modifies the CSS to insert those unique identifiers e.g. example[data-v-f3f3eg9].

Now, Vue's approach is more complex then what I want to do. Dunno if they put the unique identifier on ALL HTML elements, or only the ones mentioned in the CSS. But basically I don't know how they know that .example belongs to the top-level element.

I think what we could get away is to:

1. Identify the top-level HTML elements in the template
2. Insert some unique identifiers, similar to data-v-f3f3eg9
3. Prepend the CSS definition with something like this:

   [data-v-f3f3eg9] {
       --my-var: red;
   }

Then, it should be enough for users to just write:

.my-class {
    background: var(--my-var)
}

and different instance of the same component will be able to have different values of --my-var.

---

But for this all to work, we need to parse the output of the component as HTML, and then insert that data-v-f3f3eg9 to top-level elements.

[dalito]

I am no expert at all on this but wonder if css-scope-inline be an alternative for css? Longer description of what it does at https://blog.logrocket.com/simplifying-inline-css-scoping-css-scope-inline/

[EmilStenstrom]

First of all, let's split JS variable support, and CSS variable support into two different features. They can be worked on and released independently.

Hmm... lots of options for CSS here. I wonder if there's a way to do this without necessarily including the CSS over and over again for each time we include a tag.

Here's one very light-touch idea:

Given this component "example":

• Template: <div class="example">Text</div>
• CSS: .example { background: var(--my-var, red) }

And the following calls:

{% component "example" / %}
{% component "example" my_var="blue" / %}
{% component "example" my_var="yellow" / %}
{% component "example" my_var="yellow" / %}
{% component "example" my_var="red" / %}

We would like to:

• Scope all CSS to our components so it doesn't leak to the rest of the page
• Allow changing the CSS on a component by component basis
• Maximize the cacheability of the CSS
• Minimize the size of the generated HTML/CSS

We could render this:

<style> 
    /* Fully cachable and only needs to be included once */

    .example.comp-f3f3eg9 { 
        background: var(--my-var, red) 
    }
</style>
<div class="example comp-f3f3eg9">Text</div>
<div class="example comp-f3f3eg9" style="--my-var: blue">Text</div>
<div class="example comp-f3f3eg9" style="--my-var: yellow">Text</div>
<div class="example comp-f3f3eg9" style="--my-var: yellow">Text</div>
<div class="example comp-f3f3eg9" style="--my-var: red">Text</div>

This would mean that we need to parse the HTML and the CSS. When we do, we add a unique identifier to the HTML and CSS, set any CSS variables with a style tag.

Update: We could in fact let the use handle setting the style by modifying their template:

• Template: <div class="example" style="--my-var: {{ my_var|css_safe }}">Text</div>

Thoughts?

[JuroOravec]

This would mean that we need to parse the HTML and the CSS. When we do, we add a unique identifier to the HTML and CSS, set any CSS variables with a style tag.

@EmilStenstrom I was thinking of this too, but:

1. Don't know if there's a library that supports this (and I would rather not try regexing the HTML ourselves, it feels like it could have a lot of edge cases)
2. By parsing the HTML fully, we can support multiple top-level elements, like React and Vue does.

Otherwise I agree with everything!

[JuroOravec]

@EmilStenstrom Sorry, update, I didn't read it properly, I'm a bit hungover today 😅

This part I fully agree with:

We would like to:

• Scope all CSS to our components so it doesn't leak to the rest of the page
• Allow changing the CSS on a component by component basis
• Maximize the cacheability of the CSS
• Minimize the size of the generated HTML/CSS

---

When it comes to this:

<style> 
    /* Fully cachable and only needs to be included once */

    .example.comp-f3f3eg9 { 
        background: var(--my-var, red) 
    }
</style>
<div class="example comp-f3f3eg9">Text</div>
<div class="example comp-f3f3eg9" style="--my-var: blue">Text</div>
<div class="example comp-f3f3eg9" style="--my-var: yellow">Text</div>
<div class="example comp-f3f3eg9" style="--my-var: yellow">Text</div>
<div class="example comp-f3f3eg9" style="--my-var: red">Text</div>

Personally I'd go with example below:

<!-- User's component CSS -->
<style>
    .example {
        background: var(--my-var, red) 
    }
</style>

<!-- CSS variables -->
<style> 
    [data-css-f3f3eg9] { 
        --my-var:  red;
    }
</style>
<style> 
    [data-css-03ab3f] { 
        --my-var:  blue;
    }
</style>
<style> 
    [data-css-bb22cc] { 
        --my-var:  yelllow;
    }
</style>
...

<div class="example" data-css-f3f3eg9>Text</div>
<div class="example" data-css-03ab3f>Text</div>
<div class="example" data-css-bb22cc>Text</div>
...

While it is more verbose, it doesn't touch the style attribute. I prefer not touch style, because I'd prefer to avoid mixing library's internals with user's definitions.

I'm imagining that someone would be using django_components and doing snapshot testing. And where they define component's HTML something like:

<div class="example" style="background: red">
    Text
</div>

Then they would get back e.g.:

<div class="example" style="--my-var: blue; --my-other-var: red; --my-other-other-var: green; background: red">
    Text
</div>

Instead, if we just insert the identifiers, there'd be less potential for clashes:

<div class="example" style="background: red" data-comp-f3f3eg9>
    Text
</div>

---

Update: We could in fact let the use handle setting the style by modifying their template:

Template: <div class="example" style="--my-var: {{ my_var|css_safe }}">Text</div>

Don't like this as much, for 3 reasons:

1. If there was multiple top-level elements, they would have to do it for each of them.

2. One would have to track the CSS variable through Python, HTML, and CSS files. On the other hand, if we defined the CSS vars in get_context_css_data, and then used them in the CSS files, it'd touch only the Python and CSS files.

3. It feels like a user would need to have more experience / knowledge to know how to do this. On the other hand it feels like a lower mental barrier if we hide the style="--my-var: {{ my_var|css_safe }}" away.

[JuroOravec]

I am no expert at all on this but wonder if css-scope-inline be an alternative for css? Longer description of what it does at blog.logrocket.com/simplifying-inline-css-scoping-css-scope-inline

@dalito That's an interesting one! Reading about it, I realized there's actually 2 separate feature ideas floating around in this discussion on CSS:

1. Using Python variables in component CSS
2. Scoped CSS (AKA CSS selectors do not leak outside of the HTML of the component)

In one of my comments above, I mixed the two when I talked about Vue's scoped. My bad.

So when it comes to "scoped CSS", I can imagine that there could be 2 modes of operation:

1. By default, the CSS would NOT be scoped (like described here).

2. User could set the CSS to be scoped for given component. In that case, we would add some identifier to the CSS's <style> tag, e.g. <style data-css-03ab3f> ... </style>. And we would modify the script of css-scope-line to work only with the style tags that start with this identifier. The identifier would also be used to find the corresponding HTML elements.

So, this way, we should be able to support both, CSS variables, and scoped CSS.

[JuroOravec]

Ayooo, I've got the proof of concept working!

[JuroOravec]

Also got the scoped CSS working 🎉

[JuroOravec]

One more remaining change is to allow to merge the JS / CSS, because currently the CSS / JS for every component and every input is fetched separately:

[JuroOravec]

And got that working too!

Click to expand

Compare inlined: vs not:

---

Now, I took a different approach than we discussed in https://github.com/EmilStenstrom/django-components/issues/478. Instead of specifying which JS / CSS to load via request headers, I inserted a <script> tag that is generated to call the client-side API.

With this, IMO a support for HTMX fragments could be implemented as so:

1. Introduce Component.fragment() and Component.fragment_to_response() methods, which accept all the same inputs as Component.render() and Component.render_to_response(), except also an additional fragment kwarg (required), specifying the CSS selector of the fragment.
2. Internally, fragment() would call render() to get the rendered HTML.
3. Then, it would find the CSS selector within that HTML using selectolax, and insert into it the equivalent of {% component_dependencies %}.
4. Next we'd call the same logic for processing the JS / CSS dependencies, as I used in the demo in the previous comments. This would create a JS Githubissues.
5. Githubissues is a development platform for aggregating issues.