Arbitrary nested data

[ewjoachim]

Hello, and thanks for the project,

I'm wondering if there's a way to add nested data in the Jinja context: if I try to put nested data in combine.yml variables (such as a dict), the variable is not injected in the context anymore.

What I'm trying to achieve is build a 4 page static website (1 main page + 1 page for terms & conditions, both with translations in 2 languages), and I want as much as possible of the data to not be hardcoded in the repo of the code, so I'll probably have a Python script read the data from the source and write a yml file somewhere before running combine. Nesting data in this case is needed to ensure that I have the same structure in the 2 languages, and avoid having to redefine my template twice with just different variable names, but if you think there's a better way to achieve this, I'll take it :)

Thanks !

[davegaeddert]

Thanks for pointing it out! I think it will probably work if you put it under the default key, like this:

variables:
  test:
    default:
      key: val
      key2: val2

That's an interesting quirk of how it's designed though...

For what it's worth, I have also been thinking that a from_file would be a nice feature (sort of like from_env) that would know how to load a YAML, JSON, etc. based on the filename. So you could potentially do something like this including your script:

# combine.yml
steps:
- run: python generate_data.py

variables:
  data:
    from_file: data.json

Let me know if the default key does it for you. I'd be open to pursuing from_file if that helps (or merging a PR if that's something you want to contribute).

[ewjoachim]

Thanks for your answer ! I'll check it out and let you know!

[ewjoachim]

It worked :)

If you're interested, I created a static website, with a single page, and different sections, and tried to make it so that as much as possible can be modified through Netlify CMS (which is an static website generator editor: it builds yml & md pages for you by committing them in git through the API)

The part that I found missing in Combine was that it's not possible from a given page to access the data for all other pages, which meant it's complicated to have 1 md file per section, and end up building a single html page from the different sections. What I ended up doing is a small python script that creates a dict from all .md and .yml files in the content part, and inject it into my combine.yml file. The problem with that is that it means calling this script can't be a part of the combine pipeline itself, because the combine.yml file is not re-read. It's not shiny, but it works :D (https://github.com/mathilde-website/mathilde-website)

(oh yeah, btw, it's not exactly the same website than the one I openned the ticket for, this one doesn't have the translations, but it's roughly the same issue)

[davegaeddert]

Looks cool! I've looked at Netlify CMS before, but haven't really tried it. I do think it would be useful to have some more tools to make that easier to integrate.

One of the things on my list is to refactor Combine a little bit to where a "Page" is a class/object and you do have the ability to access other Pages, filter/traverse them, etc. I've wanted similar things for building a table of contents style page and sidebar menus, but in the end it hasn't really been a big enough problem for me personally.

Anyway, I think some combination of these things would probably make it easier to build what you're after! Thanks for asking the question!

(and I hadn't really considered how macros could be used in Combine, so I'm glad I saw that!)

[ewjoachim]

I ended up doing the same thing for the other website (the one with 2 languages). https://github.com/mawi-website/mawi-website/tree/main/content

Yeah, it would be much easier if I could access all page data from all pages.

Using tailwind without using "macros/components/mechanisms that let you define chunks of code once" is a straght way to consistency hell because you need to repeat a whole lot of classes a bunch of time. Even then, I'm pretty convinved Jinja is not the right tool for the job.

(BTW, I should open an issue, but having an index.md is not supported apparently, I had to make an index.html, whereas en.md, cgv.md and cgv-en.md work well)

[davegaeddert]

If you mean that index.md didn't work in combine, feel free to share more details / an example. I use that pretty regularly and it works fine, so I would guess it's an implementation issue.

[ewjoachim]

I'll open a dedicated issue.

EDIT: hm strange. What I got the other time (and I reran combine multiple times to ensure it was not a 1-time bug) was that it created an empty index.html, AND it created an index.md that contained what my index.html should have contained.

Though... I just tried again and it worked the way it's expected.

Well, one less bug in the world 🤷

[davegaeddert]

Just a heads up — I added the from_file feature described above. I think that should make these kinds of build pipelines easier to do in general.

I think nesting raw dict variables under default is probably going to be the solution to this issue for now. Probably a good thing to consider revisiting when there's a new major version of Combine.

Still on my radar to provide more cross-page/site data in variables too, but that's going to be a more involved thing to consider.

Thanks for asking the questions!

[ewjoachim]

I think nesting raw dict variables under default is probably going to be the solution to this issue for now

Cool ! Please make sure to document it (if it's not already done), thanks !

[ewjoachim]

Still on my radar to provide more cross-page/site data in variables too, but that's going to be a more involved thing to consider.

Shall we create a issue to track that ?

[davegaeddert]

Added it to the docs.

You're welcome to make an issue for the page variables if you'd like — personally I tend to take the "it will come back up if it's a big enough need" approach, but an issue would give me an easy way to let you know if/when it gets added :)