Ability to split metadata yaml into files

[m4rcelofs]

Our use case:

• A few hundred tables
• YAML files manually configured/updated on Git
• CI/CD based on git and the cli-migrations-v2 image

The CLI forces us to have all table configuration in a single tables.yaml file that is a few thousand lines long and... a mess.

We would like to be able to something like this:

/metadata
-- table1.yaml
-- table2.yaml
      ...
-- tableN.yaml

Which would allow a much easier (and safer) process of development and review.

This might be somewhat related to https://github.com/hasura/graphql-engine/issues/3545, but I've been unable to find a PR for that and the CLI fails if you don't have a tables.yaml file.

[GavinRay97]

If you need a workaround solution, I happen to have a script for personal use which does this.

It takes a directory of "model" folders containing a SQL DDL model for the table, and a table.yaml, and merges them all into one schema.sql and tables.yaml. So you could run this as a pre-commit hook or CI step.

[m4rcelofs]

If you need a workaround solution, I happen to have a script for personal use which does this.

It takes a directory of "model" folders containing a SQL DDL model for the table, and a table.yaml, and merges them all into one schema.sql and tables.yaml. So you could run this as a pre-commit hook or CI step.

In hindsight, our case is "simple" enough that we can run this in our docker image:

RUN cat /hasura-metadata/tables/*.yaml > /hasura-metadata/tables.yaml

I'm not sure how that applies for other metadata, though; actions.yaml, for instance, has an actions: main key that seems to make it harder than to simply concatenate text files.

[malkhuzayyim]

It would be absolutely awesome if this was the default. Multiple files instead of one master "tables.yaml" file.

[ferm10n]

@m4rcelofs how would it decide how things should be split? I was thinking maybe it could do it based on the schema the table belongs to? But that only really makes sense for my use case. I assume others might want something different.

If the metadata is already split, then it's easier for the CLI to determine where things should go. You'd only have to worry about where new things added while using the hasura console should go.

And then there's the issue of non-table related metadata. Splitting by schema doesn't work for things like actions, which aren't associated with a schema! But again, if it's already been split before it's easier to deal with.

Maybe the solution is to require that metadata folders are already created and the stuff is split out in advance. Then let the user select the active folder to put new metadata into by running hasura console in there. To get it to work though, you'd need to have a reference to the other metadata folders in there, so when the CLI wants to write the metadata, it knows what already exists in other folders and doesn't create duplicates.

So, maybe your project with migrations and metadata could look something like this:

hasura-stuff
├── dir1
│   ├── config.yaml
│   ├── metadata
│   │   ├── tables.yaml
│   │   └── ...
│   └── migrations
│       ├── 0000000000000_dir1
│       │   ├── down.sql
│       │   └── up.sql
│       └── ...
└── dir2
    ├── config.yaml
    ├── metadata
    │   ├── tables.yaml
    │   └── ...
    └── migrations/
         └── ...

Then in each config.yaml you could have

version: 2
# ...
metadata_directory: ../*/metadata
migrations_directory: ../*/migrations

But that might get annoying with redundant config.yamls

[malkhuzayyim]

@ferm10n If I may chime in.

I think a primary candidate for splitting is the "tables.yaml", as i imagine it is by far the most used file in hasura for the average use case, as all table info, permissions and events go there.

So the directory structure may stay the same for everything else in a hasura project, just the tables.yaml would become something like:

tables/
- table1.yaml
- table2.yaml
- table3.yaml
...etc

That would be a quick and easy win I believe, its probably very viable to do that while maintaining backwards compatibility too.

[ferm10n]

@malkhuzayyim so you're thinking of having a yaml file per each tracked table?

Or some arbitrary splitting defined by the user?

Maybe this issue should be renamed "Ability to split tables yaml..."

[malkhuzayyim]

@ferm10n just a standard "file per tracked table/view" would be a huge win over having a single yaml file which is in the thousands of lines long. And the switch is pretty straight forward.

It's true I'm not considering other .yaml files in the metadata, but I'm personally only invested in having the tables file split up. The rest I can live with being in one file as I don't rely on them that much.

[ilijaNL]

Bump!

[jcarrus]

I'm pretty sure multiple files are used in config v3, which is the default since graphql-engine (hasura) v2.0.0

https://hasura.io/docs/latest/graphql/core/migrations/upgrade-v3/

[malkhuzayyim]

True, this issue should be closed as hasura-cli config v3 now works this way.