Blueprint for Pattern Lab’s Internal Plumbing?

sghoweri commented 5 years ago

In my opinion, I think we should take a step back to make sure we understand all the different ways that Pattern Lab should be able to be configured / customized in order to be suitably flexible for a huge range of use cases and diverse needs.

Step 1. Configure

Ignoring X ticket for a moment, I think we can boil things down into the 3 high level ways that Patterns themselves should be able to get configured:

Via the physical structure and names of folders / files
Via pattern-specific configs
1. JSON
2. YAML
3. Markdown front matter
4. JavaScript?
Via a global config file (basically allow the first two layers to be overwritten)
1. JSON
2. JavaScript / .rc config, etc

Step 2. Aggregate

Without diving into the specifics, each one of these layers of data gets created via some under the hood processes (ex. recursively walking through the file system).

Ultimately, we end up with up to 3 layers of data that get merged together — this represents what we want Pattern Lab to look like, how we want the navigation to be named and structured, what components need to get compiled into HTML, etc.

^ basically we end up with a blueprint of what and where things need to get compiled and how we want things structured.

This step includes things like:

Generating the final data that’ll get used to compile a particular template
Generate the data that powering PL’s navigation
Generate the data that powers PL’s search

Step 3. Compile

The last major step (as I see it) is to take the blueprint of what we want to build and to actually compile all the different parts that comprise Pattern Lab:

This includes things like:

Compile all the different templating languages for the different templates found into HTML
Outputting those compiled templates onto the file system in a place that PL knows where to find it

I know I’m glazing over a few details, however when I hear about bugs like the “Viewall Bug” or see incoming breaking changes like removing file / folder prefix support, I find myself asking the question:

What’s broken / what’s changing in this simplified architecture?

Is it how we implicitly / explicitly configure PL?
Is it how we gather up + mix and match the data from these different locations?
Or is it how we take the final blueprint of what we want and compile the final static site?

One thing in particular I’d really like to sketch out is what are all the different ways the file system / folder configuration approach gets used:

For example:

_patterns/
  atoms/
    button/
      button.twig
  molecules/
    card/
      card.twig

_patterns/
  atoms/
    button/
      button.twig
    card.twig

_patterns/
  atoms/
    button.twig
    card.twig

_patterns/
  overview.twig

_patterns/
  overview.twig
  atoms/
    button.twig
    card.twig

Taking a step back and coming up with a physical JSON or YAML schema that describes how each component’s internal PL metadata is structured seems like a reasonable place to start.

There’s lots more for me to dig into to make sure Pattern Lab is as flexible and stable as possible!

sghoweri commented 5 years ago

Cc @bradfrost @EvanLovely @bmuenzenmeyer

bmuenzenmeyer commented 5 years ago

This is an interesting discussion, with a lot to unpack! Thanks for taking the time to write it up

bmuenzenmeyer commented 5 years ago

It's cool you can conceptualize the steps needed to get PL marching. So much of it is tribal knowledge, too tightly-coupled with Mustache logic, and littered across files and callback hell to make much sense of it. I think you've captured the essence, omitting a lot of the details that make it work today like the PatternGraph, lineage, plugins (i know i know)

All of this adds up to a near or partial re-write of core. That's why I put the 4.0.0 milestone on this issue. That's not to say we cannot use this in 3.X land, but our priority should be in stabilizing the existing 3.x logic and clearing out the relevant PRs like #1008, then shifting to this new and shiny.

Step 1. Configure

Ignoring X ticket for a moment, I think we can boil things down into the 3 high level ways that Patterns themselves should be able to get configured:

Via the physical structure and names of folders / files

Via pattern-specific configs

JSON

YAML

Markdown front matter

JavaScript?

Via a global config file (basically allow the first two layers to be overwritten)

JSON

JavaScript / .rc config, etc

I'd love to get us back onto a stable footing, as far as configuration. The patternlab-config.json is as old as the fork itself, and better strategies exist. I think the CLI layer caused an extra bit of confusion. We started work in https://github.com/pattern-lab/patternlab-node/pull/1000 but I am unsure of it's status.

Ultimately, I'd love to use cosmiconfig to make configuration extensible.

Documenting the order of config and data inheritence is clearly important. Not sure what you mean by "Via a global config file (basically allow the first two layers to be overwritten)" as I think it'd be the opposite, with more and more specific data and config overriding global.

Step 2. Aggregate

Without diving into the specifics, each one of these layers of data gets created via some under the hood processes (ex. recursively walking through the file system).

Ultimately, we end up with up to 3 layers of data that get merged together — this represents what we want Pattern Lab to look like, how we want the navigation to be named and structured, what components need to get compiled into HTML, etc.

^ basically we end up with a blueprint of what and where things need to get compiled and how we want things structured.

This step includes things like:

Generating the final data that’ll get used to compile a particular template

Generate the data that powering PL’s navigation

Generate the data that powers PL’s search

I like step 2. Step 2 is :100: And what's pretty apparent to me is that we can refactor/rebuild toward these outcomes incrementally.

Step 3. Compile

The last major step (as I see it) is to take the blueprint of what we want to build and to actually compile all the different parts that comprise Pattern Lab:

This includes things like:

Compile all the different templating languages for the different templates found into HTML

Outputting those compiled templates onto the file system in a place that PL knows where to find it

🎉

I know I’m glazing over a few details, however when I hear about bugs like the “Viewall Bug” or see incoming breaking changes like removing file / folder prefix support, I find myself asking the question:

What’s broken / what’s changing in this simplified architecture?

Is it how we implicitly / explicitly configure PL?

Is it how we gather up + mix and match the data from these different locations?

Or is it how we take the final blueprint of what we want and compile the final static site?

This catches me off guard a bit - but it's valid criticism. My suggestion to remove the folder convention is outlined in https://github.com/pattern-lab/patternlab-node/issues/1028 - but it is a tactical retreat from the mess that is core after 7 or 8 years of evolution. The data model is too confusing to wrap one's head around. Obvious fixes dont work. The contract between core and uikit is undocumented and hard to debug (maybe I need spme help with sourcemaps).

One thing in particular I’d really like to sketch out is what are all the different ways the file system / folder configuration approach gets used:

For example:
_patterns/
  atoms/
    button/
      button.twig
  molecules/
    card/
      card.twig
_patterns/
  atoms/
    button/
      button.twig
    card.twig
_patterns/
  atoms/
    button.twig
    card.twig
_patterns/
  overview.twig
_patterns/
  overview.twig
  atoms/
    button.twig
    card.twig

This feels like a good issue. A TDD style approach where we document the supported paradigm and then code to it 😄

Taking a step back and coming up with a physical JSON or YAML schema that describes how each component’s internal PL metadata is structured seems like a reasonable place to start.

Agreed. I created this "schema" but I am not sure how many folks know about it.

There’s lots more for me to dig into to make sure Pattern Lab is as flexible and stable as possible!

your help is always welcome!

stale[bot] commented 5 years ago

It's hard to keep track of everything. This issue has been automatically marked as stale because it has not had recent activity, neither from the team nor the community. It will be closed if no further activity occurs. Please consider adding additional info, volunteering to contribute a fix for this issue, or making a further case that this is important to you, the team, and the project as a whole. Thanks!

JosefBredereck commented 5 years ago

One thing, in particular, I’d really like to sketch out is what are all the different ways the file system/folder configuration approach gets used:

For example:
_patterns/
  atoms/
    button/
      button.twig
  molecules/
    card/
      card.twig
_patterns/
  atoms/
    button/
      button.twig
    card.twig
_patterns/
  atoms/
    button.twig
    card.twig
_patterns/
  overview.twig
_patterns/
  overview.twig
  atoms/
    button.twig
    card.twig

One more thing to add here

_patterns/
  atoms/
    form/
      button/
        button.twig
  molecules/
    blocks/
      card/
        card.twig

But also

_patterns/
  atoms/
    form/
        button.twig
  molecules/
    blocks/
        card.twig

And the other points are good in common. But for the following there is one point that I want to call into question.

The last major step (as I see it) is to take the blueprint of what we want to build and to actually compile all the different parts that comprise Pattern Lab:

This includes things like:

Compile all the different templating languages for the different templates found into HTML

Outputting those compiled templates onto the file system in a place that PL knows where to find it

Explicit this Point

Compile all the different templating languages for the different templates found into HTML

Is it really necessary to have all the different template languages as a template to compile the source to HTML or would it be a good idea to focus on one? Then you have the possibility to inform the user how they can use their own/favorite templating language and can polish the one you are focusing on. It's like the discussion of "evaluate official sass support" or the use of TypeScript instead of JavaScript. It's better to give the users the tools to use the specific technology instead of building all by ourselves.

stale[bot] commented 4 years ago

It's hard to keep track of everything. This issue has been automatically marked as stale because it has not had recent activity, neither from the team nor the community. It will be closed if no further activity occurs. Please consider adding additional info, volunteering to contribute a fix for this issue, or making a further case that this is important to you, the team, and the project as a whole. Thanks!

pattern-lab / patternlab-node