drupal-pattern-lab / roadmap

2 stars 0 forks source link

Solve the problem of mapping Drupal fields to pattern Variables #8

Open cybtachyon opened 7 years ago

cybtachyon commented 7 years ago

Summary:

Benefits:

To Do:

One of the hard problems with making Drupal play nice with a pattern library is mapping your backend Drupal entities and fields to the pattern variables. Some patterns expect strings, or arrays, or can even accept complicated objects for rendering that make use of recursive rendering from the parent platform.

This would be relatively simple to solve if we just use schemas for our patterns and imported those at run-time into Drupal. Unfortunately for that approach, many Drupal sites have a complicated set of backend data, entity, and field organization that makes it impossible to map fields 1:1 to a frontend. This also makes it hard to dynamically generate data models in Drupal for all scenarios, since often we’ll want to include values contained in other fields/entities.

1:1 mapping is easier though than tackling data mapping, and there are a couple of solutions that implement it pretty well.

A common solution for data mapping in Drupal 8 is to use “presenter” twig templates that include a pattern from the library and do all the complicated data mapping in the Drupal “presenter” template. This essentially looks like:

Drupal controller, view node response render -> node.tpl.php (presenter data-mapping template) -> twig include patterns/myCustomPattern.twig

One possible solution is to use a Pattern Renderer API service, that allows a configuration tool to display and set replacement tokens for Drupal to manage. The issue with this is that it relies on developers to create those tokens for all of the backend data.

Another possible solution is to have a Drupal module that has a configurable set of data mapping transformations, much like Views, that allows assigning fields to the pattern schema variables.

Here are some of the current open source community attempts at solving this problem, in order of scope from smallest to largest:

The teams I work with internally already have a number of projects in this vein and are planning on moving forwards with a consolidated project starting August 21st.

cc @ThatGuyCND @EvanLovely @evanmwillhite

jesconstantine commented 7 years ago

@cybtachyon very excited to see this issue over at the Commonwealth of Massachusetts! On the openmass project we are using a custom module (named after our Pattern Lab design system) to handle data mapping, transformation and structuring. It's resulted in a developer-centric process for adding content types and patterns. I'd love to see the community standardize on a method that takes advantage of site building and Drupal's UI to handle some of this.

Something that I think ties into some of the projects you've listed above is using pattern data structure "documentation" or "definition" (via json schema for pattern builder and yml definition files for ui patterns) in that data mapping process. I think it would help the community to standardize on a format for this definition / documentation and have those files live in Pattern Lab (much like the json schema files live in pattern kit).

What I really like about the way that pattern builder / pattern kit uses json-schema is that it serves multiple purposes:

For those reasons json schema would get my vote, but I suspect there are ways to accomplish multiple purposes with yml or some other format as well.

Should I create an issue on the roadmap and link to it from here?

EvanLovely commented 7 years ago

I think this is crucial work; thanks for getting the ball rolling! I don't have the bandwidth to dive fully into this, but it's something I want to. That being said, I think a great first start would involve something along these lines:

Essentials

Nice to haves

Really, really nice to have

Dreams

cybtachyon commented 7 years ago

@jesconstantine @EvanLovely Thanks for the initial support and details. I'm not sure we need another roadmap item if this issue covers all the bases.

I'd like to recommend that a good end result for this roadmap issue would be either a new Drupal contrib module or a fork of an existing Drupal module.

There is a matching issue for Pattern Lab schemas here: https://github.com/pattern-lab/patternlab-php-core/issues/117

In the past I've exclusively used Twig DocBlock annotations and the PHP annotation engine to load pattern schemas for a Drupal editor. Personally, I think it'd be great to support DocBlock, JSON, and YAML, although the RH team will be focusing on JSON initially since that's how our internal pattern lib is built out.

We have an initial D7 draft in https://github.com/PatternBuilder/pattern-builder-drupal that we're looking at:

I definitely like https://www.drupal.org/project/ui_patterns but much of it is architected around a Drupal 8 1:1 mapping of data to view model, which just doesn't work at the enterprise level. A bit of a pipe-dream for me would be for PatternKit's Drupal module to become the D7 version with an upgrade path, and for the D8 ui_patterns to be refactored to allow arbitrary data mapping. Part of the work needed for that has been done here by @ademarco : https://github.com/nuvoleweb/ui_patterns/issues/81

sghoweri commented 7 years ago

I wholeheartedly agree with everything mentioned here - holy crap yes!

By far, this whole data definition / translation / validation piece has been the single most common Pattern Lab + Drupal painpoint we've experienced over the past year in building up and scaling out our design system across our digital properties.

Considering how far Pattern Lab and Drupal integrations have come in the past couple years, sharing the same templates, sharing the same namespaces, sharing the same data structures, and more recently, sharing the same Twig extensions, I’d say this is probably THE biggest yet-to-be-solved problem moving forward, even if it’ll take some work to get us there.

Regarding Markdown-based Solutions

Just as a quick aside, I think markdown is an excellent way to document patterns in a Design System / Pattern Library if the content or data being documented is automagically always up to date with the source of truth (ie. not hard coded).

For example, let’s say you had a card component in your pattern library that allows it’s background color to be set to one of a handful of options that’ve been set in your CSS.

Rather than having to manually copy and paste the data for these options into your markdown docs (and hope your documentation doesn’t inevitably fall out of sync), we’re instead automatically exporting the Sass map data for these background color options as a JSON file to the pattern lab and referencing those values in Pattern Lab AND in our markdown docs.

👆🏼That’s why I put in this still open PR 9 months ago to have Pattern Lab process markdown files for any embedded templates so that things like this are possible: https://patterns-dev.pega.com/?p=viewall-molecules-cards

So Where Do We Go From Here?

In my opinion, I think we should take a seriously hard look at @micahgodbolt’s Silex-based (and seriously awesome) Pattern Kit project and incorporating the best aspects of it into Pattern Lab.

Pattern Lab needs to evolve to keep up with the massively growing needs of Design Systems big and small.


Top things that come to mind:

  1. Modernize the way patterns get rendered. We should be rendering HTML via server-side, path based routes instead of outputting every single pattern as an HTML file in order to address scalability issues with PL and to seriously cut back on the amount of code being changed with even the smallest of template or json file changes. i.e. render patterns like how every other system like Symphony, Grav, Drupal, Silex, etc renders templates.
  2. Decouple the server / rendering side of Pattern Lab from the viewing / Interactive side of Pattern Lab. Pattern Kit does a great job with this — that’s a fairly large reason why / how there’s a Drupal 7 module that talks to the Pattern Kit rendering / validation routes to make sure shared templates validate. Pick your shares template, pass in data, validate the data, render, return the results, profit. Why couldn’t Pattern Lab do the exact same thing?
  3. Data validation via json schemas. I love love love the idea of having “blueprint” specs for patterns being defined that @jesconstantine mentioned. Stub out the data structure, the things allowed / disallowed to be passed in, etc and automatically generate form for Pattern Lab / Drupal / whatever to pass data into said pattern. Easily the biggest feature in Pattern Kit with tons of examples out there doing this in React, Angular, or just plain vanilla JS
  4. Client side rendering (thanks to #s 1, 2, and 3), even if we need to fake it. Thanks to the JSON schemas defined for each pattern + server rendering + ability to pass in data to patterns in Pattern Kit while doing it, doing things like the a client-side schema / data editor are possible

Pattern Lab isn’t Perfect But Neither Is Pattern Kit

All this said, there are still some issues I have with the current state of Pattern Kit — namely missing many a lot of the most important Pattern Lab features like lineages and pseudo patterns, but also, like pattern Lab, being far too opinionated on where patterns, data files, schema files, etc need to live on the file system.

For example:

  1. Why can’t we just have a couple base folders to look for pattern-related files in and then find files with the same base pattern name + special file extension and call it a day?
  2. Why do all our Twig templates have to live in the same, single _patterns folder in PL?
  3. Why can’t I have a schema file live right next to my Twig file in Pattern Kit?
  4. Why can’t I bring my own UI to interface with the behind-the-scenes processing Pattern Lab provides?
  5. When are we going to be able to get installable, versioned patterns in Pattern Lab or is automatically symlinking the crap out of your Design System the best way to go?
  6. Why does all this have to be so difficult?

Given all the really awesome, really exciting ideas being floated around here would be a major departure from Pattern Lab (and given that we still hasn’t seen any PRs reviewed or approved in over a year 😒...), I seriously think @EvanLovely, @evanmwillhite and I need to have a heart to heart video chat with @bradfrost.

I’m really excited by all the awesome progress that has been made by the Drupal Pattern Lab Working Group in the past 5 months — I just think we need to figure out if this growing initiative we have here can still ultimately get merged back into the main Pattern Lab umbrella or if we’re fast approaching a point where this fork has no choice but to start becoming it’s own separate thing...

cybtachyon commented 7 years ago

@sghoweri The RH team already has these issues up for PatternLab that address almost all of your Pattern Lab feedback and suggestions here:

https://github.com/pattern-lab/patternlab-php-core/issues/116 https://github.com/pattern-lab/patternlab-php-core/issues/117

They could definitely use your :heavy_plus_sign: 1.

We don't really see this as a major departure from Pattern Lab - rather we see this as a natural evolution for the ecosystem and part of maturing Pattern Lab to a major enterprise-level UX tool.

As far as PR approvals - you can rest assured they won't hold us back. The RH team has an internal deadline of August 21st to fully slot development of whatever tasks result from these issues... whether or not there's a community consensus, as we have a strong business case and need to see them developed. PR approvals or not, our team needs to move forwards, and we'll be sharing our progress with the open source community the best way we can.

micahgodbolt commented 7 years ago

Love seeing all this activity happening around pattern builder/ kit! @cybtachyon you're in Bellevue? When are you hopping over to the Microsoft campus? We should do lunch.

sghoweri commented 7 years ago

we see this as a natural evolution for the ecosystem and part of maturing Pattern Lab to a major enterprise-level UX tool.

@cybtachyon 100% agree, whole heartedly and we may be able to help quite a bit with some of the heavy lifting with this as well.

I don't want to speak for @EvanLovely or @evanmwillhite but I feel the issues brought up here and here, and here (among the others referenced) would be more than welcomed additions to maturing and strengthening the Pattern Lab ecosystem, hands down.

Monorepo Side Thought

As I'm sure you're aware of, there's a lot of moving parts involved with this overarching initiative -- to the point that I'm now wondering if a monorepo setup like Symphony or Babel would make a lot more sense here.

For example, with regards to Issue #117, I'm thinking about what synchronized changes would be required in the Pattern Lab PHP Core, the Pattern Lab Twig Pattern Engine, and the Pattern Lab Styleguidekit Assets Default repos in order to add something like Pattern Kit's Twig rendering engine behind the scenes (both for rendering and for serving up the HTML) which could then open up the door for getting validation, routing, .patternlabrc configs, etc in place...

Integrating a Schema Editor w/ Pattern Lab...

Regarding the schema editor in Pattern Lab's interface itself (assuming the backend was in place to support it), about a month ago, I took a rough first pass at refactoring the PL Styleguidekit Assets Default project's JS into something that'd be a lot easier to work with / build off of (now using Webpack with proper dependency management) in addition to starting the process of replacing the PL interface's templating engine from Mustache to Twig.

There's still some mustache templates left that I'd like to get removed so the entire PL interface would entirely be getting handled via Twig (and I didn't crack open any major JS refactoring just yet) however this could potentially be a good starting place to get a lot of the new front-end architecture required off the ground...

https://github.com/drupal-pattern-lab/styleguidekit-assets-default/tree/feature/modernize-patternlab-js

evanmwillhite commented 7 years ago

This all sounds amazing. I'm on board for helping with any of it that I can, including doing what's best for the Pattern Lab ecosystem. This is very exciting - thanks to all involved.

EvanLovely commented 7 years ago

What's the MVP here? How can we ship something that just barely fits our needs in two weeks?

cybtachyon commented 7 years ago

What's the MVP here? How can we ship something that just barely fits our needs in two weeks?

@EvanLovely Sorry - to clarify, that's when we're planning on finalizing requirements and starting scheduling work (by August 21st).

The RH team has an internal deadline of August 21st to fully slot development of whatever tasks result from these issues

The hope is to have enough feedback that what we're doing won't preclude the deliverables being useful to the open source community, and coordinate efforts with anyone else working these issues.

bradfrost commented 7 years ago

Hey All, Thanks so much for all this. Further integration with Drupal sounds really exciting and valuable.

Given all the really awesome, really exciting ideas being floated around here would be a major departure from Pattern Lab (and given that we still hasn’t seen any PRs reviewed or approved in over a year 😒...), I seriously think @EvanLovely, @evanmwillhite and I need to have a heart to heart video chat with @bradfrost.

I guess I just want to clarify that I'm not really in the weeds with the Pattern Lab codebase. I follow along and help in a way with the overall strategy and vision of the tool, but I'm fairly ignorant of the nitty gritty. Both @bmuenzenmeyer (Node) and @dmolsen (PHP) are the primary maintainers of the project's code base, and would be better equipped to field questions, PRs, and so on. I also know the Gitter channel is a place where a lot of conversation happens: https://gitter.im/pattern-lab/

bmuenzenmeyer commented 7 years ago

I've been summoned!

I am interested in all this effort, and it jives well with thoughts I've had regarding the state of the codebase and the direction PL should grow to remain useful to existing and new teams, across platforms.

Where I'm most interested in helping / contributing / providing input is in the frontend code, styleguidekit-assets-default. I am a firm supporter of the core tenant Dave imparted in Pattern Lab 2.0 specification that the frontend should be platform-agnostic, allowing both (and any others) platforms to benefit from a shared user experience and continual updates. Therefore, I'd prefer to avoid a monorepo structure if at all possible.

Scanning through this and https://github.com/pattern-lab/patternlab-php-core/issues/117#issuecomment-319820513 it's clear to me there is energy, manpower, and already in-flight effort behind this proposal and further deepening the offering of PL PHP / Drupal. That's :fire: :fire: :fire: I haven't looked at the code closely enough to understand if it could "just" be a plugin - but I have a feeling the answer is no.

I'd add that there is equal energy, community and support around Node efforts, with a major release on slow-burn, engines for Mustache, Handlebars, Underscore, React, Twig and others in the works... plus a number of plugins, a full-fledged, CLI, etc etc etc. What I am trying to say is that we all have a lot on our plates and I welcome this effort whole-heartedly - with the request that we make it platform-agnostic.

I've seen a couple clear themes in supporting PL, and specifically the frontend, over the years:

Is there an appetite for creating a tracking issue / milestone in https://github.com/pattern-lab/styleguidekit-assets-default to better understand a punch list?

Lastly, I strongly suggest this large change be formally described using the structure Dave setup in https://github.com/pattern-lab/the-spec/. It has served us well in the past and lets us think through major changes methodically and even track their implementation progress. I see no reason why this schema feature being discussed doesn't qualify as a major cross-platform feature addition describing what Pattern Lab is (which will then also cascade into documentation on the .io site, too)

boring personal note At the risk of sounding like a broken record, I love love this and am excited to see it make it into the light of day for users. My personal life is calming down a bit finally, after client work, a job change, and a move to a different state, so you should be hearing more from me from now on :)
sghoweri commented 7 years ago

@bmuenzenmeyer 👍👍👍👍

Where I'm most interested in helping / contributing / providing input is in the frontend code, styleguidekit-assets-default. I am a firm supporter of the core tenant Dave imparted in Pattern Lab 2.0 specification that the frontend should be platform-agnostic, allowing both (and any others) platforms to benefit from a shared user experience and continual updates. Therefore, I'd prefer to avoid a monorepo structure if at all possible.

Is there an appetite for creating a tracking issue / milestone in https://github.com/pattern-lab/styleguidekit-assets-default to better understand a punch list?

Very fair points. And I'm 100% for keeping the community and the project ecosystems together wherever possible so that the work being done can mutually benefit everyone across the board. Soooo, would you be opposed to tag-teaming on a major revamp of styleguidekit-assets-default?

I'd be more than happy to put some time into writing up my own recommendations for styleguidekit-assets-default from a front-end architect point of view (keeping everyone mentioned here thus for heavily into consideration).

Quite frankly, I just don't want the experience people have with the Pattern Lab ecosystem (developers, designers, UXers, content authors... everybody) feel antiquated, horribly outdated and sluggish compared to the mind-blowingly awesome work being done in the community.

my own comment Keep the frontend templates hogan. Switching to Twig feels like it deviates too far from platform-agnostic to me.

What about something like composable web components, nunjucks, or Vue.js... (assuming we could maintain IE9+ compatibility out of the box)?


Scanning through this and pattern-lab/patternlab-php-core#117 (comment) it's clear to me there is energy, manpower, and already in-flight effort behind this proposal and further deepening the offering of PL PHP / Drupal. That's 🔥 🔥 🔥 I haven't looked at the code closely enough to understand if it could "just" be a plugin - but I have a feeling the answer is no.

Right. This is just one out of many many instances @EvanLovely and I have found ourselves butting heads with Pattern Lab's current limitations without many options (besides a lot of hacks and workarounds using Gulp / Node.js)

I'd add that there is equal energy, community and support around Node efforts, with a major release on slow-burn, engines for Mustache, Handlebars, Underscore, React, Twig and others in the works... plus a number of plugins, a full-fledged, CLI, etc etc etc. What I am trying to say is that we all have a lot on our plates and I welcome this effort whole-heartedly - with the request that we make it platform-agnostic.

One crazy absurd idea to throw out there... what if Pattern Lab PHP and Pattern Lab Node didn't need to be two separate projects... for example - assuming we could still use .php Twig extensions in Pattern Lab (so Drupal 8 compatibility is still maintained), couldn't we use something like https://github.com/bitmade/node-twig to use PHP template rendering, but with all other PL logic living in Pattern Lab Node?

Lots more to think and talk about, especially on how schema validation and mapping might work, how we can use the best parts of Pattern Kit as a source of inspiration, and how best to get the ball rolling on all this!

cybtachyon commented 7 years ago

Thanks everyone for the amazing feedback and suggestions.

It looks like we have two hot items that need addressing:

  1. A Drupal module that allows mapping data models from Drupal to a pattern library
  2. Platform-agnostic Pattern Lab updates to make it work better as a dev-tool dependency of other projects

A Drupal module that allows mapping data models from Drupal to a pattern library

This topic is primarily what this issue was created for, and I have a summary of proposals based on all the feedback so far:

Drupal 7

Drupal 8

One consideration is that ui_patterns could be too heavy weight (6 modules, plus lots of plugins) to bake into a best practices Drupal / Pattern Lab integration or add to core as an experimental solution for ingesting patterns.

Platform-agnostic Pattern Lab updates to make it work better as a dev-tool dependency of other projects

This is a bit of a topic derailing but it is important to making this issue work as well as developing the ecosystem. As @bmuenzenmeyer and @sghoweri recommended, we need the following issues:

If we're able to reach a consensus, we can go ahead and make the necessary issues, forks, and close this one out.

micahgodbolt commented 7 years ago

just wanted to loop @andrewmriley into this discussion.

SiliconValet commented 7 years ago

I'm totally on board for pitching in on a Drupal implementation once we're pointed at a solid implementation. (Commenting mostly to subscribe)

Also, just a heads up, I whipped up a lighter wrapper for patternkit here https://github.com/SiliconValet/pkplugins I believe that's what dreese was referring to when he said panels integration, patternbuilder doesn't have that afaik...

mariohernandez commented 7 years ago

Would love to see this implemented.

bmuenzenmeyer commented 7 years ago

Please review https://github.com/pattern-lab/styleguidekit-assets-default/milestone/1 for completeness from a shared frontend standpoint

bmuenzenmeyer commented 7 years ago

When someone more familiar with the technical aspects of the schema editor solution has time, could they please open an issue on https://github.com/pattern-lab/the-spec so we may discuss and formalize behavior? There are plenty of examples in closes issues how we typically format each issue.

Thanks!

jesconstantine commented 7 years ago

Hi folks! I’m curious to know if / how Drupal’s cache api (specifically cache tags, respecting cache invalidation) will be implemented or addressed in the work proposed here? Or if this is an issue (learn more on d.o) that component-based Drupal themers using pattern lab / non-drupal twig have already solved?

cybtachyon commented 7 years ago

@jesconstantine Great question - right now I'm mostly looking at solving this in the panels/block & field layer before it even gets to the Twig template. The idea I'm running with right now would be that "pattern" blocks as provided by a D8 port of https://github.com/SiliconValet/pkplugins would require "component" field formatters that immediately bubble their cacheability metadata to the block level. The downside being a need to implement new field formatters for many different types of fields.

Essentially, this is sidestepping the problem. I'm sure there are a bunch of things I haven't thought of yet that will show up in implementation, but that's the path I'm headed down at the moment.

cybtachyon commented 6 years ago

Coding is happening here: https://github.com/cybtachyon/patternkit

Immediate goals (D7 only):

Secondary goals (Q2 2018):

cybtachyon commented 6 years ago

Tagging in @SiliconValet , @bdone , @nielsonm , @ThatGuyCND , and @mrjmd as additional interested parties on this effort.

cybtachyon commented 6 years ago

Patternkit D7 v0.0.1-alpha1 code has been completed and is scheduled for internal review tomorrow, Tuesday January 30th, with subsequent deployment to drupal.org.

https://github.com/cybtachyon/patternkit

I'll be presenting the work in a talk at this upcoming PNWDS - stay tuned! https://pnwdrupalsummit.org

cybtachyon commented 6 years ago

Patternkit Super thanks to @SiliconValet - we have Patternkit D7 v1.0.0-alpha3 out and are working our way through the remaining alpha items before we start looking at beta.

https://www.drupal.org/project/patternkit/issues/2945656

This is already being used live on a production Red Hat property and we're looking at adding another relatively soon, so you know we're not joking around too much. ;)

Presentation The presentations are available here: https://www.fldrupal.camp/sessions/design-theming-front-end-development/implementing-enterprise-design-system-red-hat-way and https://pnwdrupalsummit.org/2018/sessions/implementing-enterprise-design-system-red-hat-way

It's mostly backstory and implementation discussion, but Patternkit + UI Patterns is featured (even if there are some minor hiccups).

Drupal Pattern Lab Mapping We've settled on @SiliconValet 's solution of using Drupal tokens to handle our mapping for now. It's a very robust solution that's easily extendable, and doesn't preclude adding other solutions or mapping UI's to the modules. The path we're headed down now is to get solid implementations for token-based mapping in D7 and D8 along with making them portable/migratable.

If you can add a few PatternLab Pattern Libraries to a D7 site (either in a module or theme), build a couple pages with them, and then easily migrate to D8, I'd wager we've achieved our first set of goals for this issue.

cybtachyon commented 6 years ago

Patternkit Alpha 4 is out: https://www.drupal.org/project/patternkit/releases/7.x-1.0-alpha4

We could absolutely use some help/attention on the remaining beta release blockers: https://www.drupal.org/project/patternkit/issues/2945656

Our internal team is focusing on Red Hat Summit (May 8-10) after which we will have a little more time to refocus on PatternKit D7 beta and its Drupal 8 port.

cybtachyon commented 5 years ago

Working on this (or at least chatting about it) at Drupalcon Seattle 2019.

cybtachyon commented 5 years ago

This is now actively being worked on at Red Hat.

Some good work from Phase2 here as well: https://www.drupal.org/project/preprocess/issues/3059988#comment-13147949

cybtachyon commented 5 years ago

Patternkit 8.x-1.x-dev has been released! https://www.drupal.org/project/patternkit/releases

Presentation: https://www.youtube.com/watch?v=8Ms4KIV4GP4

Additional WIP Issues: https://github.com/pattern-lab/patternlab-node/issues/999 https://github.com/pattern-lab/patternlab-node/issues/1005

cybtachyon commented 4 years ago

Patternkit dev has been added to Drupal-Pattern-Lab org in preparation for Alpha release. Timeline is 1-3 weeks.

https://github.com/drupal-pattern-lab/patternkit