Liquidgets: a counter-proposal

Hey guys,

I think our conversation about the roles of templates, layouts, liquid, and widgets has been really valuable. I think we're getting to the bottom of a lot of decisions we'd been putting off, and I'm glad we're dealing with it now.

However, I think the proposed overhaul based around liquid templates goes too far. I think there are some good ideas we should adopt, but I think that there is still a role for widgets. What I'm pitching here is a marriage of the widet plan with Omar's plan, pretty similar to what Eric and Tuuli have been suggesting, though similar in different ways.

Before I get into the specifics of my proposal though, I want to highlight a couple of priorities that I think should guide our design here. These are things that have come to me between reading the Google docs that led to widgets in the first place, re-reading Omar's proposal email and plugin github issue, reading the demo code he pushed up, talking to Tuuli and to Tara for a while each yesterday, and from all our preceding other conversations about the goals of Champaign.

Priorities -

expose the full power of markup, css, and js behaviour to power users
allow campaigners to complete their day-to-day workflow without seeing an HTML tag
allow optimizers to create any style of web interaction (eg a modal) without needing the developers' help
allow optimizers to control the layout and content of a wide variety of pages, such as after-action thank you pages / share pages
allow power users to create a microsite page, ie one that has none of the layout chrome (header footer etc) of the rest of the website
allow other developers to extend and contribute functionality by creating installable plugins with a basic suggested structure
enforce a consistent UI by default while allowing unlimited customizations
allow user input of content and configuration for any widget (eg thermometer goal, petition CTA text) through form fields subject to validation, not inline in markup

I have a plan that I think meets all these criteria. I'm calling it Liquidgets cause it wraps the fluidity of Liquid templates around the rock solid core of Widgets.

Here's the plan.

the Template formerly known as LiquidLayout

The Template class up to this point has been kind of awkward, and will be replaced by the LiquidLayout class that Omar created for his demo. Henceforth, I'm calling the new class Template cause it's more concise. Editing a Template involves editing raw HTML with Liquid tags, and is entirely responsible for defining the structure and layout of the page. However, the Liquid tags define only slots.
```
 <div class="full-width-header">
   { slot1 }
 </div>
 <div class="body-area">
   <div class="left-column">
     { slot2 }
   </div>
   <div class="right-column">
     { slot3 }
   </div>
 </div
```
In the example above, the CSS classes get style rules from either a) CSS from our codebase, which we've refined and committed to create mobile responsive implementations of agency's design, or b) whatever ad hoc CSS defined in a <style> tag in the liquid template itself by a precocious user. <script> tags are cool too.

Beside the one big HTMLiquid textarea, a template would also have a checkbox whether or not to display within the site-wide layout defined in the codebase that handles headers and footers, thereby enabling microsites and full-layout experimentation.
Widgets + Layout = Page

Once a diligent developer or optimistic optimizer has created a template, it makes it trivial for a careening campaigner to swing through and fill out the content. The campaigner picks the template. For each slot, they select a widget type from a drop down. Once a widget type is selected, the fields to fill out that type of widget appear in the form (this functionality is what I built last week). They fill out the fields, hit submit the widget, and it validates and saves on the server via AJAX while our daring campaigner plows on.

In our example above, slot1 would have an image widget, or a slideshow widget, or a video widget. slot2 would probably be be a text body widget (oh hell yeah we still have those) and slot3 would be a petition widget or a donation widget. Importantly, those are the kind of widget swaps I think a campaigner might reasonably make. But I reckon we should be able to set default widget types in the Template - see 'Outstanding Questions' below.

Here I'd like to add that we should design the CSS of the widgets so that they display reasonably well no matter how wide or how narrow the slot is. I'm hoping to write a lot of that CSS and it's gonna be totally chill.
Widgets got Ruby like its Tuesday

Despite Omar's complaints, I still believe that a ThermometerWidget Ruby class is an ideal way to organize code related to server-side behavior of a ThermometerWidget. Furthermore, I also love that we made it so we can write regular rails validations but still allow the widget to define whatever fields it wants through the magic of JSONB. That system is badass and robust and I think it's valuable to keep around. And with the inheritance from widgets and the association of pages to widgets, I think we've created any easy interface onto them for the rest of the app.

It's true that the Widget system was slightly complicated to build, I don't think it's complicated to use. I think there's inherent complexity in what we want to build, and the existing server side Widget inheritance model does a great job addressing the specifics of it - and if we trash it, we're eventually going to end up writing new code to deal with that complexity.
Would you like a Widget with that plugin?

I think that Omar's suggested plugin system through Ruby gems seems like a fantastic fleshing out of an issue that's been worrying me since we started this project. This propsal unabashedly rips it off, but changes what the expected directory structure is.
```
thermometer
|
|__ config.rb
|__ en.yml
|__ routes.rb
|
|__ models
|   |__ thermometer_widget.rb
|
|__ assets
|   |__ stylesheets
|   |   |__ thermometer.scss
|   |
|   |__ javascripts # js or .js.jsx
|       |__ thermometer_form.js.jsx
|       |__ thermometer_display.js
|
|__ views # can be slim, erb, liquid, whatever they want
|   |__ thermometer_form.slim
|   |__ thermometer_display.slim
|
|__ spec
|   |__ thermometer_widget_spec.rb
|
|__ controllers
```
There's a couple key files that define the Widget. The model defines the fields persisted in JSONB, any validation for the form, and any server side logic. The form is either given as a React component (high bar) or as a basic HTML form (low bar) which we automatically pick up into a React component for the aforementioned sweet interactive form. The front-end display of the widget is defined by its view, and optionally a javascript file or even a React component.

For most of the widgets we create, they'll live in the main repo, not in gems. But they'll still use the same encapsulating files, making them easy to extract to gems as we choose to.

It's also worth pointing out that plugins as widgets is just a recommendation. If someone wants to go hog-wild customizing champaign with a plugin, they can do whatever they want cause they have all the freedom of a gem.
Pages quit that campaign-a-normativity

So far we've referred to pages as campaign_pages throughout our codebase. But this customizability is useful for not just campaign pages, but for many types of pages - donation pages, after-action pages, share pages, etc. This isn't pressing for this proposal, but I think it's worth remembering that it's not just direct action pages that our optimizers want to be messing around with.

And that's pretty much it!

Outstanding questions

what happens if someone references the same slot twice in a liquid template?
what's the best way for a Template creator to define a default type of widget for a given slot?
what content should be first class fields of a page? I don't think we want a title widget, so besides the slots, I think we want pages to make some other first class liquid tags like title to be available.

Let me know your thoughts. I apologize for the length, but I wanted to be comprehensive in my specification.

Hey, so you didn't make it possible to edit your Google Doc. As a result, I'm going to do a lot of in-line quotes here to respond. Overall, I think this is a great proposal and you did a really nice job putting it together. I'm also really happy that you voiced your concerns so effectively.

I've got a long response here, and I want to give the context for this: My intention here is to respond as an engineer, not your boss. This response is not intended to shut down the conversation and none of these responses are final. I don't think my answers here are indisputably right, so I want the conversation to continue. So, if you read something that says "I don't think you've got this right", that's intended to say "You're wrong and this is the end of the conversation", but rather, it's intended to say "I don't think you've got this right, based on my point of view".

expose the full power of markup, css, and js behaviour to power users

I don't feel like this is a priority we want to optimize for. Our Power Users currently have all sorts of capability to edit HTML, JS and CSS and the overwhelming response has been "I don't want to do that" (both vocally and non-vocally, though the continued use of copy and paste mechanics).

allow optimizers to create any style of web interaction (eg a modal) without needing the developers' help

Honestly, I don't think this is a realistic goal, and I don't think it's been listed as a priority to this point; the most aggressive I can remember being is saying that a campaigner should be able to add a style of web interaction to a page without developer intervention. To me, creating new methods of interaction will very likely be a premature optimization that we'd be lucky if it was used five times outside of testing.

allow power users to create a microsite page, ie one that has none of the layout chrome (header footer etc) of the rest of the website

I'd make a tweak here that this should be "developers" and not power users. It seems very unlikely at this time that we'll be in a position where non-developer users are creating microsites.

allow other developers to extend and contribute functionality by creating installable plugins with a basic suggested structure

One thing that I think gets missed here a lot is that this functionality isn't (exclusively) for other developers. It's very much for us. We need a simple system of adding new functionality to the system.

enforce a consistent UI by default while allowing unlimited customizations

As we've spent more time thinking and talking about and bumping into the issues that come with unlimited customization, I've become much more comfortable with the idea that unlimited customization is an over-engineering case. It is OK for us to say no to some requests.

allow user input of content and configuration for any widget (eg thermometer goal, petition CTA text) through form fields subject to validation, not inline in markup

This I think is probably understated. Quality validation is critical to good UX and necessary for Champaign to be considered a successful system.

However, the Liquid tags define only slots.

I'm having difficulty understanding how this differs significantly from how I'm imagining it in my head. How do you see this as significantly different from what Tuuli's proposed?

Beside the one big HTMLiquid textarea, a template would also have a checkbox whether or not to display within the site-wide layout defined in the codebase that handles headers and footers, thereby enabling microsites and full-layout experimentation.

Want to plug this as something that we probably want to do no matter what. This is a great suggestion.

slot2 would probably be be a text body widget (oh hell yeah we still have those)

My big question here is what content would we need to display so that text body should be a widget instead of a part of the page? Even in our most image-heavy formats, I'd expect that we'd want some basic text there (for instance, even an infographic would likely have a bit of header text). If we're in that boat, is there a reason why there's a need for a replaceable text body widget instead of simply rendering an empty string for the page body?

Despite Omar's complaints, I still believe that a ThermometerWidget Ruby class is an ideal way to organize code related to server-side behavior of a ThermometerWidget.

The more we've talked about Thermometers, the more I'm thinking that making them a core part of the system makes sense. Thermometers are actually really, really simple when we don't over-engineer them (and I honestly expect that removing a lot of thinking about thermometers would be something our campaigners would appreciate). Essentially, my thought is that a thermometer is stored in its own table in the database, with a pair of fields - the current count and the goal. Any page could link to a thermometer object, and multiple pages could link to the same thermometer object (to handle linked thermometers). Each page also provides a basic label for the thermometer in the form of a string ("Actions Taken" or "USD Donated" or "Phone Calls Made") and we have some basic logic (encapsulated in the Thermometer) to provide default goals and goal increments, and campaigners basically never have to worry about interacting with Thermometers. They're a feature that's dead simple and I think that ActionSweet right now badly overthinks the entire thing.

Furthermore, I also love that we made it so we can write regular rails validations but still allow the widget to define whatever fields it wants through the magic of JSONB. That system is badass and robust and I think it's valuable to keep around.

Is that system more robust than simply allowing the widgets to define their own tables, though? Don't get me wrong - what you guys did with the JSONB validations was fantastic work, but if we're defining widgets as engines to be loaded, we put ourselves in a situation where each widget has full Rails-level capability made available to it. Do you feel like there are advantages to JSONB that outweigh providing each widget its own full Rails stack?

if we trash it, we're eventually going to end up writing new code to deal with that complexity.

Can you elaborate on what complexity you feel like using Engines is going to present that Widgets already handles?

This proposal unabashedly rips it off, but changes what the expected directory structure is.

I think this is a fantastic layout, but I also think it's worth raising that what you've created here is basically exactly the layout of a Rails Engine, give or take a config file or two.

For most of the widgets we create, they'll live in the main repo, not in gems. But they'll still use the same encapsulating files, making them easy to extract to gems as we choose to.

This flows from above - I'm not following the logic that says we shouldn't split them out, when they're already in a format which suggests that they should be. To me, the most convincing argument for splitting them out is dogfooding - if the plugin API works well enough for us to build these features now, it works well enough to build these features for other folks (or ourselves) in the future.

So far we've referred to pages as campaign_pages throughout our codebase. But this customizability is useful for not just campaign pages, but for many types of pages - donation pages, after-action pages, share pages, etc. This isn't pressing for this proposal, but I think it's worth remembering that it's not just direct action pages that our optimizers want to be messing around with.

It's possible here that campaign_pages are poorly named, because the intention is for them to encapsulate all or most of that functionality, though it's possible we might want to split say petition/donation/call/infographic/video pages into campaign_pages and create a separate concept for after action pages, as an example. But yes, very good point to bring up.

what happens if someone references the same slot twice in a liquid template?

I'd expect that this would be something we'd want to validate and return an error on.

what content should be first class fields of a page? I don't think we want a title widget, so besides the slots, I think we want pages to make some other first class liquid tags like title to be available.

It kind of feels like a lot of the discussion at this point is simply orbiting around this question. My gut currently says Title, Page Image, Thermometer, Main Text with everything else being defined by a couple of containers for widgets/plugins that could be slotted into them.

OK, that's all for me. Definitely welcome comments/questions/responses. This was a really good writeup, and you did a great job with it, as well as making sure that we could talk about the questions you still had. Thanks for putting this out there!

Just to comment on some of the priorities that Neal raised and that Eric responded too (noting that I'm not technically proficient enough to know how the "Neal approach" achieves these priorities better (or worse) than the "Omar approach". Given I'm product manager :)

Neal ID'd these priorities (and others):

expose the full power of markup, css, and js behaviour to power users allow campaigners to complete their day-to-day workflow without seeing an HTML tag allow optimizers to create any style of web interaction (eg a modal) without needing the developers' help allow optimizers to control the layout and content of a wide variety of pages, such as after-action thank you pages / share pages allow power users to create a microsite page, ie one that has none of the layout chrome (header footer etc) of the rest of the website

I think I somewhat land somewhere in the middle of Neal and Eric on this. If there were no engineering costs to this approach (maybe there aren't), everything that Neal outlines sounds pretty great. But I do think some of it is negotiable if there are engineering costs. For me, I think, the key thing is really less about whether it's a power user or a developer who needs to be involved, it's really about how easy and fast the results are to achieve.

The advantage of a good templating system as I see it is that it would allow anyone with a good knowledge of HTML/CSS/JS, but a poor or no knowledge of Champaign's code base or Ruby, to go in and customize all or most of the stuff that Neal is talking about. (Eric, I don't think this is currently the case on our stack). In other words, whether it's a "power user" like Eoin or an average developer who's never really worked with Champaign, the barriers for them diving in and doing one of the following should be very low. It should be a quick, easy task for anyone with the enough HTML/CSS/JS knowledge to:

set up a custom branded "microsite"
change he whole look and feel of the page, like moving the placement of page element or their style
(less clear on how this would work) add a new type of web interaction by turning the petition form into a modal (although agree with Eric that adding a very new type of interaction to a page feels like something that would definitely be a developer task, but again, it should be EASY to do)

The other advantage of a good templating system, as Neal correctly highlights as a priority, is that in the day-to-day workflow all of this is hidden from the campaigner. They don't need to engage in this at all.

Eric, I don't think this is currently the case on our stack

Assuming you mean ActionSweet - definitely not the case at the moment. In terms of Champaign, this is becoming more and more the case as we work on the Liquid proposals and both this one and the one that Omar proposed.

In other words, whether it's a "power user" like Eoin or an average developer who's never really worked with Champaign

As an aside, I think it's safe to simply shortcut anyone who can edit HTML/CSS/JS as a "developer". If you can write and debug JavaScript, you're a developer, regardless of whether that's your primary job title. Right now, that's the dev team, Eoin, and yourself.

It should be a quick, easy task for anyone with the enough HTML/CSS/JS knowledge to:

I feel confident about all three of these, in both the proposals that are on the table at the moment. In reality, I'm not sure that they differ significantly on these goals.

Great. I think the key thing I was trying to get at (and that Eric is saying all approaches achieve) is that there is some way in which the page HTML/CSS/JS is abstracted/surfaced/handled that means that anyone with some skills there can come in and change anything, without having to mess around with anything other than a template (i.e. don't have to mess around with source code), both for elements shared across almost all pages (e.g. the header/footer wrapper) and those associated only with a particular page template or a particular individual page.

expose the full power of markup, css, and js behaviour to power users I don't feel like this is a priority we want to optimize for. Our Power Users currently have all sorts of capability to edit HTML, JS and CSS and the overwhelming response has been "I don't want to do that" (both vocally and non-vocally, though the continued use of copy and paste mechanics).

I think I actually I actually copy-and-pasted that line from Omar's proposal. I stand by it, and based on the conversation above I think

However, the Liquid tags define only slots. I'm having difficulty understanding how this differs significantly from how I'm imagining it in my head. How do you see this as significantly different from what Tuuli's proposed?

I don't see this as significantly different than what Tuuli proposed at all - it's totally an appropriation of her ideas.

My big question here is what content would we need to display so that text body should be a widget instead of a part of the page? Even in our most image-heavy formats, I'd expect that we'd want some basic text there (for instance, even an infographic would likely have a bit of header text). If we're in that boat, is there a reason why there's a need for a replaceable text body widget instead of simply rendering an empty string for the page body?

I can definitely imagine cases where you want to play with that. What if you want to have WYSIWYG text in two different places - in an inset and a column, or a compare-and-contrast two column layout? It's not that I think we should make text body a widget cause we're going to not want to use it, but that I think it might want to go into multiple slots.

The more we've talked about Thermometers, the more I'm thinking that making them a core part of the system makes sense. Thermometers are actually really, really simple when we don't over-engineer them (and I honestly expect that removing a lot of thinking about thermometers would be something our campaigners would appreciate). Essentially, my thought is that a thermometer is stored in its own table in the database, with a pair of fields - the current count and the goal... They're a feature that's dead simple and I think that ActionSweet right now badly overthinks the entire thing.

I agree that Thermometers are really simple. But I don't think it makes sense to prioritize them over anything other particular effect. What if we test and find that showing recent signers blows it out of the water? I reckon they should obey the same encapsulation and stuff as every other widget type, especially since I think applying that encapsulation evenly makes them easier to work with.

...more to come...

This flows from above - I'm not following the logic that says we shouldn't split them out, when they're already in a format which suggests that they should be. To me, the most convincing argument for splitting them out is dogfooding - if the plugin API works well enough for us to build these features now, it works well enough to build these features for other folks (or ourselves) in the future.

Yeah, I'm agnostic about how aggressively we use the plugin system ourself for the widgets, though I imagine some (like petitions, thermometers, donations) should come in the box with the repo.

Is that system more robust than simply allowing the widgets to define their own tables, though? Don't get me wrong - what you guys did with the JSONB validations was fantastic work, but if we're defining widgets as engines to be loaded, we put ourselves in a situation where each widget has full Rails-level capability made available to it. Do you feel like there are advantages to JSONB that outweigh providing each widget its own full Rails stack?

I think it's more robust in that it plays nicely with ActiveRecord. If they each get their own table, doing stuff like @page.widgets becomes non-trivial. Furthermore, I think it's going to overkill and a ton of boilerplate to give every widget its own controller - most just don't need them. Beyond that, with JSONB it does have the full stack - it has a AR model with validations and ruby, views, react components... And if it needs a controller action, that's fine too.

expose the full power of markup, css, and js behaviour to power users I don't feel like this is a priority we want to optimize for. Our Power Users currently have all sorts of capability to edit HTML, JS and CSS and the overwhelming response has been "I don't want to do that" (both vocally and non-vocally, though the continued use of copy and paste mechanics).

This flows from above - I'm not following the logic that says we shouldn't split them out, when they're already in a format which suggests that they should be. To me, the most convincing argument for splitting them out is dogfooding - if the plugin API works well enough for us to build these features now, it works well enough to build these features for other folks (or ourselves) in the future.

I think exposing liquid templates would achieve with much more flexibility what we wanted to accomplish with the HTML widget. I was already talking with @paulaferris about what to do if campaigners will want to e.g. embed a tracking pixel into what is essentially another widget - e.g. middle of the text in a text body or inside a petition form.

slot2 would probably be be a text body widget (oh hell yeah we still have those)

My big question here is what content would we need to display so that text body should be a widget instead of a part of the page? Even in our most image-heavy formats, I'd expect that we'd want some basic text there (for instance, even an infographic would likely have a bit of header text). If we're in that boat, is there a reason why there's a need for a replaceable text body widget instead of simply rendering an empty string for the page body?

I thought we could have an 'abstract' as the mandatory text element. It'd be a ~paragraph long summary of the contents on the action page, and it'd work as a mobile body.

Yeah, I'm agnostic about how aggressively we use the plugin system ourself for the widgets, though I imagine some (like petitions, thermometers, donations) should come in the box with the repo.

The plugin system comes with the same shortcoming as the widget system - not everything needs to be a widget, and not everything needs to be a plugin. Not every pluggable component needs its own gem - what do we do about interactive widgets that just rely on some basic JavaScript, or even just HTML5? I'd make plugins for 'widgets' which require either extensive logic on our end, or which require third party service integrations. I feel like the conversation on whether a component should be baked into a page, a local widget or a plugin should be done on a case-to-case basis, which could have the unfortunate side-effect of making things messy.

Is that system more robust than simply allowing the widgets to define their own tables, though? Don't get me wrong - what you guys did with the JSONB validations was fantastic work, but if we're defining widgets as engines to be loaded, we put ourselves in a situation where each widget has full Rails-level capability made available to it. Do you feel like there are advantages to JSONB that outweigh providing each widget its own full Rails stack?

I think it's more robust in that it plays nicely with ActiveRecord. If they each get their own table, doing stuff like @page.widgets becomes non-trivial. Furthermore, I think it's going to overkill and a ton of boilerplate to give every widget its own controller - most just don't need them. Beyond that, with JSONB it does have the full stack - it has a AR model with validations and ruby, views, react components... And if it needs a controller action, that's fine too.

I really like how things are handled in JSONB and I think it makes sense to keep the current architecture for widgets that won't need to rely on their own plugins.

I think it makes sense to keep the current architecture for widgets that won't need to rely on their own plugins.

I guess my big question here is - what's that line? How do we know the difference between a widget that requires a gem and one that doesn't?

I think it makes sense to keep the current architecture for widgets that won't need to rely on their own plugins.

I guess my big question here is - what's that line? How do we know the difference between a widget that requires a gem and one that doesn't?

IMO, if it has a third party integration, or comes with a ton of very specific dependencies that would simply not be used in the rest of the project, it's a plugin. For the other cases it's a more difficult question. We could implement them locally, and if they start getting bloated, see a lot of reuse, or if we see ourselves building on top of them a lot, they could be extracted into plugins. It's not a black and white distinction and I see it as something where we should aim at being nimble.

Something I'd see as plugins right away would be e.g. share and donation functionality.

While I am not really able to follow much of this conversation, I am so glad we are having it, and love reading it!

The one line I want to address is this: "Expose the full power of markup, css, and js behaviour to power users"

To me "power users" should not refer to campaigners in general (who I agree should never see markup if it can be avoided), but to Paul, Eoin, me, Simona, and -- most importantly -- contractors that we may want to hire to work on new designs. (note: I understand the meaning of the word 'design' is ambiguous, just hang in with me for a bit). I think that it is important that we be able to hire outside people to make the site, or pages in the site, look significantly different than the default. For instance, with ActionKit, we were able to hire someone, inexpensively, to create a template that made AK look very much like AS. I'd like to be able to do the same kind of thing in Champaign, without putting too much burden on the dev team.

Eoin and I worked on a document of design questions that we had, based on some discussions we were having a few weeks ago. At this point, it may be helpful for the dev team to look at our list, and think about how difficult these things would be under the different architectural proposals. If any of these things are prohibitively difficult to do, then I would really like to understand why.

https://docs.google.com/document/d/1RgchqbPI4jcrJxD1QXIU9xdNiayzoHaKGJK1JIy9uYk/edit#heading=h.gq1ee8yn21w2

Let me know if you have any questions about this doc,

Tara M. Harwood Manager of Analytics & Data Science SumOfUs.org

tara@sumofus.org (510) 725-2979

On Thu, Aug 6, 2015 at 11:04 AM, Tuuli Pöllänen notifications@github.com wrote:

I think it makes sense to keep the current architecture for widgets that won't need to rely on their own plugins.

I guess my big question here is - what's that line? How do we know the difference between a widget that requires a gem and one that doesn't?

IMO, if it has a third party integration, or comes with a ton of very specific dependencies that would simply not be used in the rest of the project, it's a plugin. For the other cases it's a more difficult question. We could implement them locally, and if they start getting bloated, see a lot of reuse, or if we see ourselves building on top of them a lot, they could be extracted into plugins. It's not a black and white distinction and I see it as something where we should aim at being nimble.

Something I'd see as plugins right away would be e.g. share and donation functionality.

— Reply to this email directly or view it on GitHub https://github.com/SumOfUs/sumofus/issues/11#issuecomment-128460489.

SumOfUs / sumofus

Liquidgets: a counter-proposal #11

Here's the plan.

Outstanding questions