Champaign vs SumOfUs.org

[NealJMD]

From @NealJMD on September 8, 2015 0:12

As we're getting ready to move forward with the designs from agency, I've started thinking about how to manage our designs with Champaign. There are a couple tradeoffs we have to consider, and I want to start a conversation so that we're ready by the time we get the designs next Monday.

Making static pages editable

There are plenty of static pages that will be part of our website - homepage, about us, staff, media, contact, etc. We want some of these to be editable by non-tech people. For example, adding a new staff member to the site probably shouldn't require a code change and a deploy.

On the other hand, some of these pages need programmatic logic. For example, the homepage pulls in recent victories and recent signatures from AK.

To decide if a page should be implemented using liquid in the database or slim in the repo, I think we'd want to evalute its logical complexity and its frequency of change.

• Changes often, high complexity - staff page (interactive, includes photos of each staff)
• Changes often, low complexity - jobs or FAQ pages
• Changes rarely, high complexity - homepage
• Changes rarely, low complexity - contact

The only category of those enumerated above that I think should be handled as a slim template in the repo is high complexity that changes rarely. I can only think of the homepage as something that fits this description.

Proposal - using campaign pages for all types of pages

When a user creates a page, they would select from a dropdown of the type of page that it is. The options that occur to me now are 'campaign', 'share', and 'base'. The primary function of choosing the type would be to change the url at which it's accessed - either /campaigns/my-slug or /share/my-slug or /my-slug. They can pick a layout too, which for things like the staff page would probably be a custom layout just for that one. Logic like pulling in recent signatures would be bundled into plugins, which would be referenced in the partials used in the layout.

One issue with this approach is consistent internationalization. I think the easiest way to handle it would be to have three different page instances for each slug, one for each language, and for the controller to sort out which one to serve.

Bundling assets outside of Champaign

Regardless of whether the pages of our design are implemented as liquid layouts or as Rails views, I think we'll want to have them in version control somewhere for development, along with the CSS and JS that makes it work. We don't want that stuff to exist in the main Champaign repo if we want our system to be usable by other people.

Up to this point, we've talked about bundling those things in the champaign-flute repo. However, I don't think that flute is the right place for that. Flute is a page-serving speed optimization. It's a bit complex and unconventional, and not something that I think we want to be a blocker for an MVP release. Furthermore, if we do get it working properly, it should be usable by anyone using Champaign, not bundled with the assets for SumOfUs.org.

Instead, I propose that we create a theme gem for the assets for SumOfUs.org. It would have a bunch of CSS and JS, and then it would have slim or liquid templates for each of the static pages. (The liquid layouts would get loaded into the db through a rake task like they are now). We could also give it the ability to override certain templates. For example, the campaign show page should probably have a slim template in the theme repo. Champaign would continue to hold the markup for the admin interface, and simple versions of the campaign pages, like it does now.

I've touched on a lot of different stuff here. I'd love to hear your thoughts. It could also be a case where what I've suggested here is a good enough first approach, but that as we move forward, we realize shortcomings that we want to change.

Copied from original issue: SumOfUs/sumofus#15

[NealJMD]

From @osahyoun on September 8, 2015 17:45

There are plenty of static pages that will be part of our website - homepage, about us, staff, media, contact, etc.

I wasn't aware of this. Champaign is a campaigning platform. I don't think we should also provide CMS functionality. Let organisations pick their own platform for that sort of stuff. There are plenty to choose from.

I think the pragmatic approach would be to style our campaign pages from champaign. When we've a solid MVP, let's come back to this issue, and decide how and where we should store our assets.

What do you think?

[NealJMD]

Definitely disagree. I think that most of the campaigning platform tools we've built have been a glorified CMS. You shouldn't need to install a whole different system just to add a few static pages to your campaiging website.

As to styling our campaign pages from champaign... why ever start down that road if we know we'll have to pull them out eventually anyway? It'll just be more confusing to extract Champaign stuff from SumOfUs.org stuff later. It also makes our open source project much less approachable to have a bunch of our assets committed into it.

[NealJMD]

From @osahyoun on September 8, 2015 18:8

Styling our campaign pages from within Champaign will make the job much easier. The developer feedback loop is just a basic file edit/ save/ refresh browser. You're definitely right; we should be concerned about separating SOU assets from champaign. We can do this by creating manifest files used only for SOU: sou.js and sou.css, then js/css/image assets can be namespaced within their own sou directories.

Re content management features, I hope we don't go down that route. It's really hard to do many things well. If we do (what are our medium term goals - @EricBoersma @paulaferris ?) then I think it should be managed by separate application (flute?).

[NealJMD]

From @EricBoersma on September 8, 2015 18:12

Definitely disagree.

I'd also disagree regarding CMS functionality - this is something that's actually a notable pain point for the current team, as they have to manage and remember two different systems depending on which part of the site they want to use. This makes some things which would be otherwise easy (like A/B testing our home page) pretty difficult. Being able to do the home page within Champaign is pretty highly anticipated.

[NealJMD]

Styling our campaign pages from within Champaign will make the job much easier.

That idea with namespacing with directory and imports is a good one, and I agree that the file edit/ save/ refresh browser feedback loop is critical for doing front end work. However, I'm pretty sure we can get that loop even when editing a gem by referencing it locally in development.

[NealJMD]

From @EricBoersma on September 8, 2015 18:28

I feel like the last response was to a different discussion, unless I missed something?

Edit: Sorry, I missed that it was referencing the stuff at the end of the original post. I wasn't in that frame of mind given the rest of the conversation.

[NealJMD]

two parallel discussions have opened - whether to add static page editing ability to Champaign, and whether to keep our SumOfUs.org assets in the Champaign repo.

[NealJMD]

From @Tuuleh on September 8, 2015 18:33

As I said earlier - when it comes to starting with the static pages, I'd begin with keeping it simple. We have so much to do elsewhere that I think we should concentrate our efforts in doing one thing well. Additionally, most of our users are digital campaigners, and will only be modifying campaign pages. I can see page customization as something that would essentially fall on the tech team (after all - there's no-one who's directly responsible for static pages when campaigners only create campaign pages).

I, for one, would like to know what the role of flute is in all this.

On Tue, Sep 8, 2015 at 11:29 AM, NealJMD notifications@github.com wrote:

to parallel discussions have opened - whether to add static page editing ability to Champaign, and whether to keep our SumOfUs.org assets in the Champaign repo.

— Reply to this email directly or view it on GitHub https://github.com/SumOfUs/sumofus/issues/15#issuecomment-138661662.

[NealJMD]

I feel like it would be good to get some product folks (@paulaferris, @edubsky) to weigh in on the priority of editing static pages through Champaign. If it's a not a priority, I'm happy to just use Slim templates.

[NealJMD]

From @EricBoersma on September 8, 2015 19:18

I think Tuuli's point here is a good summary of what I'd point out too: I don't think doing a heavy engineering pass on front page pages at the moment is going to be the best use of our time right now. We definitely don't have an aversion to re-working significant parts of our code base as we work on it, which makes me confident that when we've got something that our campaigners can use and we can iterate on, we can spend some effort at that point to engineer this into something that's well-architected, but I don't think we quite know what our full use case is going to be at this point. Trying to architect it out fully right now is (in my opinion) just going to chase us back into the widget-type discussion where we're churning on topics that might not end up being that important.

I'm serious about this, by the way: I'm OK pushing this decision on architecture for front end stuff and letting it live messily for a while (and I'm going to tag @paulaferris here so that he's conscious of the tradeoff we're making) in exchange for having a better conversation about architecture when we have more information about how exactly we want to use it.

I, for one, would like to know what the role of flute is in all this.

I'm genuinely not sure it should still exist (and if it doesn't, that's fine; it's had very little work go into it).

[NealJMD]

From @paulaferris on September 8, 2015 19:36

Let me know if it feels like I'm not responding to the right things here, in case I've missed or misunderstood the key thread of this discussion.

Omar, it's true right now that the fact we have to manage a Wordpress site where we have to manually load fresh campaigns onto the front page (where we profile campaigns) is a pain point for our team. That, along with the assessment it wasn't very hard, is the reason behind hosting static or semi-static front pages on Champaign, too.

Neal's initial post was useful for me in terms of setting stuff up in a four way matrix between low and high complexity, and low and high frequency of change.

Right now, I see two key areas where we'll want to be updating things on a daily or near-daily schedule:

• The list of campaigns that are featured on our front page/home page campaign highlights (will include both victories and ongoing campaigns). You can see that here or the banner thing (concept will change) on our current front page.
• Our press releases and press hits page. You can see that here

Both of these involve the need to profile things (campaign pages, releases, links to articles) that are or can be hosted elsewhere, rather than requiring the ability to customize the page heavily. I think we need to handle that at the outset without developer intervention.

In terms of other things like the "about" page, the rest of the front page, the jobs page, etc. -- I definitely don't think we need to have them editable by campaigners for Champaign v1.0. As long as they are fairly trivial for a developer to update (i.e. we don't need to wait for a weekly release or something like that) then I think it's fine for us to initially be reliant on developers to update these pages. To try and allows CMS-type functionality on these pages out of the gate doesn't feel very MVP (unless the right way of handling them is just as regular, custom HTML Champaign pages, and that's easy.) That said, I assume at some point this will want to change, but doesn't need to be out of gate.

Does that make sense to people? What am I missing or not being clear on (sure there's a few things!)

[NealJMD]

That's helpful Paul. Clarifying question - I assumed that the homepage campaign highlights would be selected automatically by the system from our most active recent campaigns (which would be simple to implement). Is that a feature you'd like, or do you prefer to be able to do it manually?

[NealJMD]

From @paulaferris on September 8, 2015 19:44

Ideally it would be done by:

• A checkbox on the page to feature/unfeature
• Some way of toggling that centrally, too (this is less thought, but I imagine it would be annoying to have to find, go in and uncheck one page and find, go in and check another to manually get the right combination to show -- althought that might be post-MVP).

I think it definitely can't be fully automatic as we run a lot of campaigns that we won't want to actually feature on our site :)

[NealJMD]

OK, that makes a lot of sense, and the feature/unfeature stuff is actually really straightforward.

It's tricky that the press hits page needs to be updated so often. We could build press hits into the db itself - publication, link, pull quote - or we could let them edit the page itself. Either way, it'll be some engineering overhead. Curious what others think about that.

[NealJMD]

From @EricBoersma on September 8, 2015 19:57

My gut reaction on press hits is to build out a simple interface for it, then hallway test it with whoever is most commonly working on updating press quotes (I actually don't know who normally does that). My guess would be that they'd be able to tell us really quickly whether what we built was going to be valid for their workflow.

[NealJMD]

From @edubsky on September 8, 2015 19:58

Ledys maintains the press clippings on our site at the moment afaik.

[NealJMD]

From @paulaferris on September 8, 2015 20:0

Eoin is correct, although Carys will have opinions on how we should present this information. I think the MVP is likely that you can insert a link (to a press hit, or a Google Docs hosted press release which is what we already use for sending out releases), title and short blurb and it formats it much as WordPress currently does.

[NealJMD]

well it seems like for now as I plan to move forward, the CMS functionality is something that isn't a short term priority, but that will probably be built out in the future. Thanks Paul for clarifying that.

That leaves tech team to decide about where the code and markup lives for those slim templates. I'm advocating gem and Omar is advocating Champaign repo. @EricBoersma and @Tuuleh thoughts?

[NealJMD]

From @EricBoersma on September 8, 2015 20:11

I'd advocate for code and markup inside Champaign for now. I think there's a very good chance that we'll want to refactor those into something down the road, but I don't think we know enough about what that's going to look like at the moment to reasonably say how that's going to work, and I don't want to commit to an organizational pattern that might not make sense when we can make that move at some point in the future.

[NealJMD]

This actually seems like an ideal application of a Rails engine. I'm still resistant to contaminating Champaign with SumOfUs specific stuff cause to anyone looking at the repo, it makes it seem like we're not actually committed to Champaign as an open source project. It seems unprofessional not to maintain a separation of concerns.

I'm also confused about the advantage of committing to an organizational pattern we know we're going to have to refactor instead of one that we're worried we might have to refactor. I feel like realizing the shortcomings of a certain approach (eg using an engine vs a gem) is something that we're going to realize only once we actually start doing it, not something that will be clarified by building like a regular one-repo rails site.

That said, I think I'm starting to repeat myself a bit, so if I you want me to just drop it then I will.

[NealJMD]

From @EricBoersma on September 8, 2015 20:54

This actually seems like an ideal application of a Rails engine.

Which part? I feel like we're talking about a number of different topics here. If you're talking specifically about the front page style stuff, that could be a Rails engine, or it could live in Flute (which, if that portion of the project is ever going to be relevant, it should be the place where we stick SOU-specific code, and it makes sense to me for it to live there, personally).

If we're talking about posting news hits, I think that can be a part of core Champaign, since that's a very standard workflow and it's something that a bunch of people are going to want to be able to do.

I'm also confused about the advantage of committing to an organizational pattern we know we're going to have to refactor instead of one that we're worried we might have to refactor.

I'm operating under the assumption that no matter the path we choose, we're going to want to heavily refactor. It could be that I'm over-reading what happened with widgets et. al, but given the history with this team (and to be clear, I'm entirely in favor of that history, I'm simply trying to make it work for us) I'm expecting that we're going to reach a point where we want to do some significant refactoring. My point is that instead of trying to figure out where that point is ahead of time, guessing incorrectly and doing it all over again some time later, we simply accept that we know we're not going to get it right the first time, start in, get some feedback and then start aggressively figuring out how to correctly engineer what we're on the path toward.

Taking an inventory, our system here is pretty simple in terms of architecture: We need to be able to feature a campaign, have a central place for making that campaign "un-featured", and post press releases. Those three things all make sense to live in Champaign core, in my mind.

We also have a need for a bunch of SOU specific code for the front end and displaying our main site, etc. Right now, none of that needs to be campaigner accessible. To me, it makes perfect sense for that to live in Flute, because that's part of what I see Flute as enabling, is a repository for all our SOU-specific stuff, including petition page display. Structurally, I don't see a lot of difference between a rails engine and simply using Flute, but if there's a strong case I could be persuaded otherwise.

That said, I think I'm starting to repeat myself a bit, so if I you want me to just drop it then I will.

This conversation has still been valuable for me, so I wouldn't encourage you to drop it.

[NealJMD]

From @paulaferris on September 8, 2015 21:9

well it seems like for now as I plan to move forward, the CMS functionality is something that isn't a short term priority, but that will probably be built out in the future. Thanks Paul for clarifying that.

That sounds right to me. Conversation has moved on to other things, but I'd just say -- also, if it just makes sense for the press page just to be a regular Champaign page with no form/etc and a liquid template, that would work for campaigners too I think.

[NealJMD]

that's part of what I see Flute as enabling, is a repository for all our SOU-specific stuff

I think flute was conceived of with having two purposes that shouldn't necessarily be packaged together. It was to be both a performance optimization to serve pages cached in the DB, and a concern separation to manage SOU-specific assets. The first is something that we'd want to make available to anyone developing with Champaign, and is the kind of speed optimization that doesn't go into an MVP. The second is something that we could handle a myriad of different ways, and I don't think that it should be packaged with (or blocked on the development of) our caching software.

My point is that instead of trying to figure out where that point is ahead of time, guessing incorrectly and doing it all over again some time later, we simply accept that we know we're not going to get it right the first time, start in, get some feedback and then start aggressively figuring out how to correctly engineer what we're on the path toward.

I guess that I see "guessing incorrectly" as the "start in, get some feedback" phase, which is why I feel bummed that you guys didn't want to start in on a suggested approach, cause I worry we won't actually get feedback about separation by leaving it in the repo. But now that I'm excited about the idea of reformulating Champaign as an engine as suggested in #16, I'm ok to just let it slide for now because that's a bigger refactor than I'm ready to take on right now.

[NealJMD]

Closing because a decision has been reached.

[Tuuleh]

General research on architecture

This is again relevant now that there's been a lot of talk about starting to split our work into smaller services. There's some anxiety around that which I can understand - having more services also means maintaining more services, deploying more services and service orchestration. Any kind of a split to smaller services warrants a good look at architecture, which is why I read up a little about different approaches and thought I'd do a write-up on what I found out that could be relevant for us, with the intention to elicit some conversation.

My main material for the day were Martin Fowler and Sunit Parekh - http://martinfowler.com/bliki/EditingPublishingSeparation.html and http://martinfowler.com/articles/two-stack-cms/ - and their thoughts of so-called two-stack CMSs - content management systems that have been split to two stacks - an editor and a publisher.

• When it comes to CMSs, it's a good idea to separate the stack used to edit from the stack used to publish content.
• The editor and the publisher stack have totally different needs: the editor is used by a handful of people, whereas the publisher is only used to distribute the content to a large number of viewers with read access.
• The publisher stack needs much better scalability and availability, where the editor side is usually quite happy as a single node application
• Between the publisher and the subscriber, the same page can also be stored in different formats. E.g. page is edited in one format and published in another - such as editing in markdown and publishing in html. In this case the editing module should store the markdown form while the publishing module stores html. The publishing module can also do some page composition work on the stored copy.
• By Fowler's split, we'd have two main stacks - the internal software, or the editor; and the member-facing stack, the publisher. The campaigner-facing CMS would be a pretty light-weight service in the editor stack. It'd be used to construct the content for the member-facing page on the publisher. The content would be sent over to the publisher, which would construct and create the page on the Member-facing website.
  • Publishing is controlled by a single transfer operation that copies content from the editor to the publisher. This transfer operation is the only writer to the publisher system.
  • All of the member-facing assets would be a part of the publisher stack - this would be a great way to extract SumOfUs.org out of the CMS itself.
  • In our case, the editor and the publisher can't be totally agnostic of each other in page creation - they both rely on the notion of templates and plugins. Champaign/the editor would need to know which plugins belong to different templates, because it'd need to be able to construct the forms for the objects to be sent to the publisher to be used in page creation
  • organization-specific static assets would live on the publisher stack, the editor stack could be totally org-agnostic
  • Fowler suggests a 'staging publisher' as the preview stage (not to be confused by a staging server for the developers). Before an article is published to the general audience, it can be published in a preview system, where it gets approved and once it's approved, it can be published in the live publisher. The preview system is basically a clone of the live system. When the article is finally approved, the data transfer is simply done to a different endpoint.
  • The split between the editor and the publisher is not a split between a back-end and a front-end. It's a split between the internal software and the member-facing site. Features and complexity can be added to both sides of the system independent of one another, and the only way they should communicate is through that one request from the editor to create a page on the publisher.
  • for a more useful graph of the architecture for the member-facing publisher stack: http://martinfowler.com/articles/two-stack-cms/#cds, or for my version of one way that Champaign would look like as a two-stack CMS, look at https://www.gliffy.com/go/html5/10313871 (note that some of the assets could be combined - e.g. single RDS, AK action worker and AK page worker combined into one, etc). I'm new to Gliffy and don't know if you're able to view that page without creating accounts so I'm attaching a screenshot below.

Finally, "When content volume is low and only few trained people are responsible for content creation and maintenance, any added complexity of a two stack CMS isn't worthwhile. Such complexity makes it harder for content creators to learn how to use it and adds significant effort in system maintenance". -> Fowler's two-stack CMS could be a way for us to decouple Champaign from SumOfUs.org (since sou.org would only live in the publisher), but it might be overkill.

[osahyoun]

I've moved Tuuli's post to https://github.com/SumOfUs/sumofus/issues/19