RFC: Where can we make big gains in productivity / speed of development

[JonForest]

It feels like a bit of an elephant in the room at the moment that our projects are currently taking longer/costing more than our old PHP stack, if I understand correctly.

I have no experience of that time, so it's difficult for me to identify what things have changed such that things take longer. However it's pretty gutting that a tech stack I really enjoy isn't delivering for us.

I've been going through old PRs of mine from My Worksites and Leapfrog trying to find common items that felt like they took too long, and if there are any lessons that can be learned from this. Here's what I've got:

• A lot of time was 'wasted' trying to get the relationships on Ember models updated after returning json-api from a custom end-point. Seeing lots of things like this: https://github.com/mediasuitenz/tmp-for-all/blob/75d6393989f75a96e523b598c10ad7e4c5291aed/client/app/models/worksite-draft.js#L24 and https://github.com/mediasuitenz/tmp-for-all/blob/develop/client/app/models/tmp-submission.js#L92 Both are ugly hacks, and there feels like there must be a better way - I just don't know what it is. Solution feels like a combination of using fewer custom end-points and cataloging a list of known hacks that make it work
• Acceptance tests, and tests in general, took me quite a long time to write and get working. Right now, even though I'm super happy we're using tests, it feels like a cumulative loss on our time frame (time spent writing > (time saved in fixing bugs + increased refactor speed)).
  Generally API tests seem easier. Learning lunch on techniques for writing Ember unit, integration and acceptance tests. Page object overview?
• I feel like we're still learning lots of things about our stack. Small things like how you need to manually increment PostgreSQL's auto-increment when manually inserting PK fields, how the PostgreSQL connector fails to handle undefined properly and therefore doesn't send that argument into your query, etc.
  If we come across something frustrating and fix it, maybe post to the #Ember or #Loopback channels on Slack with a 'here's what I found, and here's what I did'?
• Permissions and Roles, in both Ember and Loopback, have taken up a lot of time for something that must be common in many applications. We have different implementations across Leapfrog and My Worksites, unsure about others. It doesn't feel to me like we've yet got a re-usable, Media Suite canonical implementation of this yet. What blockers do we have to creating a reference implementation we can re-use, and extend?

(Note: not all things that take a long time took too long; some things are just hard or require some serious thought. Hopefully though, there are things to learn that we can apply to future projects) If we can collectively identify types of activity that feel like they really hurting us from a productivity point of view, then maybe we can generate some reference implementations, best practices or alternatives that allows us move more quickly.

[Resonance1584]

For me this comes down to two separate issues:

Why Node/JS? Pros:

• Everyone knows javascript
• Learning curve lets you have a lot of power on day 0
• Same language / libraries front and backend? (Not true anymore as we compile frontend)

Cons:

• Insane language for business (dates, timezones, money)
• Toxic ecosystem (npm) libraries are full of bugs, get oversold and then dropped fast for newer shinier things, there are no solid/consistent frameworks (hence the 'there's no good alternative to loopback' issue)
• Unstable - LTS is very short and best practices are constantly changing. Callbacks->Promises->Async/Await? Streams API? Error handling?

Why SPAs? Pros:

• You can go offline e.g. Marlborough Cruise Guide
• For small apps it's a smoother more fluid ux experience
• Solves the 'getting state into the js runtime' issue

Cons:

• 99% of the internet is HTML forms, but we have to literally reinvent the wheel to use things like radio buttons
• Duplication of code everywhere to make the backend and frontend behave the same - twice the work in some cases. Any changes need to be made in both sides (shotgun surgery)
• Horrendously more complicated to implement permissions correctly on an API as it needs to be correct for all cases and has no context on the request being made
• Huge overhead of requests to load related data, check permissions etc makes the app much slower than pre-rendered HTML
• We lose lots of nice things that 'just work' in HTML like #anchor tags

[pnw]

My question in response to this: Some of the things you identified as having taken time to figure out, @JonForest , are they still taking time or are they things that have been "solved"?

Example:

A lot of time was 'wasted' trying to get the relationships on Ember models updated after returning json-api from a custom end-point.

Initially, there was definitely an effort to figure out how to do something like that, especially since there is little documentation or canon around how to make ember-data work with custom http requests and RPCs. However, we have a system in place now, and I don't see hooking up a custom remote method to the client anything more than the most minimal time investment.

My suggestion is that perhaps we exclude things that we've identified as "solved problems" from this discussion so that we can focus on the things that continue to consume a disproportionate amount of resources.

Examples:

• Permissions (myworksites)
• State management (e.g. form cleanup on cancels; ember-data is basically a cache layer)
• client-side testing?

[digitalsadhu]

I think we need to be a bit careful about our thinking. What are we comparing against?

Multidimensional change

• language
• framework(s)
• team size and composition
• paradigm (SPA vs server rendered)
• team member skill level
• testing vs not testing
• process (how do we design, meetings, communication, code review, testing etc)
• maintainability

How do we measure slow vs fast? I think when most people think, "things are slow but they used to be fast" what we completely fail to acknowledge is that we are absolutely NOT isolating the differences and their effects between our "used to be fast" and our "now its slow" comparison points.

[digitalsadhu]

@JonForest had a brief discussion with @ewen and @anotheredward but we are leaving it here and keen to pick it up with you and others as part of ongoing discussion.

[tfentonz]

:+1: for learning lunch on testing Ember and testing in our apps. This came up in peer review as something that would be generally useful.

[pnw]

Some other areas of investigation:

• Data Seeding
• Migrations
• Forms, form validation

[JonForest]

Okay, I reckon we need to form a working group to discuss this further and make a plan of how to take it forward. I'll set something up for next week. @digitalsadhu @ewen @anotheredward @Resonance1584 @pnw - you all keen to be involved? I think six of us is quite a large group, but this is a subject that clearly there are a lot of areas to consider. Happy to curate/organise this.

[anotheredward]

I feel like we've moved forward on this issue and are trying out options, closing.