Discuss Knack scalability, sustainability, and ease of use

[amenity]

Objective

As DTS begins building bigger Knack applications, let's take a moment to discuss the implications in terms of scalability, resiliency, and ease of delivery.

On the horizon:

• Extending Data Tracker app (Signs Migration)
• Extending Finance app (Warehouse Inventory)
• Comprehensive mobility project tracking app (MoPeD)

Needs:

• The ability to version and roll back specific Knack configurations for an app — can currently export as JSON, but still have to manually create new application with schema as a reference
• The ability to export and import all an app's data — can currently export using API, but can't import back into another instance of the app
• Basic dev-ops workflow — cannot currently build and test in a safe environment (e.g. dev server) and then promote an application to production without taking down production — only option is copy apps in the builder, modify them, and then point the production URL to the new app
• The ability to leverage modular architectural approach where specific apps serve specific use cases but data flows between them — right now individual apps can only be extended by growing, so we are in danger of creating too-big-to-fail apps for business-critical applications
• Low-stress app-building — right now, you can break form, record, and database rules with the click of a button and there's no system log to show you what you did

Potential solutions:

• Knack builder release — may include features that address the above issues
• Build a deployment tool or buy one from a third-party
• Graduate some of our apps to a new platform
• ...?

Participants

Diana, John, Mateo, Amenity

---

• [x] Schedule meeting
• [x] Meet
• [x] Attach meeting notes to this issue
• [x] Create resulting issues

[johnclary]

My notes:

Future Solutions to Consider

• CMS’s: Drupal, Wagtail
• Other no code apps that scale better
• 3rd Party Knack Extensions
  • Some kind of data sync
  • Builder API hacking to promote app changes

Things We Can Do Today

• Rigorous QA / Testing process
• Break apart existing monoliths
• “Fake” integration by linking to other apps

Feature Wishlist (in order of priority)

1. Version control
2. Shared tables across apps
3. Developer documentation within the app builder. E.g., business rules
4. Usage Analytics and Logging

[amenity]

(A few additional points from my notes)

Summary While there are risks associated with giant apps, lack of version control, and kinda hack-y deployment processes, they are ultimately low: These are not critical resident-facing apps and it is acceptable for them to be offline when we are extending them. Were something to go very wrong, Knack takes daily snapshots of the app and database and can restore them for us.

For the time being, the benefits — speed of delivery and cost — far outweigh the risks. We are open to use a more scalable platform if the need arises, but nothing is urgent enough to warrant a full-blown evaluation of alternative tools.

Feature wishlist

• Ability to pull data out of Knack dynamically from external non-Knack APIs - e.g. Vision Zero app using locations and work orders from Data Tracker. This would allow us to use data in non-Knack apps and in combination with data from other sources without having redundant data in each system. (likelihood of getting this feature is admittedly low since it's not really in the company's interest 🙃)
• Ability to promote code and sync data between servers/environments

App size

• We could define a limit for our apps' size...But app size would ideally be dictated by business needs rather than technical limitations.
• We have and will continue to have a lot of small, freestanding apps — HR things, formy things, etc.
• Things we could potentially pull out of Data Tracker (basically leaving Signals):
  • TCP
  • Performance Measures
  • Signs and markings

Knack object (post-meeting discussion with @mateoclarke)

• Devs can access Knack. object, e.g. Knack.user, Knack.scene, Knack.user, see these examples from our Mobile Signals Work Order app
  • How we're saving the Knack object to use throughout the app
  • Using one of the variables from the Knack object
• This object isn't documented in the API, so we should ask about it in #204
  • Is the structure versioned? Is there documentation of it? (If not, we shouldn't count on it being stable long-term.)

[amenity]

@mateoclarke @dianamartin - feel free to add anything we missed!

[mateoclarke]

@amenity on the wishlist...

Ability to pull data out of Knack dynamically via an API

We do have this ability with the Knack API. I think the thing that is not possible is related to shared tables really but going a step further. Pulling tables dynamically from the Knack API and including them in another Knack app. And/or including data other foreign (non-Knack) APIs/endpoints. At least, thats what I understood yesterday.

so maybe instead it says:

Ability to pull data into Knack dynamically from external (non-knack) APIs

[sergiogcx]

@amenity I was speaking with @mateoclarke about the possibility of reaching out to them and ask for webhooks, it would be amazing to have real-time integration to our databases, that would also help us simplify our pipelines (avoid cron jobs or any complex sync jobs)

I did find 3rd party integrations that do something like this, but nothing on knack's side: https://zapier.com/apps/knack/integrations/webhook https://zapier.com/apps/knack/integrations/postgresql

[amenity]

Thanks, @mateoclarke. If we can't get it from other Knack apps though, should the wish be "Ability to pull data into Knack dynamically from another Knack app or an external (non-Knack) API"?

Pulling data from one Knack app into another would theoretically have the same result as being able to share tables... but maybe the data-feed approach would be more likely to open the door to accessing the data from non-Knack apps?

[johnclary]

Yes, ditto on webhooks. I've been wanting this for so long I forgot about it. I've talked to knack about this before, and they did mention a webhook-like functionality coming in the next major release. They described it as being able to create HTTP-request triggers attached to form submissions.