jasperweyne / helpless-kiwi

Manage members, create and manage activities, send mails and more
Apache License 2.0
11 stars 7 forks source link

Registration integrations #266

Open jasperweyne opened 2 years ago

jasperweyne commented 2 years ago

In its current form, Kiwi is a tightly integrated platform. All functionality necessary is present within the system itself and the only connections with external code is the OpenID Connect integration (and maybe the Swiftmailer integration, if you count that as well). This keeps the platform relatively simple, with very few opaque moving parts. However, this runs the risk of turning Kiwi into a monolithic structure, where all functionality is integrated.

To address this, a GraphQL API has been added last January in PR #226. This adds a lot of additional ways to integrate Kiwi within different solutions. Examples of (planned) integrated systems are sassy-fox, spicy-hamster and funky-puffin.

The problem, however, is although this makes it possible to integrate Kiwi into other systems, it's currently impossible to integrate other systems into Kiwi. For example, an administrator might want to gather additional personal details that are necessary for users to participate in an activity (the users age when alcohol is involved, for example). Or an administrator might want to make sure a user has payed for their ticket, before their registration is confirmed. Basically, an administrator might want something to happen at registration time.

One might suggest that this can be made possible through the API. For example, an external service could poll the server, or a subscription could be added to the API, which would require the integration of a message broker, where mercure would probably be easiest and could be relatively cheap as well when running it on an AWS EC2 Spot instance. If the usage would stay under 750hrs a month, this would even be free.

However, a pub/sub or polling API doesn't suffice when Kiwi itself needs to check whether an external integration has succeeded. A logical example for this could be the check whether the user has payed. Moreover, it's also not possible to integrate an external service into the registration flow. Staying with the example of a payment service, an administrator might want to send the user to a payment webpage when they click on the registration button. Lastly, when the external service isn't able to either subscribe or poll (for example, server side PHP code), this solution falls short as well.

Basically any use case where Kiwi needs to check something at registration time, currently requires tight integration into the Kiwi codebase. This is not the only shortcoming; something that should happen at the creation of an account could pose a problem as well, for example. However, I expect that the problem would mostly surround registrations.

Given the limited number of parties that are currently actively using Kiwi, one may decide that that's okay. All other integrations are possible or can be made possible through the API. However, I feel this limits the potential of the system in the long term. To solve this issue, I see two viable options:

  1. The first option would be a webhook-like situation. Here, an administrator would have to configure a list of URLs that are queried to see whether the registration conditions are met. Additionally, any user redirects would have to be configured here as well.
  2. The second option would be a plugin-like situation. Here, an administrator would upload any custom integration code. The plugin could offer its own way to enable or disable the behavior for a specific activity, or this could be done from Kiwi itself.

The advantage of the webhook solution is its relative simplicity. No extra files would have to be hosted on the filesystem, as all integration options could be stored in the database. Therefore, it would fit relatively easy in the current structure of Kiwi. However, its downside could be the configuration of data that's send to the webhook and how it would check whether the webhook response is valid. Moreover, if any additional integration options prove to be necessary, this solution wouldn't be very flexible either.

The advantage of the plugin solution would be its flexibility. If more integration options apart from registrations are necessary (let's say a theming system), these could be added with relative ease. With regards to registrations, any integration could be tailored to the specific system. However, this comes at the cost of requiring more upfront work to enable a plugin system. The updater system would need to respect these and the plugin would need to hook into events that would need to be added to Kiwi. Moreover, if per-activity integration isn't provided by Kiwi, the plugin would need to hook into the theming system, so an extensive number of hooks would need to be added here as well.

Given the fact that our development resources are limited, I'm inclined to say that a webhook system would be the way to go. However, I'd like to hear more thoughts on what the right direction would be!

mkrugr commented 2 years ago

I concur that webhooks would be the best solution for Kiwi. Mostly because the modularity of the integration. A rudimentary webhook integration can be implemented relatively fast and only on the places that we need it.

An example of such a webhook is a simple check if the user is allowed to register at all. This webhook could trigger at the rendering of the activity page (which contains the registration button). Or even at the submit of the registration after pressing the button, depending on the event to which we hook. A suitable response could be a "continue with registration" message or a "disallow the registration" message with a suitable explaining payload.

This would only require a interrupt within the controller function, that continue or fail after the webhook response.

Webhooks with a redirect will obviously be a lot more complicated, but equally doable.

More generally, such events could (potentially) be generated at every step of the controller functions or any other line of code within Kiwi. And the response of the webhook can handled anyway we want. Thus we can start with a small webhook.

The disadvantage of such a system is that we (Kiwi devs) still need to

  1. Determine which functions or actions within Kiwi will be an event, label them suitably and trigger them with relevant and secure payloads.
  2. Write code that handled interruption or redirects for the webhook.
  3. Write custom integration within Kiwi itself on what Kiwi should do with the response from the webhook.

This necessarily limits us to a select number events (preferably the most useful and obvious), as we cannot check triggers on every line of code. And complicated and very custom response-handling code will be sunk in the kiwi codebase, which would make the project harder to understand and work on. So this code will have to be light-weight too.

For truly heavy-duty integrations with non-obvious triggers and response-handling we should reconsider plug-ins. But at the moment, within our limited users, I do not think there is need for such heavy-duty intergrations.

The pub/sub and polling API is a whole other issue in it of itself, as this concerns the subscription of data to 3rd parties without any (active) Kiwi intergration. The webhooks (if implemented) will get triggered by an event with (probably) a simple POST request, and thus do not have need for complex subscription machinery.

DavidckPixel commented 2 years ago

I personally believe that a plugin based system would be the best option for Kiwi. This will allow system administraters the best flexibility within their own kiwi envoirement.

A downside is that it would be more work/ more difficult to implement then a webhook based system. However I personally believe this is worth the extra effort for the additional options that it will allow:

Kiwi (base) would remain a simple and managable system, and that active development and feature additions might at some point stop. Instead we could have a kiwi Base system, and add new features through plugin development. This can be especially usefull while multiple organisations are developing/using kiwi. Allowing webmasters to easily activate and disable certain features.

A-Daneel commented 2 years ago

I have some spare time, and a lot of thoughts. So I'll leave an initial message.

Overall I think a plugin based system would be the nicest/best fit for the project. It would leave kiwi easier to configure/set up for the less technical people. And if kiwi grows in usage that will be a large portion of the users. Having more (technical) configuration could also lead to someone setting it up now, and handing maintenance over to a less tech savy person. Thereby forcing them to maintain something they don't fully grasp. Besides this a plugin based system would be easier to for third parties to contribute by building their own.

An added benefit would be that a plugin based system would be able to handle more use cases then the web-hooks can. As @jasperweyne has stated in this issue. This is where I think the crux of our choice now lays. Not what is better, but what is better at this moment.

Judging from this issue I only see use cases build around registrations. Most (if not all) of the features I could think off could be handled with web-hooks. As such I don't think the choice should be plugins vs web-hooks, I think it's just a matter of time. How much time are we willing/able to spend at this point in developing and supporting a plugin system?

I do feel that that if we start supporting plugins we have a bigger obligation to support it and them, for web-hooks it's more of a "we have an API endpoint and some hooks, this is how to use/tie into them. good luck!" kind of deal.

Given these musings and the face that we've spend the majority of our last two sprints on refactoring and testing our current code base, I see the benefit of setting up a web-hook based integration for the registration based events.

My fear is that we start building web-hook based integrations now, and then since that's what we're used to keep using them. Therefore my suggestions would be the following:

We start with a hook based system for the cases having to do with registrations. This is the easier/faster option to get up and running. And changes kiwi's code base the least. But we try to keep it to just these events.

Simultaneously we would also spend some time better thinking out a plugin system. Since I for one do still think in the long run this would be the best option. I also agree that at this point in the project it might not be the moment to attribute our resources to it.

Spending some time now on, at least thinking about and writing some ideas down, will also leave us less likely to lock ourselves into using hooks for everything.

I'll try to leave a bit more coherent/elaborate comment if I find the time. And would suggest this is something we also discuss during the next meeting.