Integration Lifecycle Handling

[rhuss]

At the moment, when integrations are created they have completely disconnected from the Syndesis DB afterwards. There is no way yet to update a running integration nor see the state of the currently running integration.

Therefore it is suggested to introduce a versioning for integration and rethink the state model with "active" being a valid state.

The following points should be considered:

• Multiple versions of an Integration must be able to exist.
• Exactly one of these integrations can be active.
• Each deployment of an Integration creates a new version.
• Users can switch integration versions from the UI
• Versioned Integrations can be inspected from a history page.
• Old integrations can't be changed but can be used as a blueprint for new integrations.
• As soon as an integration is changed, a new version of an integration is created. All changes go to this new integration until it is deployed by the user. After this, its gets a fixed version and cannot be changed anymore.
• The user must always be able to see and examine the currently active integration along with its version number.

@kcbabo @chirino I really would like to discuss this lifecycle handling of integrations a bit in more detail. I'm sure that our simplistic handling of integration versions is not sufficient (even not for PR1)

[jimmidyson]

I wonder how much we can piggy back off of deployment revisions here? I guess we need to version the integration in the DB though, not keen on doing that tbh but can't see a way round it.

Just to get things moving with this, how about a model something like:

type Integration {
 Spec IntegrationSpec // The user selected integration spec (reconciled against Status.ActiveSpec to look for changes)
 SpecVersions IntegrationSpec[] // Array of versions, ordered by age, oldest first. Integrations can be deleted from anywhere in this array, unless it is the Status.ActiveSpec

 Status IntegrationStatus // As we talked about 
}

// IntegrationSpec should be immutable, editing an IntegrationSpec should create append a new revision to Integration.SpecVersions
type IntegrationSpec {
 ... Current spec fields...
}

type IntegrationStatus {
 ActiveSpec IntegrationSpec // The currently active integration spec

 ...Other status fields...
}

As IntegrationSpec is immutable, we could also hash the IntegrationSpec and use the hash in IntegrationStatus.ActiveSpec and store the hash in IntegrationSpec.Hash potentially?

[rhuss]

So iiur then Integration.Spec is the desired spec and Integration.Status.ActiveSpec the current spec.

Wouldn't it make sense to reconcile on a whole status as it also there is a target status (like 'running', 'stopped', '...') which should be reconciled ? Or is this part of the spec ?

Would it make sense to include a version number within the IntegrationSpec so that the comparison between targetState and currentState is easier ? Also to see what is older, what is younger ? (not so easy with a hash).

Maybe we just call it Integration.TargetSpec to make it clearer and more symmetric to ActiveSpec.

---

Whatever, we should start with some design proposal which is important as the state model and the versioning is really at the heart of Syndesis. I'd pretty happy to copycat K8s Deployment model wrt to status / versioning.

[jludvice]

Did we consider various race condition situations ?

• developer doing git push with his changes
• syndesis doing
  • save integration > push changes - we may expect conflicts here
  • opening existing integration and loading changes can we get conflicts here?
  • pulling changes into integration pod probably no conflicts here, right?
• two citizen developers editing the same integration at the same time

Assuming expert developer makes some changes directly to git, how will he trigger redeploy to verify his changes?

[chirino]

@rhuss you could use the KeyGenerator to generate your version ids. The keys will be unique and always in increasing order (it has a timestamp component to it).

[rhuss]

@chirino although this might help it would be awesome to go exactly to the last version by decrementing the version number by 1 (or any calculation without looking up all versions). The version needs to be unique for this specific integration only, so a plain counter is completely sufficient.

[rhuss]

@jludvice these are all good points. I wonder whether we should avoid merging at all and use orthogonal branches for the integration (so each deployment of an integration creates a new branch corresponding to the version number). In that case, we don't have issues with merging but still can get back and forth by switching branches.

The interaction with the git repository happens only when an integration gets deployed/redeployed, which is an atomic operation (i.e. should be locked somehow so that only one deployment at a time is allowed).

For the expert user to work directly with git I would say this is not a valid use case for now, but theoretically, he could change data on a branch and hence modifying a current version. For this flow to work (i.e. from git to syndesis db) a lot more things would have to be considered to get changes into the DB.

For now we should only consider one direction: From Syndesis -> Git.

[jludvice]

@rhuss I like the idea of decrementing by one to get previous version

Regarding "Exactly one of these integrations can be active." It's common to have several envs like dev >> test >> stage >> production right? Should we have concept of envs in ipaas? Could we eventually implement that with "hard lock" on one instance? Would it be easier to use Copy+Paste to create devel version of integration?

As a citizen user I probably won't have a glimpse of dev >> stage>> production envs, but having a way of "playing harmlessly" until it's ready is easy to understand. I could make dozens of attempts until I do data mapping and stuff right and I might need do replace certain connections with their test/mock counterparts.

I understand it's a big thing to implement. Can we manage to implement this issue in a way that won't close a door for requirements like this in the future?

[dongniwang]

Is there a UX design need down the road for this?

[rhuss]

@dongniwang @jludvice @kcbabo I think we need some initial discussion what exactly of this kind of lifecycle management affects the UI and also which parts of the UI. So at the moment the answer is probably "yes, there is, but we still have to check which".

[jludvice]

@rhuss regarding the integration state model,

• suspend/resume should be swapped
• why do we only allow deleting of drafts? does it mean I can't delete some old revision of integration?
• what if I create integration which crashes / goes to error state and I have no idea how to fix it? How do I delete it to start fresh?

anyway +1 for nice graph ;)

[rhuss]

• Indeed suspend / resume is wrongly placed. Thx !
• That's a good question whether we want allow undeployed integrations to be deleted. Depends of course on the business case, so it's up to @kcbabo to decide (one would loose auditing history, but I have no strong feelings here :).
• For the error case the 'fix' can be of different nature:
  • Fixed automatically because it was only a temporary error (e.g. in the backend)
  • Some action required. This is individual to every error kind and there should be an indication e.g. in the error message how it can be fixed.
  • Disable the integration by moving it to "undeployed". Also I wonder, whether an Inactive integration can move to Error, but that might happen when one tries t "Resume" ? Same for "Undeployed" when doing a "Reactivate" ? wdyt ?

[rhuss]

Graph is btw created via plantuml

[rhuss]

Current status: Design Proposal parts for the backend are finished, the UI is not yet perfectly shaped out. Next steps: Change backend to reflect the state model, create UX design for the integration overview.

[sjcox-rh]

Updates to the integration lifecycle designs can be found here:

https://github.com/syndesisio/syndesis-ux/pull/14/

FYI: @chirino @jimmidyson @jludvice @iocanel

[jimmidyson]

Going to move this from sprint 17. We now have history without sweeping API changes thanks to syndesisio/syndesis-rest#593 and we can relook at this in future sprints.

[dsimansk]

@rhuss after a while of using resource constrained project on Online cluster. I think we need a way to gather exceeding events from OpenShift and report them back to user in a simple && meaningful way. E.g. permanent warning banner similar to OS console, that quota is exceeded etc.

This seems like a complex task(backend,ui,uxd) partially related to lifecycle, but should I open new issue for it? Wdyt?

[rhuss]

@dsimansk Yes, I think this a good idea to present any resource constraints like this, possibly with a link to further details and how to get around it.

if you could open an issue in syndesis-project that would be awesome. I think we already have an issue which is about an UI integration of the limit restrictions, maybe you can look to this, too ?

[kcbabo]

@sjcox-rh @dongniwang @amysueg Note the request for a notification when exceeding resource limits.

[amysueg]

@kcbabo @dongniwang @sjcox-rh @dsimansk @rhuss See my comments on #156