Support IIA version 6.2.0 and 7.0.0 at the same time

[JoelBlancoCloserIdeas]

Most providers will deploy to Production their changes to the IIA API and will publish the version 7.0.0 in 6 or 7 months (at the end of the year). We will have to support both versions (6.2.0 and 7.0.0) when we publish the version 7.0.0? We would find it easier to support just the latest version.

[janinamincer-daszkiewicz]

As I see it now we will switch to 7.0.0. If somebody is not be ready, will not be able to communicate with 7.0.0. nodes.

[janinamincer-daszkiewicz]

We would find it easier to support just the latest version.

Me too, however from the discussion going on in https://github.com/erasmus-without-paper/ewp-specs-api-iias/issues/109 it seems that we cannot agree on how to handle old versions of IIAs with the new versions of APIs. Let's try to approach the problem from a different angle. For easier reference let's call this 'v6/v7 proposal'.

Assumptions are:

1. We introduce v7 and this is the last version for the current Erasmus+ programme.

2. We keep v6 until the end of the current Erasmus+ programme (which is not so far ahead).

3. Every system remembers a version number of the stored IIA (v6 or v7).

4. APIs in version 6 handle IIAs in v6, and APIs in version 7 handle IIAs in v7.

5. Versions v6 and v7 co-exist in the network.

6. If our partner does not implement v7 yet we:

   • wait until the partner is ready,
   • continue using v6 with that partner.

7. All IIAs mutually approved in v6 stay in v6. If no modifications are needed they will stay in v6 until they expire.

8. When the system starts supporting IIAs in version v7:

• All IIAs which are not mutually approved have to be upgraded to v7 and handled by API in v7.
• New IIAs can only be created as v7.
• If modifications are needed to IIA in v6 it has to be upgraded to v7 and processed in v7. If partners do not agree on these modifications they will revert back to old v6. During the negotiations only v7 is available in the network.
• There is no need to call IIA get for IIAs stored as v6, as they will never change in v6.
• If we need to update fields that are outside the approval and we want to share these changes with the partner, we have to upgrade to v7.

11. When we get IIA CNR or IIA Approval CNR we proceed with the highest version of IIA API our partner supports.

12. Version 7 does not differ from version 6 in respect to IIA Approval, hash calculation, hash handling. We keep the technical changes to a minimum.

13. Every local system can handle fields which were optional and became mandatory in its own way (e.g. keep them empty when in v6 but setting some values when upgrading to v7).

I am aware that this proposal is not yet fully elaborated. I want to know if such approach can help us to solve the issue of mandatory fields introduced in v7 and avoidance of re-approval of mutually approved IIAs.

[janinamincer-daszkiewicz]

After internal discussion I withdraw this proposal.

[rolandfrancois]

@janinamincer-daszkiewicz After an internal meeting in our service, we thought of a solution close from the one that your proposed here. We would like to know the reason why this solution got withdrawn. We know that technically it would be a lot of work but it would avoid the re-approval of mutually approved IIAs due to XLS versions changing.

[janinamincer-daszkiewicz]

We would like to know the reason why this solution got withdrawn.

Because when consulting various providers we were told that supporting both versions would be a lot of extra work.

[Deresium]

We would like to know the reason why this solution got withdrawn.

Becasue when consulting various providers we were told that supporting both versions would be a lot of extra work.

Hi ! Working with rolandfrancois here.

So what is the plan ? To find a "lazy" solution ? By chosing a solution based on a quantity of work, we will report our technical debt in the future and it will make the situation way worse. We are waiting for real technical issues about this solution or considering it as a valid possible one.

[janinamincer-daszkiewicz]

I proposed it so I see its merits :) I do not mind adding it as one of the options if DG EAC accepts.

[janinamincer-daszkiewicz]

P.S. I remember one more argument. In this solution to get ALL IIAs via index you have to call IIA get in v6, IIA get in v7 and merge the results.

[Deresium]

If we want flexibility and let every partner switch from a version to another at different times, it seems logical that we have to work with the 2 API versions during this transition. There are a lot of examples of companies that provide services throught APIs and keep the previous versions for retro-compatibility until they decide they can get rid of them. Thank you for the reconsideration of this solution !

[demilatof]

The current requirements allow to keep both of the version, because what is mandatory in v7 can already be exchanged with v6, so now we are lucky. But if in a future v8 IIA we will have something new (e.g. BIP mobility), we cannot impose a provider to not include the BIP in an IIA just because its partner hasn'y implemented the new version yet. I add that if we allow both of the versions, a provider can choose to not implement the new API version

[kkaraogl]

I gathered that, after thorough testing, we would agree on a date that everyone would switch to the new version. This is why it was communicated so early on. Is there a technical reason to keep them both?

Nevertheless, this issue is tightly coupled with #109. If #109 concludes with some straightforward voting options, everyone will be more confident to decide on this one as well.

[janinamincer-daszkiewicz]

I add that if we allow both of the versions, a provider can choose to not implement the new API version

Not quite, because there will be some deadline for all imposed by DG EAC. Also the customers of this provider will be pushing, I presume.

Is there a technical reason to keep them both?

Yes, the solution proposed here is the only one which allows to smothly handle an upgrade from v6 to v7 without any extra technical mechanisms like extra hash or XSLT.

[demilatof]

Is there a technical reason to keep them both?\n\nYes, the solution proposed here is the only one which allows to smothly handle an upgrade from v6 to v7 without any extra technical mechanisms like extra hash or XSLT

No, this is not right, we only postpone the problem. As soon as we will have only version 7 the problem will arise again

[janinamincer-daszkiewicz]

As soon as we will have only version 7 the problem will arise again

I do not follow, what problem will arise?

[georgschermann]

we had versions 3-6 in parallel for some time, the index endpoints delivered the same set of iia-ids and all iias could be requested in any version, so if hosting multiple versions is the plan, than this may need to get specified in detail.

Also returning an 5xx error with error message on the v5 endpoint, that this iia moved to a new endpoint could also be a feasible way, since error handling should be able to cover this. The iia would not be fetchable until the partner moves to the new version, but that should have no effect on the iia.

[demilatof]

@Deresium

If we want flexibility and let every partner switch from a version to another at different times, it seems logical that we have to work with the 2 API versions during this transition.

I think too we need 2 API versions to not compromise any IIA exchange. Otherwise, we have to choose if we want to exchange IIAs with partners that use v6 or v7.

But if we decide to support two versions, we need extra specifications about what we can or we cannot do. E.g.: if the two partners have both of the versions, they most probably should have to exchange IIA only with the higher version. But then, this statement then requires to be specified what can do a partner that receives a call to the lower API version.

[milsorm]

Our plan is easy:

• if couterparty supports 7.0.0, we will communicate according to 7.0.0
• if couterparty supports anything below (6.3.0, 6.2.0 etc.), we will use 6.3.0
• if couterparty supports both, we expect 7.0.0 way

I cannot imagine that all providers will switch at the end of March. We will do the most, but big showstopper is support of modifications and reverting in 6.3.0 (we are able to do snapshots now, but recognizing, if counterparty revert or make further changes, it is nightmare). So we expect both versions simultaneously.

[demilatof]

I agree with you again, but during the last infrastructure forum I explicitly asked if I could support both of the versions in my system and the answer was that or all providers must support two versions or none of them can and we all have to switch to v7 at the same time.

[fioravanti-unibo]

This afternoon the document 'Release note on IIA version 7.0.0' was shared and it is expressly stated that support for both versions is effectively prohibited.

Personally I also think like you that this is simply absurd.

Our current client implementation tries to handle major versions of the API so we can download agreements from all partners and I think we will do the same with version 7.

As for the server, we would like to expose both versions for as long as necessary.

Regarding the hypothesis of a partner who wants to go back to using an old version after having used 7 or who wants to alternate the use of the two versions, it seems to me quite unrealistic; In any case I think it's enough:

• as server, keep track of the version with which an agreement was last downloaded by the partner and return an error if a previous version is used

• as a client we can keep track of the latest version of the API we used to call the partner and refuse to download the agreements if it exposes an older version

[demilatof]

This afternoon the document 'Release note on IIA version 7.0.0' was shared and it is expressly stated that support for both versions is effectively prohibited.

That document is absurd, to be polite. It makes several assumptions that have no basis. The main points are:

• "After the end of March 2024, version 7.0.0 will be the only IIA API version supported by the EWP Network". (this involves that all HEIs have resources to develop and test the new version in less than six months)
• "We will not support versions 6 and 7 in parallel. (...) Furthermore, version 7.0.0 is not backward compatible." and "To support developers and allow for early testing, the EWP+ consortium will provide a backward compatible interim version 6.3.0." (if v.7 is not backward compatible, playing with 6.3.0 is useless)
• "Supporting versions 6 and 7 can lead to unwanted technical complications and requires more resources from all development teams" BUT "The EWP+ consortium will schedule testing sessions with developers. These sessions can include both early testing of version 6.3.0 and the mandatory testing of version 7.0.0 before nodes go in production". (to do these testing sessions and to develop v. 7 while we need to give support to v.6 we have to implement both of the versions at the same time, even if they say the opposite; where is the advantage of having only one version in PROD, if until the last day we need to have two versions in DEV?)
• "This means that in that week versions 6.2.0 and 7.0.0 can coexist in the production network." (I hope that they mean not for the same HEI, otherwise this is an evident contradiction)
• "Until then, providers will need to keep or have a valid backup of versions 6.2.0 and 6.3.0 as a fallback in case of problems" (It's clear that they want us to make a fork of our software, with all the consequent problems)
• "It is essential that all nodes in the network implement and test version 7.0.0. Nodes that are not ready in time will stay in the development network until they are ready to go in production" (this means that a provider that cannot implement v7 before the end of March will break all the mutually approved v6 IIAs damaging all the partners; and last but not least, currently the DEV environment is quite empty, with inconsistent data).

And these are only some first considerations.

[geostarling]

Hi, Sorry for reviving this old thread.

I (rep. our in-house provider) have a few follow-up questions regarding the transition from v6 to v7 of IIA spec:

1. It has been mentioned here that the IIA 7.0.0 Release Note document clearly states that:

After the end of March 2024 , version 7.0.0 will be the only IIA API version supported by the EWP Network.

How is this going to be enforced in practice? If our IIA API implementation won't support v7 by the end of March, will our endpoints be forcefully removed from from production registry? Or will our inability to communicate stem from simply them no longer exposing v6 version of their API? Some providers proposed here, that plan to keep the support of communication over v6 spec. for providers that did not implement v7 yet. I guess that is no longer possible after this announcement?

2. The Release Note further states that:

   All existing inter - institutional agreements concluded before version 7.0.0 remain valid after March 2024.

Does it mean that our IIA v7 endpoints should return both v6 IIAs that were created before v7 API deployment and new v7 IIAs created after?

3. In which situations (if any) should the IIA schema for particular v6 IIA be updated from v6 to v7? a. From spec I gather that mutually approved IIAs stay as they are. b. But how about partially approved (approved by only one of the partners) or unapproved IIAs (be neither side) but requiring no further modification before approval. c. The article 2.12. IIA major version change impact on CC hash in 2023-07-31-ChangesInFamilyIIA.pdf states that:

Whenever they need to be modified, the fields are adapted as part of version 7.0.0 and re - approved as a new version 7.0.0 IIA. So, the problem will gradually become smaller.

Okay, so on update IIA we should upgrade version to v7 and specify all mandatory (in v7 schema) fields? But what if partners copy of the agreement is still in the v6 format. What if partners system won't upgrade their paired copy of IIA from v6 schema to v7 in this case?

Thank you.

[demilatof]

How is this going to be enforced in practice? If our IIA API implementation won't support v7 by the end of March, will our endpoints be forcefully removed from from production registry? Or will our inability to communicate stem from simply them no longer exposing v6 version of their API? Some providers proposed here, that plan to keep the support of communication over v6 spec. for providers that did not implement v7 yet. I guess that is no longer possible after this announcement?

What I've understood is that if you're not ready by the end of march, your manifest will be excluded from the registry with all of your APIs, until you don't have a working IIA V7.

All existing inter - institutional agreements concluded before version 7.0.0 remain valid after March 2024.

Does it mean that our IIA v7 endpoints should return both v6 IIAs that were created before v7 API deployment and new v7 IIAs created after?

I don't think so, not exactly at least. Your IIA v7 endpoints could return even v6 IIAs approved before v7 but only using the v7 format.

3. In which situations (if any) should the IIA schema for particular v6 IIA be updated from v6 to v7?

Always, the v6 will not exist anymore after 31 march. If you want to exchange the old IIA, you have to use the new IIA schema. Your partner, thanks to the XSLT, can verify that your v7 IIA hash code (new algorithm) is still equal to the recalculated hash code of his snapshot of your previously approved v6 IIA.

   a. From spec I gather that mutually approved IIAs stay as they are.
   b. But how about partially approved (approved by only one of the partners) or unapproved IIAs (be neither side) but requiring no further modification before approval.

You should have to populate again the XML, according the new v7 format.

Okay, so on update IIA we should upgrade version to v7 and specify all mandatory (in v7 schema) fields? But what if partners copy of the agreement is still in the v6 format. What if partners system won't upgrade their paired copy of IIA from v6 schema to v7 in this case?

Please keep always in mind that v6 IIA will not exist anymore. Everyone must keep the snapshot of the mutually approved IIAs; before switching to v7 they have to add the new hash code to their saved IIA (I suppose they stored it in their DB). If they will not perform this task, they most probably will trigger a new round of approvals, even if it's useless.

This is what I understood, but please wait for @janinamincer-daszkiewicz o @mkurzydlowski opinion.

[janinamincer-daszkiewicz]

Thank you @demilatof for answering the questions from @geostarling. I would give the same answers.

[milsorm]

When we saw how the others deal with progress of 6.3.0 there is absolutely no trust that everybody will be ready by the end of March for 7.0.0. So there is several possible options:

• deadline will move forward and we will be blocked in other development (be ready with 7.0.0 means stuck with 6.x.x functions)
• mutual co-existence of both versions will be allowed (later than soon)
• half of network will stop cooperating and big trust issue happen

When I saw this analysis I decided to prepare for co-existence for both version and ask our clients to pay for it. It is the only safe options because:

• if everything will be better than expected, we will have 7.0.0 ready and can ignore simultaneous solution
• if mutual co-existence of both versions will be later allowed - we will be ready
• if deadline is moving forward we can still develop 6.x.x functionallity and test in parallel 7.0.0 functionallity

It is much safer for 3rd party providers to be ready with both versions and mutual co-existence.

And what co-existence means?

• if partner has version 7.x.x in registry, i will communicate with him in version 7.x.x
• if partner has version 6.x.x in registry, i will continue in old fashioned way
• once partner use 7.x.x I will assume he can support 7.x.x forevery (new structures)
• if any other parter offer both version, we will use 7.x.x

Transition for us will be seemless regardless Warszaw decision about deadline. And also we don't need gap week, we can test anytime 6.x.x and also 7.x.x

Estimated costs are several mandays. Not big deal. Safer progression.

I recommend the same for those who have a very small trust in success of this project (last time in Warszaw we can see how many providers are ready with 6.3.0 and I don't expect better result on Jan/Feb testing workshop with 7.0.0).

[demilatof]

I completely agree with you. I can only add that in my opinion there are even some issues that we need to solve before thinking to be ready for version 7. One of them is the approval; the approval is the core of the IIAs system, but whoever makes a serious analysis, he discovers that the current specification doesn't allow to be sure that the two parts of the IIAs are equal for both of the partners. And the risk is that my outgoing cooperation conditions overwrites your incoming rules. I think nobody wants this.

In my opinion, the consequence of this analysis is that only getting rid of the outgoing mobilities from the IIAs we can speed up and simplify the whole process, including the approval.