IIA version change impact on cooperation conditions hash

[mkurzydlowski]

As we already know IIA major version change impacts the cooperation conditions hash, as it changes the XML namespace that is part of the cooperation conditions XML element being hashed. So even if a IIA version change doesn't impact the fields in this element, the hash changes.

Let's discuss how this case should be handled and what impact this has.

Should the cooperation conditions hash change after version change or should it be stored and stable until IIA changes? If we choose to recalculate hashes then should we send CNRs? If the partner chooses to recalculate hashes, should we recalculate IIA approvals?

[demilatof]

I think that your question may involve three issues:

1. When there is a major version, should we all conform to it and in what amount of time? Or should we be able to use both of them? Until when? Because they can become three versions, four an so on...
2. If we all switch together to the new versions, there could not be any problem for the IIA even mutually approved; but if we don't?
3. As concern the mutually approved versions, I think that we could save the snapshot as a pure XML and return it if we'll have the possibility to ask for the last approved version and not only for the current one. Who receive the XML should read the version that is written in the XML (I think it should be possible...) to elaborate it correctly.

But I think that we should realize that the hash code is a useless computation. I have to compute the hash code for the same piece of code you have considered too and check that they are the same. Bravo! Me and you are able to compute a hash code! And now? It's useful only as version identifier; let's keep it as it arrives from the partner and all the problems go away.

[umesh-qs]

I think that your question may involve three issues:

1. When there is a major version, should we all conform to it and in what amount of time? Or should we be able to use both of them? Until when? Because they can become three versions, four an so on...
2. If we all switch together to the new versions, there could not be any problem for the IIA even mutually approved; but if we don't?

This is not true. Hash will change if the major version changes. Timing of implementation has no impact.

3. As concern the mutually approved versions, I think that we could save the snapshot as a pure XML and return it if we'll have the possibility to ask for the last approved version and not only for the current one. Who receive the XML should read the version that is written in the XML (I think it should be possible...) to elaborate it correctly.

Last approved version can be retrieved from local backup as well, if backups are kept. Problem here is, how would such agreements be show to IROs, so that they don't have to spend time on them.

But I think that we should realize that the hash code is a useless computation. I have to compute the hash code for the same piece of code you have considered too and check that they are the same. Bravo! Me and you are able to compute a hash code! And now? It's useful only as version identifier; let's keep it as it arrives from the partner and all the problems go away.

Agree. Hash should be just considered as version id. IIA approval should be reset based on the change in the content of the cooperation condition.

[demilatof]

I think that your question may involve three issues:

1. When there is a major version, should we all conform to it and in what amount of time? Or should we be able to use both of them? Until when? Because they can become three versions, four an so on...
2. If we all switch together to the new versions, there could not be any problem for the IIA even mutually approved; but if we don't?

This is not true. Hash will change if the major version changes. Timing of implementation has no impact.

Yes, hash changes if the major version changes; but if I'm with 7.0 and you still with 6.0, how do we exchange data? The hash code is not the only problem, obviously, but one of the problems. Depending on how I have designed my internal system, I may use the saved hash code (6.0) to answer to your request about an old approval or compute it again but with the new version of my API (7.0). Same data, different hash.

3. As concern the mutually approved versions, I think that we could save the snapshot as a pure XML and return it if we'll have the possibility to ask for the last approved version and not only for the current one. Who receive the XML should read the version that is written in the XML (I think it should be possible...) to elaborate it correctly.

Last approved version can be retrieved from local backup as well, if backups are kept. Problem here is, how would such agreements be show to IROs, so that they don't have to spend time on them.

This is true if we save both of the backups; our version and partner's version. But if you look at ewp-specs-api-omobilities you can see that there are sending and receiving IIA-IDs. One of them says (the other is similar): "Clients can fetch more information on this agreement from IIAs API served on the sending HEI's servers" So, if I develop a omobilities client, I need to be able to fetch data even from the partner's system, not only for local backup

[mkurzydlowski]

We would like to propose serving a snapshot of partner's IIA in the IIA Approval API. That way version changes won't affect the approval. IIA version update would change the hash in the IIA get response but this would not affect what has already been approved.

[demilatof]

I repeat here some considerations I've already made in the other issue because this should be a more appropriate thread to discuss about the consequences of the namespace upgrade.

We would like to propose serving a snapshot of partner's IIA in the IIA Approval API. That way version changes won't affect the approval. IIA version update would change the hash in the IIA get response but this would not affect what has already been approved.

Could you provide an example? I think that it should be mandatory that the snapshot served with approval API should be saved when Approval CNR was sent. Nevertheless, this is not the point now.

The question to understand your proposal is: why I have to re-approve the IIA if the hash code changes due to namespace version upgrade? To be more clear, what are the hash codes you compare? Because I have to evaluate if your IIA requires a new approval or not, not my IIA! And with Approval API you provide me my IIA; I already know if my own IIA requires a new approval or not, I don't need your copy of my IIA for this. As a minimum, you should provide the snapshots of both IIAs.

I think, as you if I remember well, that what could rule the need of an approval is the partner will: if he sends us an IIA CNR, we can suppose that he needs a new approval (may even be he doesn't need it, of course). The main problem is that we can miss his IIA CNR. To overcome the problem it could be enough adding a new element to the general section of the IIA: <last-cnr-sent-on-date> Doing so we may implement our logic relying on partner's will, even if we have missed his IIA CNR: if it is after the approval, we have to give it again, otherwise not.

This is not the perfect solution, I know: the partner may send us an IIA CNR even if he doesn't need a new approval. We may implement some further check, even comparing the new IIA with our snapshot. But we should introduce the mobility-ID to facilitate this.

We should catch the opportunity of the next namespace upgrade to add all the elements may help us for a better management.

[mkurzydlowski]

The main problem is that we can miss his IIA CNR. To overcome the problem it could be enough adding a new element to the general section of the IIA:

I think that the modified_since parameter of IIA index should be enough.

[demilatof]

I think that the modified_since parameter of IIA index should be enough.

Do you mean the modified_since in the request of IIA Index?

[mkurzydlowski]

Exactly.

[demilatof]

Exactly.

I'm trying to make EWP Network more stable asking for something that could fix the problem for missing IIA CNRs, and you tell me that you "_think that the modifiedsince parameter of IIA index should be enough"? That is, another element that is not sure neither mandatory? From the specifications:

If given, then the server SHOULD filter the returned IIA IDs to the ones which have been either created or modified after the given point in time

This means that the server SHOULD and not MUST filter; "modified after" in our implementation (and not only in our implementation, I think) means every modification, even those we don't need to communicate immediately to the partner because we want to give to the IRO a small amount of time to avoid mistakes.

Servers MAY include IIAs which were not modified.

So it is not much useful...

Servers MAY ignore the modified_since parameter completely, and always respond with the full list of IIA IDs

Again, if this may happen, the parameter it is useless

As we previously explained here, clients MAY use the index and get endpoints as a pull-based method of synchronization, complementary to CNRs

INDEX and GET are useful for the synchronization, but we cannot substitute the IIA-CNR thanks to this parameter. Only inserting the information about the IIA-CNR in the IIA-GET response can give us a robust synchronization by means of INDEX- GET, because we receive the same piece of information we would have received with IIA-CNR, that is the exact moment when the partner decided to inform us about an update.

[janinamincer-daszkiewicz]

We are looking for the simplest possible solution which would allow us to avoid any automatic triggering of re-approval after the change of the name space in IIA (version change from 6.2.0 to 7.0.0) due to changes we plan for the current window. We have some flexibility in choosing the final solution for IIA negotiate/approve/terminate/modify/delete, but the general guidelines are the following:

• it should be simple, clear, easy to understand and implement;
• limited number of options on the network level (local system can offer flexibility, but not complicate the interoperability scenarios);
• CNR is a main tool for triggering exchange of IIA data, not index;
• CNR can get lost but this should be treated like an exceptional situation, not a general rule;
• no mercy for errors in the network (this phrase has often been repeated during the technical workshop in Thessaloniki).

I will post here the more elaborated proposal, trying to take into account recent discussion in GitHub and Thessaloniki.

[demilatof]

which would allow us to avoid any automatic triggering of re-approval after the change of the name space in IIA (version change from 6.2.0 to 7.0.0) due to changes we plan for the current window.

I noticed discussing with @mkurzydlowski that we all agree that presently there is the risk of an automatic triggering of re-approval after the change of the name space in IIA. But I even notice that due to the different implementations we may have different views of what is the element that triggers the re-approval. I think that we should specify what is that triggers a new approval and when before looking for a solution good for all. Can someone share his opinion?

[umesh-qs]

If the content is same and only change is in the namespace then it is not complicated to handle it. Just compare the content. If there is no change, calculate the hash from the content and match it with partner calculated hash. If it matches then update the hash in you local system without resetting any approval.

[demilatof]

Just compare the content.

But how? Field by field? And in which order? It can work, without warranties anyway. A mobility ID may be useful...

[umesh-qs]

Just compare the content.

But how? Field by field? And in which order? It can work, without warranties anyway. A mobility ID may be useful...

You define a object and populate that with the data. Then compare the object. It will work with full warranty :) as long as the sequence is not changed

[demilatof]

You define a object and populate that with the data. Then compare the object. It will work with full warranty :) as long as the sequence is not changed

Maybe it works, even if I had a bad experienced in the past using that system in Java (I've had to implement an algorithm that collects all fields in a given order).

But what are the two versions you try to compare? Your snapshot and the one fetched again from the partner? The new IIA should be downloaded with IIA-Get v. 7; with your solution you can resolve the changes in hash code, but how you do manage different fields due to the new version (even if for the same information)? Adding extra logic for every change?

[umesh-qs]

You define a object and populate that with the data. Then compare the object. It will work with full warranty :) as long as the sequence is not changed

Maybe it works, even if I had a bad experienced in the past using that system in Java (I've had to implement an algorithm that collects all fields in a given order).

But what are the two versions you try to compare? Your snapshot and the one fetched again from the partner? The new IIA should be downloaded with IIA-Get v. 7; with your solution you can resolve the changes in hash code, but how you do manage different fields due to the new version (even if for the same information)? Adding extra logic for every change?

If the new fields are part of hash calculation then there is a problem. In that case there either we have separate logic for each version or reset the approval.

[janinamincer-daszkiewicz]

If the new fields are part of hash calculation then there is a problem. In that case there either we have separate logic for each version or reset the approval.

This is out of the question. We have to find a solution where re-approval is not needed. I think about a timestamp which would allow us to tell if the change in the hash is caused by the business logic change or a technical change (like change in a name space). Only in the first case new action should be triggered.

I will describe this proposal it in more detail later today or tomorrow.

[umesh-qs]

If the new fields are part of hash calculation then there is a problem. In that case there either we have separate logic for each version or reset the approval.

This is out of the question. We have to find a solution where re-approval is not needed. I think about a timestamp which would allow us to tell if the change in the hash is caused by the business logic change or a technical change (like change in a name space). Only in the first case new action should be triggered.

I will describe this proposal it in more detail later today or tomorrow.

so you are saying that even if the hash logic has changed to include new fields, old approvals will remain intact even if the old IIA has new data in it?

[janinamincer-daszkiewicz]

so you are saying that even if the hash logic has changed to include new fields, old approvals will remain intact even if the old IIA has new data in it?

It depends on a specific case. The timestamp would allow us to decide when we have real changes and when only 'technical' changes. Name space is such a technical change. No need to approve again IIA which differs from the approved one only in a name space. Another example: we change the list of academic years to the range <start-academic-year, end-academic-year>. Nothing changes on a business level. Just representation is different.

[umesh-qs]

timestamp

Ok. Will wait for the timestamp proposal before further comments

[demilatof]

I think about a timestamp

so you are saying that even if the hash logic has changed to include new fields, old approvals will remain intact even if the old IIA has new data in it?

A timestamp of what? Three days ago I already suggested a solution by means of a timestamp <last-cnr-sent-on-date>.

it could be enough adding a new element to the general section of the IIA: <last-cnr-sent-on-date> Doing so we may implement our logic relying on partner's will, even if we have missed his IIA CNR: if it is after the approval, we have to give it again, otherwise not.

Doing so, if the partner for whatever reason (valid or not) ask us for a new approval, we can give him a new approval, switching to the new namespace version. If there is no request (IIA CNR), we have nothing to do.

[janinamincer-daszkiewicz]

Timestamp of a last business change in IIA.

[demilatof]

last business change in IIA.

This may require a shared definition of "last business change in IIA". May be that the solution is the set of fields computed in the hash code, but I think we have to investigate on what providers have to modify.

[janinamincer-daszkiewicz]

This may require a shared definition of "last business change in IIA".

Yes, it may be intuitive, but difficult to define in such a way that everybody will understand this the same way. Anyway, we need a method of saying that there are some cases when the change of hash does not mean a (business) change in IIA (only change in its format) and need not trigger an action on the partner side (like re-approval or even comparing the new object with the old object).

May be that the solution is the set of fields computed in the hash code, but I think we have to investigate on what providers have to modify.

We will have (at least) these changes in this round:

• format of academic years
• mobilities-per-year required
• name space

[umesh-qs]

This may require a shared definition of "last business change in IIA".

Yes, it may be intuitive, but difficult to define in such a way that everybody will understand this the same way. Anyway, we need a method of saying that there are some cases when the change of hash does not mean a (business) change in IIA (only change in its format) and need not trigger an action on the partner side (like re-approval or even comparing the new object with the old object).

May be that the solution is the set of fields computed in the hash code, but I think we have to investigate on what providers have to modify.

We will have (at least) these changes in this round:

• format of academic years
• mobilities-per-year required
• name space

How is it different when it comes to implementation, when compared with to having different logic for each version (v6 vs v7)?

[janinamincer-daszkiewicz]

How is it different when it comes to implementation, when compared with to having different logic for each version (v6 vs v7)?

I am not sure I understand your question. Moving from version 6.2.0 to version 7.0.0 will not change logic of already approved IIAs. These is what we care about. We need a solution that will allow us to automatically recognize between two situations:

• this mutually approved IIA has been modified, needs negotiations and then approval;
• this mutually approved IIA has not been modified, only its format changed from 6.2.0 to 7.0.0, no negotiations nor re-approval are needed.

[umesh-qs]

Yes I get the problem. My question was, how is the solution you are proposing different from what I have proposed in terms of implementation?

[janinamincer-daszkiewicz]

Yes I get the problem. My question was, how is the solution you are proposing different from what I have proposed in terms of implementation?

Can you link to this proposal? I have not yet described my proposal in full. Need some time to polish it.

[umesh-qs]

You define a object and populate that with the data. Then compare the object. It will work with full warranty :) as long as the sequence is not changed

Maybe it works, even if I had a bad experienced in the past using that system in Java (I've had to implement an algorithm that collects all fields in a given order). But what are the two versions you try to compare? Your snapshot and the one fetched again from the partner? The new IIA should be downloaded with IIA-Get v. 7; with your solution you can resolve the changes in hash code, but how you do manage different fields due to the new version (even if for the same information)? Adding extra logic for every change?

If the new fields are part of hash calculation then there is a problem. In that case there either we have separate logic for each version or reset the approval.

This is what I have proposed which you just rejected. In any solution you will have to add a special logic in the implementation.

[janinamincer-daszkiewicz]

I have rejected? Where? This is your discussion with Demilatof. Please wait for the full description of my proposal.

[umesh-qs]

If the new fields are part of hash calculation then there is a problem. In that case there either we have separate logic for each version or reset the approval.

This is out of the question. We have to find a solution where re-approval is not needed. I think about a timestamp which would allow us to tell if the change in the hash is caused by the business logic change or a technical change (like change in a name space). Only in the first case new action should be triggered.

I will describe this proposal it in more detail later today or tomorrow.

What does "This is out of the question" means if not rejected?

[janinamincer-daszkiewicz]

"Out of the question" relates to "we have separate logic for each version or reset the approval". This is what we don't want to have.

[demilatof]

We need a solution that will allow us to automatically recognize between two situations:

• this mutually approved IIA has been modified, needs negotiations and then approval;
• this mutually approved IIA has not been modified, only its format changed from 6.2.0 to 7.0.0, no negotiations nor re-approval >are needed.

Now I'm in brainstorming mode. I think that the main problem is the capability of distinguishing the two cases when 6.2.0 and 7.0.0 are deeply incompatible, therefore we need to pay attention in writing 7.0.0 specification. If we move from receiving-academic-year-id to receiving-academic-year-start-id (and receiving-academic-year-end-id), maybe we still have the same data and can rearrange it. But if we introduce a new element, e.g. <semester>, and we declare it mandatory, we may have a huge problem:

1. The IIA is approved and don't require to specify the semester until we don't touch it
2. BUT, if we fetch the IIA by means of IIA-Get, we have to add a semester value (may be the partner has never set this value)

That is, the need for a new approval is due to our activity (IIA-Get) and not to partner's need or will. May we avoid declaring any new element as mandatory? This could limit the problem. Or I'm wrong?

[janinamincer-daszkiewicz]

This is an overall vision how specification and implementation might look when name space is changed. Not the only one, but this one should work.

1. Some definitions

1.1 IIA object (structured record)

• living object;
• may be modelled differently in each system (different tables and relations);
• may change when the name space changes;
• (new proposal) starting from v7 contains a timestamp of the last semantic change; technical changes (like name space change) do not change the timestamp of the object;
• contains an IIA hash calculated from the cooperation conditions (CC);

1.2 Mutually Approved (MA) IIA object (structured record)

• IIA object after it has been mutually approved;
• is the basis for running mobility processes (some other objects in the local system link to it);

1.3 IIA XML (string)

• standardised by the XSD schema;
• local system creates it from the IIA object taking the current name space into account;
• from v7 contains a timestamp;
• (new proposal) signed for non-repudiation, this need not be a qualified signature;
• (open question) if we agree to sign IIA, will signatures be optional?
• exchanged via the network;
• (new proposal) stored as part of the IIA Approval object;

1.4 IIA hash (string)

• calculated from the local IIA object taking the current name space into account before being send over the network;
• calculated by the standardised algorithm;
• if two IIA XMLs have different CC sections (even if the change is only in name space) the hashes also differ;
• send over the network in the IIA get response;
• stored as part of the IIA Approval object;
• send over the network in the IIA Approval get response;
• also used in some local systems to recognize the version of an object (if the new object is different from the previous one);

1.5 IIA Approval object

• consists of IIA XML and IIA hash of the last approved partner's copy of IIA;
• from v7 separately keeps timestamp taken from the IIA XML (initially empty);
• before the local system stores IIA XML and IIA hash of the partner's copy of IIA it calculates IIA hash from the IIA XML to check if the IIA object and the IIA hash match;
• local system stores partner's IIA Approval object ("I have approved this particular object of the partner")

2. Some actions and their impact on the stored data

2.1 During negotiations, before IIA gets mutually approved (for the first time) each partner stores separately:

• local IIA object, from version 7.0 it contains a timestamp;
• partner's IIA XML, and separately its hash, from version 7.0 also separately a timestamp;

2.2 After IIA gets mutually approved (MA) each partner stores:

• local MA IIA object, from version 7.0 it contains a timestamp;
• partner's MA IIA XML, separately its hash, approval date, from version 7.0 also separately a timestamp;

2.3 Mutually approved IIAs can not be deleted.

2.4 Mutually approved IIAs can be modified. Modification triggers re-approval. Before negotiation ends each partner stores:

a) Last MA version

• local MA IIA object, from version 7.0 it contains a timestamp;
• partner's MA IIA XML, separately its hash, approval date, from version 7.0 also separately a timestamp;

b) New negotiated version (linked to the local MA IIA object)

• local IIA object, from version 7.0 it contains a timestamp;
• partner's IIA XML, separately its hash, from version 7.0 also a timestamp;

2.5 After negotiation ends and a new MA version is agreed (possibly equal to the previous one) each partner stores:

a) Last MA version (it's up to the local system if the whole list of MA IIAs is stored or only the last MA IIA)

• local MA IIA object, from version 7.0 it contains a timestamp;
• partner's MA IIA XML, separately its hash, approval date, from version 7.0 also separately a timestamp;

2.6 Mutually approved IIAs can be terminated. Termination is a special case of modification (only deletions are allowed). Termination involves re-approval. See https://github.com/erasmus-without-paper/ewp-specs-api-iias/issues/99.

2.7 Mutually approved IIAs can be modified. After modification the usual approval process follows.

3. Data exchanges

3.1 What is being sent in:

• IIA get response: -- local IIA XML (with timestamp and hash),

• IIA Approval get response : -- (open question) partner's IIA XML; separately its hash;

3.2 What happens when name space changes:

a) every system changing name space from 6 to 7 makes a change in a local database

• local MA IIA object V6 is translated to version V7 (newly calculated hash V7 replaces the previous hash V6; new timestamp field is empty)
• partner's MA IIA XML V6 does not change;
• partner's MA IIA hash V6 does not change;
• new partner's timestamp field is empty;
• approval date does not change;

b) every system which does not yet store partner's IIA XML, can get and store it (it's better to get them in advance); this is one time action

3.3 What happens in these rare cases when local system not being triggered by CNR from the partner calls IIA get to get the object from the partner?

• IIA get V7 returns partner's MA IIA XML V7 (with partner's MA IIA hash V7 and timestamp); hash V7 differs from hash V6 but timestamp from MA IIA XML V7 is the same as timestamp from MA IIA XML V6 (is also true for empty timestamps); no re-approval is needed;
• if not, re-approval starts because human's involvement is needed.

4. Other issues

4.1 How to compare own IIA with partner's IIA - helpful hints (please share, in Thessaloniki a couple of providers claimed that they semi-automatically do some comparison and show the users parts which have changed)

5. Summary

5.1 Local system always stores the last mutually approved IIA, does not need to get it from the partner.

5.2 IIA get response delivers last IIA object, either mutually approved or negotiated.

5.3 Timestamps included in the IIA object obtained in IIA get response and the last partner's MA IIA XML stored locally in the Approval object can be used for verification if the IIA object has changed.

5.4 I do not see the danger that differences between 6.2.0 and 7.0.0 will change the semantic of the already existing IIAs.

[demilatof]

Thanks God the request from Thessaloniki was for "strict and simple rules"... May be I'm wrong, but I see a lot of intrusion in the local systems telling what we have to save but not what a saved version could be useful for.

starting from v7 contains a timestamp of the last semantic change;

What is a semantic change and what elements does it take into account?

1.2 Mutually Approved (MA) IIA object (structured record)

Where is written that my internal system must have a structured record of the partner's IIA? For example, I save the pure XML and a few elements to identify it. I can create a structured table from the XML, if you compel me, but why?

every system changing name space from 6 to 7 makes a change in a local database

I'm not really really sure to like this point; the namespace changing make a change in the IIA Get response, not in the database. Not automatically.

• local MA IIA object V6 is translated to version V7 (newly calculated hash V7 replaces the previous hash V6; new timestamp field is empty)

I don't think that providers would like this approach that infringes master-master model. I cannot allow myself to manage partner's data and I cannot state that they are approved, with a hash code (the famous proof) that now I create myself. Doesn't matter that there is a technical need.

All that said, if an element would be mandatory in v7, I cannot even perform this transformation. But, most important, since the new hash code is calculated by me, do you really hope that it could be the same that will be calculated by the partner when he builds the IIA-Get response? Years talking about the fact that a simple order or space can change the hash code, pointing out that we have to compute it on the same XML received by the partner, and now you require us to calculate it separately?

It simply doesn't work.

b) every system which does not yet store partner's IIA XML, can get and store it (it's better to get them in advance); this is one time action

In the mean time it could have been changed and differs from the one approved.

3.3 What happens in these rare cases when local system not being triggered by CNR from the partner calls IIA get to get the object from the partner?

- IIA get V7 returns partner's MA IIA XML V7 (with partner's MA IIA hash V7 and timestamp); hash V7 differs from hash V6 but timestamp from MA IIA XML V7 is the same as timestamp from MA IIA XML V6 (is also true for empty timestamps); no re-approval is needed;

I don't understand the difference between an IIA-Get triggered and not by an IIA-CNR. But this is less important, given that the hash code would be always different because we cannot guess the partner's format when we do a massive upgrade of the hash code from v6 to v7.

I think that we have to look for a simpler solution that requires minimal changes in every internal system.

[janinamincer-daszkiewicz]

May be I'm wrong, but I see a lot of intrusion in the local systems telling what we have to save but not what a saved version could be useful for.

This is my vision, as I wrote in the introduction. Every provider will implement his own vision. Important is only the common understanding of what is happening in the network.

What is a semantic change and what elements does it take into account?

I hoped this is intuitive, if not some strict definition will be given. Generally it mean that nothing changes in IIA on a business level.

1.2 Mutually Approved (MA) IIA object (structured record)

Where is written that my internal system must have a structured record of the partner's IIA? For example, I save the pure XML and a few elements to identify it. I can create a structured table from the XML, if you compel me, but why?

Of course, I hope I wrote exactly that. You keep structured version of your copy and XML of partner's copy.

every system changing name space from 6 to 7 makes a change in a local database

I'm not really really sure to like this point; the namespace changing make a change in the IIA Get response, not in the database. Not automatically.

The namespace yes. But if there are other changes you will have to adapt your local model. For example, if we accept the idea of the timestamp, the extra field has to be added. If we change from the list of academic years to start-end academic years some of us will also have to adapt the model. This is what I mean by changing locally the format of the objects from 6 to 7.

• local MA IIA object V6 is translated to version V7 (newly calculated hash V7 replaces the previous hash V6; new timestamp field is empty)

I don't think that providers would like this approach that infringes master-master model. I cannot allow myself to manage partner's data and I cannot state that they are approved, with a hash code (the famous proof) that now I create myself. Doesn't matter that there is a technical need.

You do it on your copy of the object, not on the partner's copy. Master master model does not change.

All that said, if an element would be mandatory in v7, I cannot even perform this transformation.

It can be done, we just would have to decide in what way.

But, most important, since the new hash code is calculated by me, do you really hope that it could be the same that will be calculated by the partner when he builds the IIA-Get response? Years talking about the fact that a simple order or space can change the hash code, pointing out that we have to compute it on the same XML received by the partner, and now you require us to calculate it separately?

Nothing changes in the way hash is used. This is the reason why after switching to V7 the hash you have stored is in partner's copy (XML) is different from the new hash calculated by the partner. Here timestamp helps.

b) every system which does not yet store partner's IIA XML, can get and store it (it's better to get them in advance); this is one time action

In the mean time it could have been changed and differs from the one approved.

Yes, but this is different use case. Anyway, I presume everybody keeps it in his logs, so can take from the logs.

I don't understand the difference between an IIA-Get triggered and not by an IIA-CNR.

If the partner does not change the object semantically, he will not send you IIA CNR. But you may do it by your own will. In this case the timestamp helps.

But this is less important, given that the hash code would be always different because we cannot guess the partner's format when we do a massive upgrade of the hash code from v6 to v7.

This is why we need timestamp.

I think that we have to look for a simpler solution that requires minimal changes in every internal system.

This is exactly what I would like to achieve, with the help of the whole community. From your comments I see that my description is not clear. If need be we can have a quick call with whoever is available and discuss this issue in a meeting. I am available today, tomorrow befor 13, on Wednesday before 12.

[demilatof]

This is my vision, as I wrote in the introduction. Every provider will implement his own vision. Important is only the common understanding of what is happening in the network.

My bad, but I haven't understood yet what should happen in the network to solve the issue of hash changing.

What is a semantic change and what elements does it take into account?

I hoped this is intuitive, if not some strict definition will be given. Generally it mean that nothing changes in IIA on a business level.

You're explaining a definition with another definition; I need an example. May be I'm not used with such a terminology, but for me a semantic change is even the sending/receiving contact, that normally don't trigger any new approval because they are excluded from hash calculation

every system changing name space from 6 to 7 makes a change in a local database

I'm not really really sure to like this point; the namespace changing make a change in the IIA Get response, not in the database. Not automatically.

The namespace yes. But if there are other changes you will have to adapt your local model. For example, if we accept the idea of the timestamp, the extra field has to be added. If we change from the list of academic years to start-end academic years some of us will also have to adapt the model. This is what I mean by changing locally the format of the objects from 6 to 7.

Ok, now I understand what you mean; anyway it depends on a local design and not only for a namespace upgrade (we may need a new field even staying with the same version)

• local MA IIA object V6 is translated to version V7 (newly calculated hash V7 replaces the previous hash V6; new timestamp field is empty)

I don't think that providers would like this approach that infringes master-master model. I cannot allow myself to manage partner's data and I cannot state that they are approved, with a hash code (the famous proof) that now I create myself. Doesn't matter that there is a technical need.

You do it on your copy of the object, not on the partner's copy. Master master model does not change.

What do you mean by this? In my copy of the partner's IIA or in my IIA? If I have to do with my copy of my IIA, I think it is useless. If I have to do this transformation with my copy of partner's IIA to store it with the MA IIA, I'm creating a fake proof (the hash code must be generated by the owner of the IIA) and it cannot even be equal to the one calculated by the partner.

All that said, if an element would be mandatory in v7, I cannot even perform this transformation.

It can be done, we just would have to decide in what way.

Here we need strict rules

But, most important, since the new hash code is calculated by me, do you really hope that it could be the same that will be calculated by the partner when he builds the IIA-Get response? Years talking about the fact that a simple order or space can change the hash code, pointing out that we have to compute it on the same XML received by the partner, and now you require us to calculate it separately?

Nothing changes in the way hash is used. This is the reason why after switching to V7 the hash you have stored is in partner's copy (XML) is different from the new hash calculated by the partner. Here timestamp helps.

I don't understand this point; you didn't say that we have to recalculate the hash code in our copy? For what then? How does the timestamp help? I need a scenario to understand better.

I don't understand the difference between an IIA-Get triggered and not by an IIA-CNR.

If the partner does not change the object semantically, he will not send you IIA CNR. But you may do it by your own will. In this case the timestamp helps.

The problem is the word "semantically"; by specifications, a new IIA CNR should be necessary for every change in IIA, even those that doesn't trigger a new approval. And so? In this case, from the approval point of view, we have fetched an IIA that doesn't need a new approval and there is no difference if fetched after an IIA-CNR or by periodic polling

But this is less important, given that the hash code would be always different because we cannot guess the partner's format when we do a massive upgrade of the hash code from v6 to v7.

This is why we need timestamp.

If we rely only on timestamp (how?), we don't need all the complications in recalculating all the hash code in our databases.

[janinamincer-daszkiewicz]

My bad, but I haven't understood yet what should happen in the network to solve the issue of hash changing.

My bad, it seems that my description is not good enough. Generally, timestamp will tell us if we should care about the hash change.

You're explaining a definition with another definition; I need an example. May be I'm not used with such a terminology, but for me a semantic change is even the sending/receiving contact, that normally don't trigger any new approval because they are excluded from hash calculation

I have given some examples above but OK. We take into account only these changes which influence hash. The examples of changes which are not semantic changes:

• name space
• list of academic years changed to start-end academic years

Ok, now I understand what you mean; anyway it depends on a local design and not only for a namespace upgrade (we may need a new field even staying with the same version)

Yes

What do you mean by this? In my copy of the partner's IIA or in my IIA? If I have to do with my copy of my IIA, I think it is useless. If I have to do this transformation with my copy of partner's IIA to store it with the MA IIA, I'm creating a fake proof (the hash code must be generated by the owner of the IIA) and it cannot even be equal to the one calculated by the partner.

I keep my IIA as an object and I keep partner's IIA as an XML. I never change anything in this XML. I store it as part of a prove of what I have approved. When name space changes I transform my IIA objects (MA and a new under negotiation) to new formats.

Of course, I calculate hasf from the partner's XML to compare with partner's hash. I calculate hash from my object when I send it to the partner (as XML).

Here we need strict rules

Of course. For each particular change of that type. The only wating is 'The number of mobilities' as I recall.

I don't understand this point; you didn't say that we have to recalculate the hash code in our copy? For what then? How does the timestamp help? I need a scenario to understand better.

It looks the more I write the less clear it is. I am ready to explain in the meeting. I am trying to say that considering hashes - when they are calculated and in what way- nothing changes. But we need timstamps to decide that when we get IIA from the partner and the hash is different from the hash I have stored to decide if this is a 'sematic' change or 'formal' change (in data format).

The problem is the word "semantically"; by specifications, a new IIA CNR should be necessary for every change in IIA, even those that doesn't trigger a new approval.

The specification says that a new IIA CNR should be necessary for every change in IIA. Under 7.0 we will say 'IIA CNR should be necessary for every semantic change in IIA, which also changes timestamp'.

If we rely only on timestamp (how?), we don't need all the complications in recalculating all the hash code in our databases.

You do not need to 'recalculate' them. Hashes are calculated on request, when you reply to get or index.

[demilatof]

Now that I've better understood your vision, I try to summarize and simplify (from my perspective), because I think that some information may be useful, but already known, whilst something could become useless (spoiler: the hash code).

Generally, timestamp will tell us if we should care about the hash change.

We take into account only these changes which influence hash.

I like the idea of timestamp (more versatile if it is an alphanumeric string), but there is no reason to use it to tell us if we should care about the hash change. We use it and stop, the hash code is useless at this point; if timestamp tells us that the hash code is changed we don't need to calculate the hash code. And if the timestamp tells us that hash code is not changed, we have to ignore the hash code again, even if it is changed (most probably due to the namespace upgrade). We should forget that hash code is a proof, it is a proof of nothing.

Therefore, if I receive a timestamp, different from the previous one, the associated IIA needs to be re-approved. If there is a namespace upgrade it's upon the owner of the IIA to use a timestamp identical to the hash code used for the last approval received (previous namespace), it there is no semantic change. Otherwise, if there is a semantic change, he starts following the rule of the timestamp.

I keep my IIA as an object and I keep partner's IIA as an XML.

Ok, are we switching to a document management? I already have the partner's IIA as an XML, for internal use, so no problem for me. But what is the purpose of your proposal? That is, what I've to do with that XML?

When name space changes I transform my IIA objects (MA and a new under negotiation)

What should I have to transform? I think nothing for me, but I suppose nothing for most of the others too Now I have my IIA (structured), the partner's IIA (XML) and an MA with:

my_iia_id, my_hash_code, your_hei_id, your_iia_id, your_hash_code timestamp_approval_cnr_sent_to_you, timestamp_approval_got_from_you

Namespace upgrade should not change anything

Under 7.0 we will say 'IIA CNR should be necessary for every semantic change in IIA,

Ok, I agree, this is really helpful. I find all of this similar (but a bit more complicated) to the idea I suggested a week ago:

1. The partner sends an IIA CNR only if he needs a new approval
2. The partner adds the IIA CNR date (your timestamp) in IIA Get Response
3. If the above date (2) is more recent than the last approval date, I have to re-approve (at that time I thought to save both hash code and date, to be compliant with the specifications, but we can get rid of the hash code)

No more need for the hash code, no need to specify what should do every internal system after a namespace upgrade.

[janinamincer-daszkiewicz]

I talked today shortly with Michał, and he suggested something going in the same direction, but may be more revolutionary (some implementations may get more affected). Yes, we agree that hash is useless as a proof. Hash is just a version number. But if we have a version number we do not need timestamp. So:

• let's get rid of hash,
• let's replace timestamp with a version number.

Now instead of using not-so-clear phrase 'semantic change' we just say 'this is the new version of the object'.

But to have some strong proof of what was approved we should keep signed XML of partner's copy (with no hash, but with a version number).

[demilatof]

I think that we are in the good way.

But to have some strong proof of what was approved we should keep signed XML of partner's copy (with no hash, but with a version number).

I think that the word "signed" could be misleading if used without a private/public key infrastructure (as a minimum). If we decide to accept a complete XML as a proof, neither the version number or the hash code adds something more. In any case, the XML will contain that version number, therefore if someone wishes to consider his/her XML as "signed" by that version number, he/she can.

But, if we have a proof, we must be able to produce it on demand. This means that if the full XML is our proof, we should be able to show it to the partner: "this is your proof!". This last step could be better understood if we think that we're preparing the infrastructure to replace the version number with a signed digest of cooperation conditions (for example; anyway it will take some years, I suppose...).

[mkurzydlowski]

1. Hash as version id

It looks like we agree that the hash with it's calculation algorithm is a problematic tool for IIA approval. So I propose going back to the original meaning of the conditions-hash attribute where:

• we don't impose an algorithm for calculating it - this means it is not going to be verifiable (I will get back to that),
• we impose that it does change only if cooperation conditions change ("excluding sending-contact and receiving-contact subelements"); so it mustn't change when only the XML representation of the data changes (e.g. XML formatting, element ordering, XML namespace, XML structure after version change).

For the latter requirement to be fulfilled an implementer should store the hash after it generates it and change it based on the data change. You can see this hash as a cooperation conditions version id.

2. IIA XML in approval

But having only a hash in the IIA approval, that is not verifiably tied to the exact IIA data being approved, is not good enough:

• Partner B states it approves A's IIA with cc hash X and stores A's IIA get response Y to be able to prove what it looked like.
• Partner A gets the approval from B but it currently consists only of the hash (PDF is optional). So if A would need to prove what B approved he can only say "B approved my IIA with cc hash X" but we should be looking for "B approved an IIA that looked like Y".

That's why I would simultaneously change the IIA approval to serve the whole IIA get response XML that B had to store anyway. After that happens A is going to be able to prove the exact IIA data being approved by B.

3. IIA signing - proof

But for such XML to be used as a proof it should ideally be signed. Otherwise B might be contesting the truth by:

• changing or removing his IIA approval from the network,
• challenging A's word by saying that the XML provided as a proof (received from B's IIA approval) is not true.

[umesh-qs]

I am not sure what proof are we talking about. Entire EWP network is based on trust. And why do we want proof only in IIA, why not in LA? Even if it is HTTP signature, how will you prove it, unless you save the public key as well along with each object, and later on able to prove that the public signature was indeed of a particular university in case it changes multiple time.

[demilatof]

After that happens A is going to be able to prove the exact IIA data being approved by B.

I think that with the current solution "show" could be more correct than "prove" But what kind of prove it could be? "A" could show whatever it wants and claims it received with approval response from B even if it is not true. And then, if they are different? We add an XML for nothing.

I am not sure what proof are we talking about. Entire EWP network is based on trust.

I agree with @umesh-qs

With the full XML, we have only half of a weak proof: A can know what B has approved.

But B cannot know if A has badly implemented the approval, marking as approved the right XML.

If someone is interested in the fact that both of the systems are aligned in saying that B's IIA (IIA ID X, hash Y) is approved, he should be able to compare in any moment his copy of B's IIA (the snapshot he has approved) with the B's IIA approved copy. The two copies must have the same IIA ID, the same hash and the same XML. And vice versa.

[mkurzydlowski]

I might propose three solutions for point 3 from my previous comment (approval proof):

1. We decide that point 2 (IIA XML in approval) is enough to satisfy the business need of having proof that the partner has approved our IIA. Although technically it does not ensure non-repudiation.
2. We decide to use HTTP signature server authorization to ensure non-repudiation when requesting the IIA XML from IIA approval get. Clients willing to receive a proof will ask for HTTP signature on the response and will store the response with the signature present in the header. To verify who the signer was, one could use the registry log stored on GitHub.
3. We decide to use qualified electronic signature (or should this be only possible as an extension in the future?).

[BavoNootaert]

I don't think changing the namespace poses problems for the hash computation. Even if the namespace changes during the approval process resulting in a new hash, this poses no problem, as according to the specifications, partners should perform an IIA Get before sending an IIA Approval CNR. When the approval process is over, the hash is only used as a form of signature: only in rare circumstances does it need to be verified. In such case it suffices to process the original XML, with the namespace used during the approval processs. It doesn't mean that any application code still needs to function with that version of the API: the application doesn't need to be able to generate XML for that API version or parse such XML received from the partner. The proposed alternatives add unnecessary complexity and development costs.

[janinamincer-daszkiewicz]

The application should respond to IIA Get any time, also after approval. even in case when CNR has not been sent by the application before. What namespace should be used in such response? The only valid at this moment is the new one.

[BavoNootaert]

This question goes further than just the hash computation or the proof of approval. It may not be possible for an IIA that has been approved in an old version of the API to be represented in the new version (e.g. a missing element that has become mandatory). When exactly would one fetch an IIA from the partner some time after approval?

[janinamincer-daszkiewicz]

Any time you make a change in the database from nullable to not null you have to set some default values when the field is empty.

When exactly would one fetch an IIA from the partner some time after approval? Approved IIA in the new format.

[demilatof]

Any time you make a change in the database from nullable to not null you have to set some default values when the field is empty.

Of course; therefore we have to decide what are the default values we have to use when an element was not in the previous version, but it could be a value we decide to set to modify the IIA. Here could be useful the version number (instead of hash code) do distinguish if there is a real change or not

When exactly would one fetch an IIA from the partner some time after approval?

I think that several providers do a daily polling through the Index to discover if they missed an IIA CNR and there are some IIAs changed