search

erasmus-without-paper / ewp-specs-api-iias

Specifications of EWP's Interinstitutional Agreements API.
MIT License
4 stars 13 forks source link

Can we temporarily hide IIA which waits for user decision? #100

Closed janinamincer-daszkiewicz closed 1 year ago

janinamincer-daszkiewicz commented 1 year ago

This question has been asked in one of the closed issues "Can we temporarily hide IIA which waits for user decision?"

In https://github.com/erasmus-without-paper/ewp-specs-api-iias/issues/98 we discuss and will decide on deletion functionality. When we delete, we keep information on deleted IIAs. If the object has not been deleted but it disappears from the network, how should it be interpreted?

We agreed to expose to the partner only versions ready for approval (no drafts). However, when the partners negotiate (have not yet reached consensus), can the current stored version be treated as ready for approval or should rather be hidden?

When we have comments and can put some disclaimer in the comment, is hiding needed?

I suggest that we make this clear in specification whether hiding is allowed and how it should be interpreted.

umesh-qs commented 1 year ago

Comments cannot be a solution to this because we are then forcing IROs to manually check comments on each IIA to determine whether to act on it or not. We need to find a better way. I do not see any problem with hiding IIAs or explicitly marking IIA as draft. Not sure how we ok with comments in IIA but not a status field to determine at which stage the IIA is.

demilatof commented 1 year ago

I agree with @umesh-qs about comments; I think they should not be a panacea.

And if the partners negotiate a change after the approval, we should accept that an approved IIA disappears from the network? I don't like the idea that an IIA can disappear; moreover, I think that a partner needs to see the other partner's IIA to modify its own copy. Unless we think that they must exchange their copies with paper to be able to edit their own IIA in blinded fashion.

We have discussed to make both IIA IDs compulsory, so that if there isn't the partner's IIA ID in our copy, our partner should not approve it. Therefore, during negotiation could be enough not adding both IIA Ids to make an IIA not ready for approval.

mbilalari commented 1 year ago

We have see that some partners are using IIA CNRs after their IRO's changing something on IIA but they are not sharing any data on their IIA GET because of using that approach. This is just confusing, not using CNRs until ready to share data is more logical for this approach.

janinamincer-daszkiewicz commented 1 year ago

If no comments and no hiding, two options are left:

  1. We live with what we have, meaning that when A is thinking about the new proposal of B, the old proposal of A is available in the network.
  2. We come back to the idea of states to differentiate between 'ready for approval' and 'not ready for approval'. However the idea of states was rejected some time ago. In particular BPOs were strict about not having drafts in the network.
demilatof commented 1 year ago
  1. We live with what we have, meaning that when A is thinking about the new proposal of B, the old proposal of A is available in the network.

This is the only reasonable solution with master-master model: I have a set of my IIAs, you have a set of yours IIA. I have to see yours IIAs to understand:

  1. What of mines corresponds to one of yours
  2. If I have to change something in my IIA to make it matching your IIA

And you should do the same. In the meantime, you could have downloaded an old version of my IIA; this is a risk we have to accept if we do so without receiving an IIA-Cnr. But if we take into account the IIA/Approval CNR this should not be a problem.

If we think that we cannot accept the two above points, we don't need EWP: we agree on an IIA by other way (e.g. exchanging them by mail) and then we only send the two IIA-IDs in a blind fashion.

umesh-qs commented 1 year ago

@demilatof If we are basing a scenario on IIA CNR, then the IIA can be hidden temporarily while changes are being made and send CNR only after all the changes are done.

janinamincer-daszkiewicz commented 1 year ago

If we are basing a scenario on IIA CNR, then the IIA can be hidden temporarily while changes are being made and send CNR only after all the changes are done.

Any node has the right to get IIA from the partner any time, not waiting for CNR. And should get it unless it has been deleted.

umesh-qs commented 1 year ago

If we are basing a scenario on IIA CNR, then the IIA can be hidden temporarily while changes are being made and send CNR only after all the changes are done.

Any node has the right to get IIA from the partner any time, not waiting for CNR. And should get it unless it has been deleted.

I was replying to @demilatof and not supporting the idea of depending on CNR

demilatof commented 1 year ago

If we are basing a scenario on IIA CNR, then the IIA can be hidden temporarily while changes are being made and send CNR only after all the changes are done.

Any node has the right to get IIA from the partner any time, not waiting for CNR. And should get it unless it has been deleted.

I was replying to @demilatof and not supporting the idea of depending on CNR

@janinamincer-daszkiewicz @umesh-qs I'm not saying that we should depend on CNR, but that in the EWP model, CNRs are the only timing system we have for IIAs. You can get IIA from the partner any time, but at any time your partner can change it and you have no right to complain if you haven't received a CNR. CNR is most important for who sends it, because it's like a state: "now I'm ready for...". If you receive an Approval CNR, your partner has given you half of an approval

umesh-qs commented 1 year ago

If we are basing a scenario on IIA CNR, then the IIA can be hidden temporarily while changes are being made and send CNR only after all the changes are done.

Any node has the right to get IIA from the partner any time, not waiting for CNR. And should get it unless it has been deleted.

I was replying to @demilatof and not supporting the idea of depending on CNR

@janinamincer-daszkiewicz @umesh-qs I'm not saying that we should depend on CNR, but that in the EWP model, CNRs are the only timing system we have for IIAs. You can get IIA from the partner any time, but at any time your partner can change it and you have no right to complain if you haven't received a CNR. CNR is most important for who sends it, because it's like a state: "now I'm ready for...". If you receive an Approval CNR, your partner has given you half of an approval

I would somewhat agree to it. Manually pulling of IIA should be rare. Also it might be not be frequent that someone pulls the IIA manually in the same time frame while changes are being made.

janinamincer-daszkiewicz commented 1 year ago

I will try to summarise how I see it.

  1. BPOs do not want to share drafts in the network.
  2. Whatever is exposed in the network has once been agreed and potentially could be approved.
  3. If the situation changes and it is not true any more, we still have the option not to finalise the approval process.
  4. Both copies have to be approved to finalise the process. The partner who is not yet ready will not send IIA Approval CNR, but will make a change and will send IIA CNR.
  5. That means that there is no need to hide the object from the network.
  6. Object disappearing from the network should be treated as deleted.
umesh-qs commented 1 year ago

I will try to summarise how I see it.

  1. BPOs do not want to share drafts in the network.
  2. Whatever is exposed in the network has once been agreed and potentially could be approved.
  3. If the situation changes and it is not true any more, we still have the option not to finalise the approval process.
  4. Both copies have to be approved to finalise the process. The partner who is not yet ready will not send IIA Approval CNR, but will make a change and will send IIA CNR.
  5. That means that there is no need to hide the object from the network.
  6. Object disappearing from the network should be treated as deleted.

This is contradictory to https://github.com/erasmus-without-paper/ewp-specs-api-iias/issues/98 where the discussion on possibility of objects re-appearing and not changes needed in the current specification to handle DELETE.

janinamincer-daszkiewicz commented 1 year ago

I do not see contradiction. If the object disappears, I assume that it has been deleted. If it re-appears, I contact my partner to explain what is going on.

umesh-qs commented 1 year ago

I do not see contradiction. If the object disappears, I assume that it has been deleted. If it re-appears, I contact my partner to explain what is going on.

I do not understand this. I thought you mentioned that DELETE is final. If I consider your suggestion. What is the next step. Partner says I deleted by mistake. But based my process on DELETE being final and I deleted my IIA as well? What should my client do?

demilatof commented 1 year ago

I think we have always to keep present that one thing is the content of the agreement reached by the partners and another thing is the structure used by each partner to represent the agreement. This structure is our IIA.

I will try to summarise how I see it.

1. BPOs do not want to share drafts in the network.

BPOs should understand that if there was no draft in the network we would not need EWP at all: all would be already settle. We should expose the best version of our IIAs, but when we will compare them with the partner's version, one of us or both of us might need to tune its copy.

2. Whatever is exposed in the network has once been agreed and potentially could be approved.

It was agreed but the two IIAs could not have the same structure, so they are different. Once again, if they were already agreed, what we have to do with EWP? Just approve? If they were already agreed without changes, they are implicitly approved.

3. If the situation changes and it is not true any more, we still have the option not to finalise the approval process.

And then? How should we manage the IIA that cannot be approved? We have to delete it because it cannot be modified, otherwise we admit that it was a draft? Do you realize what a mess?

4. Both copies have to be approved to finalise the process. The partner who is not yet ready will not send IIA Approval CNR, but will make a change and will send IIA CNR.

And therefore it was exposing a draft version.

5. That means that there is no need to hide the object from the network.

I agree

6. Object disappearing from the network should be treated as deleted.

In my opinion should be considered "unavailable", allowing the partner to:

  1. not throw away its copy due to a network failure or a partner's system failure
  2. not throw away its copy due a partner's officer mistake
  3. recover its work if the partner's IIA appears again

An internal system can implement the above solutions and implements the best strategy for their officers once the receive a delete/mistake confirmation from the partner

janinamincer-daszkiewicz commented 1 year ago

I do not understand this. I thought you mentioned that DELETE is final.

Yes, it is final. But I assume that by 're-appearing' you mean some erroneous situation which can always happen in this world of incorrect implementations.

If partner deleted (by mistake), you deleted (because your partner deleted), your partner re-appears the object (with the same id), you contact the partner and ask what is going on. The possible option is that you treat this re-appearing object as a new one and create a copy in your system. The other option is that you ask your partner to remove it permanently.

demilatof commented 1 year ago

I do not see contradiction. If the object disappears, I assume that it has been deleted. If it re-appears, I contact my partner to explain what is going on.

I do not understand this. I thought you mentioned that DELETE is final. If I consider your suggestion. What is the next step. Partner says I deleted by mistake. But based my process on DELETE being final and I deleted my IIA as well? What should my client do?

This is the reason why I said that we should mark as unavailable an IIA that dissappears from network until we have the partner's officer confirmation. We put in standby our process tied to that IIA and after a confirmation, we decide to resume or delete it. This is the best conservative and error proof solution

umesh-qs commented 1 year ago

I do not understand this. I thought you mentioned that DELETE is final.

Yes, it is final. But I assume that by 're-appearing' you mean some erroneous situation which can always happen in this world of incorrect implementations.

If partner deleted (by mistake), you deleted (because your partner deleted), your partner re-appears the object (with the same id), you contact the partner and ask what is going on. The possible option is that you treat this re-appearing object as a new one and create a copy in your system. The other option is that you ask your partner to remove it permanently.

This process is asking for more problems for the end user. We are just complicating life of one partner due to mistake of other. Basically you are saying that my partner can recover from his mistake easily and I have to cleanup the mess that they created. How fair is this. We are against this process. I am with @demilatof on marking them 'unavailable'. In fact that is what we are already doing.

janinamincer-daszkiewicz commented 1 year ago

I am with @demilatof on marking them 'unavailable'. In fact that is what we are already doing.

Is is on the level of local implementation, not the network. Of course you can implement it that way.

demilatof commented 1 year ago

The possible option is that you treat this re-appearing object as a new one and create a copy in your system. The other option is that you ask your partner to remove it permanently.

You make it easy... Anyway, everyone could implement the strategy that it would prefer in his/her internal system, but keep in mind that often the matching process between our copy and partner copy is really time expensive for the officer. Delete everything just for a mistake and then repeat all again would be an awful time waste. Much better keep everything and delete only after a "human" confirmation.

umesh-qs commented 1 year ago

If it is for local implementation as you and @janinamincer-daszkiewicz are saying, then why are we even discussing this DELETE topic. Whoever would have faced this problem should/would have already solved it by now. I would like to here from @kkaraogl on this as he was quite vocal on this DELETE problem

demilatof commented 1 year ago

I am with @demilatof on marking them 'unavailable'. In fact that is what we are already doing.

Is is on the level of local implementation, not the network. Of course you can implement it that way.

Could you explain why treating them as "unavailable" is considered as "on the level of local implementation" whilst treating them as disappeared or deleted should be on the network level? Strange point of view. Anyway a good project should consider not only what happens on network, but even address the consequence of a network communication. And it should suggest local implementation to facilitate network communication.

janinamincer-daszkiewicz commented 1 year ago

We are discussing to find the common ground for all and give recommendations. We want to make sure that everybody will interpret the situation in the same way. And will know what will be considered as an error and what as an acceptable behaviour.

Which does not mean that errors will magically disappear from the network.

I am not trying to say that such approach to deletion is the best one, may be it is could enough for now, when many developers are still struggling with more basic errors.

janinamincer-daszkiewicz commented 1 year ago

Because "unavailable" does not differ from deleted from the network perspective.

And it should suggest local implementation to facilitate network communication.

I agree.

umesh-qs commented 1 year ago

@janinamincer-daszkiewicz you are giving mixed responses. Are you still in favor of making changes for mandatory DELETE business requirement? If so then what is the proposal?

janinamincer-daszkiewicz commented 1 year ago

you are giving mixed responses.

I do not think so :)

If so then what is the proposal?

It is expressed here: https://github.com/erasmus-without-paper/ewp-specs-api-iias/issues/98#issuecomment-1381841481 https://github.com/erasmus-without-paper/ewp-specs-api-iias/issues/98#issuecomment-1382734251

umesh-qs commented 1 year ago

you are giving mixed responses.

I do not think so :)

If so then what is the proposal?

It is expressed here: #98 (comment) #98 (comment)

Great. Going forward we will assume anything coming from Michal has your blessing. So you are in favor that there is no change needed to solve the mandatory DELETE problem. When IIA response is empty then it might be hidden or it might be DELETED. And that DELETE is not permanent?

janinamincer-daszkiewicz commented 1 year ago

Great. Going forward we will assume anything coming from Michal has your blessing.

That would be overinterpretation ;)

So you are in favor that there is no change needed to solve the mandatory DELETE problem.

I accept the pragmatic approach (which does not mean that it is my favourite).

When IIA response is empty then it might be hidden or it might be DELETED. And that DELETE is not permanent?

No, it can only be interpreted as deleted. Otherwise there is an error. DELETE is permanent.

demilatof commented 1 year ago

Because "unavailable" does not differ from deleted from the network perspective.

I don't agree and I show you in a simple way with the HTTP Error codes:

410 Gone (that is, deleted) 451 Unavailable (For Legal Reasons)

The only thing that unavailable and deleted have in common is that the resource cannot be accessed over the network. But deleted represents a well known condition that allows an irreversible decision in the internal system and that therefore the network must certify without doubts, whilst unavailable is a generically condition and it is outside the network interest.

umesh-qs commented 1 year ago

Great. Going forward we will assume anything coming from Michal has your blessing.

That would be overinterpretation ;)

So you are in favor that there is no change needed to solve the mandatory DELETE problem.

I accept the pragmatic approach (which does not mean that it is my favourite).

When IIA response is empty then it might be hidden or it might be DELETED. And that DELETE is not permanent?

No, it can only be interpreted as deleted. Otherwise there is an error. DELETE is permanent.

After all the discussion here and in other threads, you still want to handle DELETE (permanent) in this way then I am sorry to say you are asking for a catastrophe at some point. Permanent DELETE means that the existing IIAs has to be discarded and new IIAs added if there was a mistake. Imagine a node that for some reason has an issue and for some time sends empty response for all the IIA GET request. By the time they realize this, hundreds of IIAs will be wiped out from the system(s). We are not in favor of permanent DELETE with the proposed solution. If you want permanent DELETE then there has to be a unambiguous response that confirms the DELETE. And this is our position since the beginning of this proposal.

demilatof commented 1 year ago

After all the discussion here and in other threads, you still want to handle DELETE (permanent) in this way then I am sorry to say you are asking for a catastrophe at some point. Permanent DELETE means that the existing IIAs has to be discarded and new IIAs added if there was a mistake. Imagine a node that for some reason has an issue and for some time sends empty response for all the IIA GET request. By the time they realize this, hundreds of IIAs will be wiped out from the system(s). We are not in favor of permanent DELETE with the proposed solution. If you want permanent DELETE then there has to be a unambiguous response that confirms the DELETE. And this is our position since the beginning of this proposal.

I agree with @umesh-qs, with the only difference that I'm not for a mandatory DELETE in the internal system and therefore I think that we don't need an unambiguous response that confirms the DELETE. I've enough experience to know that we are not in an ideal word, mistakes happen, I want to be free to resume the same IIA-ID I used before my partner made its IIA "unavailable" for whatever reason, if it realizes having made an error. In my scenario, when the officer keeps on reading "unavailable" for some days, he/she contacts the partner to solve the doubt. Our officer will instruct the internal system what to do with that IIA, not a weak EWP communication system.

umesh-qs commented 1 year ago

After all the discussion here and in other threads, you still want to handle DELETE (permanent) in this way then I am sorry to say you are asking for a catastrophe at some point. Permanent DELETE means that the existing IIAs has to be discarded and new IIAs added if there was a mistake. Imagine a node that for some reason has an issue and for some time sends empty response for all the IIA GET request. By the time they realize this, hundreds of IIAs will be wiped out from the system(s). We are not in favor of permanent DELETE with the proposed solution. If you want permanent DELETE then there has to be a unambiguous response that confirms the DELETE. And this is our position since the beginning of this proposal.

I agree with @umesh-qs, with the only difference that I'm not for a mandatory DELETE in the internal system and therefore I think that we don't need an unambiguous response that confirms the DELETE. I've enough experience to know that we are not in an ideal word, mistakes happen, I want to be free to resume the same IIA-ID I used before my partner made its IIA "unavailable" for whatever reason, if it realizes having made an error. In my scenario, when the officer keeps on reading "unavailable" for some days, he/she contacts the partner to solve the doubt. Our officer will instruct the internal system what to do with that IIA, not a weak EWP communication system.

@demilatof we are also not in favor of permanent DELETE. But if that is mandated by DG EAC then we need a better solution.

@mkurzydlowski do you agree with @janinamincer-daszkiewicz that empty response should be considered permanent DELETE?

demilatof commented 1 year ago

@demilatof we are also not in favor of permanent DELETE. But if that is mandated by DG EAC then we need a better solution.

@umesh-qs as I said you yesterday, in my opinion the BPO wants that no one could forbid us to delete an agreement (not approved) and no one could forbid us to notify the partner about the deletion. They are not saying that we must delete an IIA that wasn't approved and that we must notify it. Therefore they give freedom to delete or not in your environment and I think that EWP Specifications should not make mandatory the deletion in any case

umesh-qs commented 1 year ago

@demilatof we are also not in favor of permanent DELETE. But if that is mandated by DG EAC then we need a better solution.

@umesh-qs as I said you yesterday, in my opinion the BPO wants that no one could forbid us to delete an agreement (not approved) and no one could forbid us to notify the partner about the deletion. They are not saying that we must delete an IIA that wasn't approved and that we must notify it. Therefore they give freedom to delete or not in your environment and I think that EWP Specifications should not make mandatory the deletion in any case

@demilatof but that is not what @janinamincer-daszkiewicz thinks. As I already said we are not in favor of permanent delete. But if it is being enforced then we need a concrete solution.

demilatof commented 1 year ago

@demilatof but that is not what @janinamincer-daszkiewicz thinks. As I already said we are not in favor of permanent delete. But if it is being enforced then we need a concrete solution.

I understand; I would like to understand too why are we enforced. I hope not in order to solve the problem of a particular provider. If someone said something like "a lot of providers asked for it" I wish to know who they are for the simple reason to understand if they are only one or thousands.

Personally, I will not delete anything in my system. Never and ever. If they want to enforce us to do something in case of deletion, we might choose the following solution.

If an IIA will be no more useful we would remove the partner's IIA Id and even if the partners will send us an Approval-CNR we'll ignore them. Therefore, we'll have nothing to do as concern listing deleted items. Are they ready for approval? Formally yes Are we interested in approval? Not for them

mkurzydlowski commented 1 year ago

@demilatof, yes you could call such "delete" as making IIA "unavailable" in the EWP network.

I also agree that the partner would gain from not removing it's local copy just yet but rather showing it to the user as being "deleted" by the partner.

We need to be able to handle IIAs reappearing in the EWP network. That should also be properly communicated to the end user.

This is how things should be interpreted by the other side from the network perspective. There should probably be business requirements build on top of the network layer that mandate how such "deletes" should be used, to prevent people from deleting and recovering IIAs in cases other than communicating to the partner a true will of deleting an IIA.

umesh-qs commented 1 year ago

@demilatof, yes you could call such "delete" as making IIA "unavailable" in the EWP network.

I also agree that the partner would gain from not removing it's local copy just yet but rather showing it to the user as being "deleted" by the partner.

We need to be able to handle IIAs reappearing in the EWP network. That should also be properly communicated to the end user.

This is how things should be interpreted by the other side from the network perspective. There should probably be business requirements build on top of the network layer that mandate how such "deletes" should be used, to prevent people from deleting and recovering IIAs in cases other than communicating to the partner a true will of deleting an IIA.

@mkurzydlowski so do you agree that empty response should not be treated as permanent DELETE?

janinamincer-daszkiewicz commented 1 year ago

empty response should not be treated as permanent DELETE?

Please explain what you mean by that.

umesh-qs commented 1 year ago

empty response should not be treated as permanent DELETE?

Please explain what you mean by that.

I think it is pretty clear. There is enough discussion on this. Cannot go over again and again. Will let others respond to this topic now.

demilatof commented 1 year ago

We need to be able to handle IIAs reappearing in the EWP network. That should also be properly communicated to the end user.

Why we should do anything? When an IIA reappears, it will be available as always:

If it was changed, or if we realize that the partner needs to be warned, we might send an IIA-CNR.

umesh-qs commented 1 year ago

@demilatof, yes you could call such "delete" as making IIA "unavailable" in the EWP network.

I also agree that the partner would gain from not removing it's local copy just yet but rather showing it to the user as being "deleted" by the partner.

We need to be able to handle IIAs reappearing in the EWP network. That should also be properly communicated to the end user.

This is how things should be interpreted by the other side from the network perspective. There should probably be business requirements build on top of the network layer that mandate how such "deletes" should be used, to prevent people from deleting and recovering IIAs in cases other than communicating to the partner a true will of deleting an IIA.

@mkurzydlowski you are contradicting yourself. You are asking for empty response to be treated as permanent delete and on the same time you agree with them re-appearing with proper communication to the end user.

mkurzydlowski commented 1 year ago

I don't see a contradiction here. What I meant is that an IIA might reappear if the partner changes his mind (probably due to a mistake on his part). You could probably call it non-permanent because of that.

But still business people might want to add additional requirements on the workflow, so people don't misuse this "delete".

umesh-qs commented 1 year ago

I don't see a contradiction here. What I meant is that an IIA might reappear if the partner changes his mind (probably due to a mistake on his part). You could probably call it non-permanent because of that.

@mkurzydlowski As soon as you make it re-appear it is not permanent. You may choose to deny it but that does change the interpretation for others.

But still business people might want to add additional requirements on the workflow, so people don't misuse this "delete".

@mkurzydlowski I am not sure where is the misuse here. If it is empty response then you show it 'unavailable' and if it re-appears start working on it. Why are you insisting on calling it deleted? It will create more confusion.

mkurzydlowski commented 1 year ago

I'am calling it delete. Hope the quotes weren't misleading. I'm also agreeing that it is not permanent in the sense that the decision might be reverted. But not in the sense that the partner wasn't intending on deleting the IIA permanently at the moment.

janinamincer-daszkiewicz commented 1 year ago

After Infrastructure Forum meeting on 2023-01-18

Waiting for guidelines from DG EAC/BPOs

janinamincer-daszkiewicz commented 1 year ago

Resolved in https://github.com/erasmus-without-paper/ewp-specs-api-iias/issues/98

umesh-qs commented 1 year ago

Resolved in #98

@janinamincer-daszkiewicz I did not get what you mean by this. Can you please share the details on how it is resolved?

janinamincer-daszkiewicz commented 1 year ago

I cite from https://github.com/erasmus-without-paper/ewp-specs-api-iias/issues/98: "according to recommendations of DG EAC, this set of changes to the family of IIA APIs will be issued as the major stable release. "

This was the last pending issue concerning Delete.

umesh-qs commented 1 year ago

@janinamincer-daszkiewicz I am still not clear. Will it be allowed or not?

janinamincer-daszkiewicz commented 1 year ago

Deletion is permanent, temporary hiding is not allowed, major number will change.