How to calculate statistics for IIAs?

[janinamincer-daszkiewicz]

During IF on 2023-01-18 doubts were raised on which IIAs should be counted in iia-fetchable.

According to https://github.com/erasmus-without-paper/ewp-specs-api-iias/blob/stable-v6/endpoints/stats-response.xsd iia-fetchable is:

Number of IIAs created in the local system which can be fetched by the partner HEI.

This number should be consistent with what the node exposes in IIA get and IIA index. In particular it should not depend on where the partner is located.

By 'created in the local system' we also mean those which have been created in response to IIA CNR sent by the partner.

The way the IIA has been created does not matter.

The number can only go down after deletion.

Dashboard counts IIAs signed between Dashboard users, I hope that other providers do the same. The number of IIAs should not depend on the installation (the same hub of this provider or a different hub).

Last but not least EWP network is for Erasmus+ programme, we consider IIAs signed under this programme.

P.S. The old issue is at https://github.com/erasmus-without-paper/ewp-specs-api-iias/issues/82.

[demilatof]

I think that all depends from the question: "what do we want to measure with statistics for IIA?" For example, if we are interested in how much a provider is efficient in EWP implementation, we should collect only IIAs that can be fetched and approved invoking an API according to EWP Network specification. That is, if Dashboard uses an internal system to create, exchange and approve IIA between two Dashboard users, these IIAs should not be counted in statistics. The same for other (third part) providers that use the same approach for their users. After that we should have two different statistics:

1. One to consider all the IIA exchanged calling EWP APIs;
2. Another one to consider only the IIA exchanged calling EWP APIs BUT from a provider to a different one

Why we need the second type of statistic? Because I think that a provider that implements both the client and the server could transmit/receive an IIA/CNR/Approval successfully even if it makes some errors from the EWP Network perspective.

[milsorm]

I didn't expect that IIA between two HEI can be exchanged through some internal process like in Dashboard. I still expect that valid IIA is "broadcasted" to the network a IIA is approved if both party is broadcasting copies with same hash. How dashboard broadcast internally discussed IIA? Is this done only during the process or also for approved ones?

[janinamincer-daszkiewicz]

"what do we want to measure with statistics for IIA?"

We want to measure data from business perspective.

[demilatof]

@janinamincer-daszkiewicz

We want to measure data from business perspective.

Could you be more specific? Do you mean that you are more interested in the number of IIAs than in the way they are exchanged?

[janinamincer-daszkiewicz]

Yes, DG EAC is interested in the number of IIAs signed by HEIs participating in the Erasmus+, using the EWP network. By 'using the network' I mean that they expose APIs in the EWP Network and can exchanged data with the partners, from the same provider or from other providers.

[umesh-qs]

Yes, DG EAC is interested in the number of IIAs signed by HEIs participating in the Erasmus+, using the EWP network. By 'using the network' I mean that they expose APIs in the EWP Network and can exchanged data with the partners, from the same provider or from other providers.

So only an IIA that is explicitly signed by HEI should be included in the IIA stats?

[janinamincer-daszkiewicz]

I answered this question in https://github.com/erasmus-without-paper/ewp-specs-api-iias/issues/102#issue-1551756573.

[umesh-qs]

I answered this question in #102 (comment).

@kkaraogl can you please confirm if only IIAs explicitly signed by HEIs are included in the IIA stats.

[demilatof]

Yes, DG EAC is interested in the number of IIAs signed by HEIs participating in the Erasmus+, using the EWP network. By 'using the network' I mean that they expose APIs in the EWP Network and can exchanged data with the partners, from the same provider or from other providers.

@janinamincer-daszkiewicz I thank you for your explanation. Do you know if Dashboard uses the EWP APIs to exchange and to approve an IIA between a Dashboard user and another Dashboard user? Or it has an internal database and bypass the APIs?

Moreover, an IIA signed between two partners of the same provider (Dashboard or not) could be a valid signed IIA but with a very bad implementation of the APIs because the provider has coded wrong methods both in client and server; and it could even fail as soon as a partner tries to use the APIs of a different provider.

I understand that DG EAC is interested in the number of IIAs signed by HEIs participating in the Erasmus+, but I think they should also interested in monitoring the good implementation of EWP APIs.

[janinamincer-daszkiewicz]

All IIAs which are fetchable via IIA index should be counted.

[janinamincer-daszkiewicz]

Dashboard team can answer questions about Dashboard implementation.

Statistics are not for monitoring errors. For that we build monitoring functionality.

[umesh-qs]

All IIAs which are fetchable via IIA index should be counted.

I thought you earlier said that only the IIA that is explicitly signed by HEI and fetchable via IIA index should be counted.

[janinamincer-daszkiewicz]

I have not used the phrase 'explicitly signed' in any of my posts.

[umesh-qs]

I have not used the phrase 'explicitly signed' in any of my posts.

So you are saying that there is a possibility that and HEI has signed an IIA but still it may not be explicit? Can you please give a practical example of such scenario?

[janinamincer-daszkiewicz]

No, I am not saying anything like that. In fact I do not know what you mean by 'explicitly signed'.

[demilatof]

Statistics are not for monitoring errors. For that we build monitoring functionality.

I'm not interested in monitoring errors, but I think we need to know at what point we are. That is, when my HEI ask me for the status of the release because Dashboard (for example) has amazing statistics whilst we have still nothing, I wish to know if we are really later or simply that statistics are boosted by implementations that nothing has to do with EWP Network. This is also very important for the timetable we all receive.

[umesh-qs]

No, I am not saying anything like that. In fact I do not know what you mean by 'explicitly signed'.

It means that the person who has signed an agreement has seen it and done a manual action on it to be considered as signed. I hope it is clear now.

[demilatof]

It means that the person who has signed an agreement has seen it and done a manual action on it to be considered as signed. I hope it is clear now.

Do you think we may have providers that have implemented an approval process in blinded fashion? I hope not, but I didn't investigate and I would not be surprised if it was

[umesh-qs]

It means that the person who has signed an agreement has seen it and done a manual action on it to be considered as signed. I hope it is clear now.

Do you think we may have providers that have implemented an approval process in blinded fashion? I hope not, but I didn't investigate and I would not be surprised if it was

@demilatof approval and signing are different processes.

[demilatof]

@demilatof approval and signing are different processes.

Now I realize, but in EWP when we have signing? I thought you was talking about approval because the signing should be outside the EWP.

[umesh-qs]

Dashboard counts IIAs signed between Dashboard users, I hope that other providers do the same. The number of IIAs should not depend on the installation (the same hub of this provider or a different hub).

@demilatof so "signed" here means approved here? And that Dashboard is only sharing approved IIAs?

[demilatof]

@demilatof so "signed" here means approved here? And that Dashboard is only sharing approved IIAs?

I think you should ask @janinamincer-daszkiewicz or @kkaraogl When I read "Dashboard counts IIAs signed between Dashboard users" I interpreted it keeping in mind the graph at https://stats.erasmuswithoutpaper.eu/chart where the green bar represents "approved by both sides". But maybe I'm wrong and they meant something different

[umesh-qs]

@demilatof so "signed" here means approved here? And that Dashboard is only sharing approved IIAs?

I think you should ask @janinamincer-daszkiewicz or @kkaraogl When I read "Dashboard counts IIAs signed between Dashboard users" I interpreted it keeping in mind the graph at https://stats.erasmuswithoutpaper.eu/chart where the green bar represents "approved by both sides". But maybe I'm wrong and they meant something different

"green bar" might represent that both partners have approved. But I am not sure what this means "Dashboard counts IIAs signed between Dashboard users". Are all IIAs shared by Dashboard signed by at least one party?

[demilatof]

"green bar" might represent that both partners have approved. But I am not sure what this means "Dashboard counts IIAs signed between Dashboard users". Are all IIAs shared by Dashboard signed by at least one party?

I don't know what Dashboard does. I even don't know if all of us are signing our IIAs in the same way. In the old paper world the IIA signed was a single document shared by both parties. Now we have a master-master model where the signing gets a meaning only after both of the copies are approved.

[janinamincer-daszkiewicz]

Dashboard counts IIAs signed between Dashboard users, I hope that other providers do the same.

I see that the word 'signed' may be misinterpreted because of the connotation with the specification. I will rephrase the cited sentence.

Dashboard counts IIAs negotiated (processed, handled) between Dashboard users, I hope that other providers do the same.

[milsorm]

Dashboard counts IIAs negotiated (processed, handled) between Dashboard users, I hope that other providers do the same.

We do the same. It is expectable behaviour.

[demilatof]

We do the same. It is expectable behaviour.

Do you do that with a distributed system, where every partner has its own software and database, or do you have a centralized system that gives access to each partner by means of a web based client? The IIAs among your users are negotiated using EWP APIs, or do you use them only with external partners?

[demilatof]

Dashboard counts IIAs negotiated (processed, handled) between Dashboard users, I hope that other providers do the same.

Negotiated how? May be you and DG EAC are not interested in, but from my point of view, this makes the difference to evaluate EWP from a network perspective.

[janinamincer-daszkiewicz]

The statistics we gather have been discussed during the infrastructure forum meetings for a long time, have been approved by DG EAC, have been implemented slowly by first gathering statistics manually, and now have been implemented by endpoints.

The question was how to interpet specs, the answer has been given. Are there still any pending questions?

[demilatof]

The question was how to interpet specs, the answer has been given. Are there still any pending questions?

Just one question: are the statistics mandatory? Because if this is your position and there is no space to change or improve them, in my opinion they are useless and don't give any real information about EWP, but are only a way to emphasize the EWP project accepting data that have nothing to do with the EWP implementation. We have a lot of IIAs exchanged with partners, we are implementing EWP APIs, but no IIA is yet exchanged via EWP. Should we communicate these numbers just to raise the statistics?

[janinamincer-daszkiewicz]

Please read "Note to EWP users on EWP statistics", a document from DG EAC located in the shared drive of IF.

We have a lot of IIAs exchanged with partners, we are implementing EWP APIs, but no IIA is yet exchanged via EWP.

This should not be reported, they are not exposed to the network via IIA index.

[milsorm]

Do you do that with a distributed system, where every partner has its own software and database, or do you have a centralized system that gives access to each partner by means of a web based client? The IIAs among your users are negotiated using EWP APIs, or do you use them only with external partners?

We already use IIA through EWP API between clients event if they are only tenants in same system (it is easier because we have both type of installations and they can be splitted/merged). So we use EWP API not only with external partners but also between our clients themselves.

[demilatof]

We already use IIA through EWP API between clients event if they are only tenants in same system (it is easier because we have both type of installations and they can be splitted/merged). So we use EWP API not only with external partners but also between our clients themselves.

I think this is the right implementation

[skishk]

We already use IIA through EWP API between clients event if they are only tenants in same system (it is easier because we have both type of installations and they can be splitted/merged). So we use EWP API not only with external partners but also between our clients themselves.

Good @milsorm !!! I agree too that is the good implementation, because you can be sure that if it works between your internal partners so it probably works fine with other external partners! I guess so you use 2 agreement between 2 partners and not 1 agreement shared between your internal partners, right? :relaxed:

[milsorm]

Good @milsorm !!! I agree too that is the good implementation, because you can be sure that if it works between your internal partners so it probably works fine with other external partners! I guess so you use 2 agreement between 2 partners and not 1 agreement shared between your internal partners, right? ☺️

Two copies of the same agreement - one for each partner. We store one copy in one tenant.