search

erasmus-without-paper / ewp-specs-api-iias

Specifications of EWP's Interinstitutional Agreements API.
MIT License
4 stars 13 forks source link

While transforming XML, empty tags should be consider or not ? #170

Open anant6767 opened 5 months ago

anant6767 commented 5 months ago

When transforming XML, if an empty tag is present, it is being included in the generation of the text-to-hash value. If one partner (Partner A) includes an empty tag in their transformation process and another partner (Partner B) does not, this discrepancy can lead to different text-to-hash values being generated, potentially causing synchronization issues within the network. It is essential to standardize the transformation approach to ensure consistency in the text-to-hash values generated by.

For example Partner A cooperation-conditions xml

<ia6:cooperation-conditions xmlns:ia6="https://github.com/erasmus-without-paper/ewp-specs-api-iias/blob/stable-v6/endpoints/get-response.xsd" xmlns:ewp="https://github.com/erasmus-without-paper/ewp-specs-architecture/blob/stable-v1/common-types.xsd" xmlns:c="https://github.com/erasmus-without-paper/ewp-specs-types-contact/tree/stable-v1" xmlns:trm="https://github.com/erasmus-without-paper/ewp-specs-types-academic-term/tree/stable-v1" xmlns:p="https://github.com/erasmus-without-paper/ewp-specs-types-phonenumber/tree/stable-v1" xmlns:a="https://github.com/erasmus-without-paper/ewp-specs-types-address/tree/stable-v1">
    <ia6:student-studies-mobility-spec>
        <ia6:sending-hei-id>ewp100-bo.staging.moveon4.com</ia6:sending-hei-id>
        <ia6:receiving-academic-year-id>2027/2028</ia6:receiving-academic-year-id>
        <ia6:receiving-academic-year-id>2028/2029</ia6:receiving-academic-year-id>
        <ia6:mobilities-per-year>12</ia6:mobilities-per-year>
        <ia6:recommended-language-skill>
            <ia6:language>ay</ia6:language>
            <ia6:cefr-level>A1</ia6:cefr-level>
        </ia6:recommended-language-skill>
        <ia6:recommended-language-skill>
            <ia6:language>eu</ia6:language>
            <ia6:cefr-level>A2</ia6:cefr-level>
        </ia6:recommended-language-skill>
        <ia6:subject-area>
            <ia6:isced-f-code>0110</ia6:isced-f-code>
            <ia6:isced-clarification>isced 001</ia6:isced-clarification>
        </ia6:subject-area>
        <ia6:subject-area>
            <ia6:isced-f-code>0112</ia6:isced-f-code>
            <ia6:isced-clarification/>
        </ia6:subject-area>
        <ia6:other-info-terms>new additonal values to be checkecd</ia6:other-info-terms>
        <ia6:total-months-per-year>33</ia6:total-months-per-year>
        <ia6:blended>true</ia6:blended>
        <ia6:eqf-level>4</ia6:eqf-level>
        <ia6:eqf-level>5</ia6:eqf-level>
        <ia6:eqf-level>6</ia6:eqf-level>
        <ia6:eqf-level>8</ia6:eqf-level>
    </ia6:student-studies-mobility-spec>
</ia6:cooperation-conditions>

Partner B cooperation-conditions xml 

<ia6:cooperation-conditions xmlns:ia6="https://github.com/erasmus-without-paper/ewp-specs-api-iias/blob/stable-v6/endpoints/get-response.xsd" xmlns:ewp="https://github.com/erasmus-without-paper/ewp-specs-architecture/blob/stable-v1/common-types.xsd" xmlns:c="https://github.com/erasmus-without-paper/ewp-specs-types-contact/tree/stable-v1" xmlns:trm="https://github.com/erasmus-without-paper/ewp-specs-types-academic-term/tree/stable-v1" xmlns:p="https://github.com/erasmus-without-paper/ewp-specs-types-phonenumber/tree/stable-v1" xmlns:a="https://github.com/erasmus-without-paper/ewp-specs-types-address/tree/stable-v1">
    <ia6:student-studies-mobility-spec>
        <ia6:sending-hei-id>ewp100-bo.staging.moveon4.com</ia6:sending-hei-id>
        <ia6:receiving-academic-year-id>2027/2028</ia6:receiving-academic-year-id>
        <ia6:receiving-academic-year-id>2028/2029</ia6:receiving-academic-year-id>
        <ia6:mobilities-per-year>12</ia6:mobilities-per-year>
        <ia6:recommended-language-skill>
            <ia6:language>ay</ia6:language>
            <ia6:cefr-level>A1</ia6:cefr-level>
        </ia6:recommended-language-skill>
        <ia6:recommended-language-skill>
            <ia6:language>eu</ia6:language>
            <ia6:cefr-level>A2</ia6:cefr-level>
        </ia6:recommended-language-skill>
        <ia6:subject-area>
            <ia6:isced-f-code>0110</ia6:isced-f-code>
            <ia6:isced-clarification>isced 001</ia6:isced-clarification>
        </ia6:subject-area>
        <ia6:subject-area>
            <ia6:isced-f-code>0112</ia6:isced-f-code>
        </ia6:subject-area>
        <ia6:other-info-terms>new additonal values to be checkecd</ia6:other-info-terms>
        <ia6:total-months-per-year>33</ia6:total-months-per-year>
        <ia6:blended>true</ia6:blended>
        <ia6:eqf-level>4</ia6:eqf-level>
        <ia6:eqf-level>5</ia6:eqf-level>
        <ia6:eqf-level>6</ia6:eqf-level>
        <ia6:eqf-level>8</ia6:eqf-level>
    </ia6:student-studies-mobility-spec>
</ia6:cooperation-conditions>

I suggest that in both the transform_version_6.xsl and transform_version_7.xsl stylesheets, a conditional check should be implemented to prevent empty tags from being factored into the text-to-hash generation process. This will help ensure consistent text-to-hash values across different partners, regardless of the inclusion of empty tags.

mkurzydlowski commented 5 months ago

You cannot require partner IIA hash to be equal to your own. The problem you describe is not an issue for this reason.

anant6767 commented 5 months ago

To ensure consistency during migration from v6 to v7 of IIAs, especially when the receiving partner is not preserving the original raw XML but storing the data in a different format, it is advisable to introduce a safeguard in the XSD transformation files. This safeguard would remove any empty tags during the transformation process. Incorporating this check will help in avoiding discrepancies that may arise from including empty tags.

mkurzydlowski commented 5 months ago

The receiving partner must store only the original raw XML! That is why the presence of empty tags does not introduce problems.

umesh-qs commented 5 months ago

@mkurzydlowski I guess this question was more in terms of future proofing the hash calculation for xml with empty tag. Is it too complicated to implement or there are any other concerns?

mkurzydlowski commented 5 months ago

I would rather suggest resolving the cause of the problem. It's not a good idea to calculate the same hash for different XML-s as it is also not a good idea to allow empty elements in the XML. I suggest to disallow such elements.

umesh-qs commented 5 months ago

@mkurzydlowski how do we disallow such element? And when do we disallow this? Is it in v7 or later in v8? If we are not doing this in v7 then any provider who wants to match hash (identical copy), might not be able to do that and such IIAs might will get stuck.

mkurzydlowski commented 5 months ago

We would do this by adding a note to the field description. I'm not able to answer the "when".

The are more important reasons for the hash not to match. See iia-id order, lists element order, xs:boolean ambiguities.

I still would suggest treating empty elements as not compliant in the updates specs, rather than purposely ignoring them in the hash calculation.

umesh-qs commented 5 months ago

@mkurzydlowski answer to when it important here. Suggested change is not backward compatible as it changes the hash. And whenever we do this, the transformation xsd will have to take care xml's that has empty elements until now.

You are right that hash should not be matched. But seeing the presentation in last IF call, EWP implementers are moving towards that direction. And unfortunately no opportunity was given to discuss the reasons for allowing such approach.

janinamincer-daszkiewicz commented 5 months ago

You are right that hash should not be matched. But seeing the presentation in last IF call, EWP implementers are moving towards that direction. And unfortunately no opportunity was given to discuss the reasons for allowing such approach.

We cannot base our solutions on matching hashes until we know the way to do it. Honestly, we do not see it yet. But this is work in progress and any help is welcome.

umesh-qs commented 5 months ago

You are right that hash should not be matched. But seeing the presentation in last IF call, EWP implementers are moving towards that direction. And unfortunately no opportunity was given to discuss the reasons for allowing such approach.

We cannot base our solutions on matching hashes until we know the way to do it. Honestly, we do not see it yet. But this is work in progress and any help is welcome.

We can allow to base our solutions to match the IIA content and not IIA hash? Why this selective approach? And who knows that when and important provider starts doing this, this approach will also sound ok.

janinamincer-daszkiewicz commented 5 months ago

We can allow to base our solutions to match the IIA content and not IIA hash?

What I am saying is that comparing hashes does not allows us to conclude if IIAs are the same. Personally I would like to find the way to compare two copies of IIA and conclude that they are the same just to be able to show this information to my users in the interface.

umesh-qs commented 5 months ago

We can allow to base our solutions to match the IIA content and not IIA hash?

What I am saying is that comparing hashes does not allows us to conclude if IIAs are the same. Personally I would like to find the way to compare two copies of IIA and conclude that they are the same just to be able to show this information to my users in the interface.

Currently hashes will not match. But probably changes in the tranfomation xsd to calculate hash, can be done to achieve this. My point is why not do changes to match hashes, when it is already allowed for some provider to stall the process (force IIA matching) if content is not matching.

mkurzydlowski commented 5 months ago

Would this changes to the field descriptions resolve this issue: https://github.com/erasmus-without-paper/ewp-specs-api-iias/compare/empty-field

janinamincer-daszkiewicz commented 5 months ago

Currently hashes will not match. But probably changes in the tranfomation xsd to calculate hash, can be done to achieve this.

We do not know such method yet, as I have said. For the time being we are not very optimistic on resolving this issue. But we will keep trying.

umesh-qs commented 5 months ago

Would this changes to the field descriptions resolve this issue: https://github.com/erasmus-without-paper/ewp-specs-api-iias/compare/empty-field

No. Adding this in v7 means that individual provider must make changes just before the release.

umesh-qs commented 5 months ago

Currently hashes will not match. But probably changes in the tranfomation xsd to calculate hash, can be done to achieve this.

We do not know such method yet, as I have said. For the time being we are not very optimistic on resolving this issue. But we will keep trying.

Ok so what I understand is we are yet to find a way to match the hash. But for now business requirement is that the content of IIAs must match and there is no need/scenario for the content (under cooperation condition) in 2 mutually approved copies to be different?

janinamincer-daszkiewicz commented 5 months ago

Ok so what I understand is we are yet to find a way to match the hash.

Yes. If anybody knows how to do that, please share.

umesh-qs commented 5 months ago

Ok so what I understand is we are yet to find a way to match the hash.

Yes. If anybody knows how to do that, please share.

But then this is totally different from what @mkurzydlowski is suggesting/thinking. There is no need to specify removal of empty tags if that can be handled by xsd transformation. We must make use of xsd transformation as far as possible.

janinamincer-daszkiewicz commented 5 months ago

But then this is totally different from what @mkurzydlowski is suggesting/thinking. There is no need to specify removal of empty tags if that can be handled by xsd transformation. We must make use of xsd transformation as far as possible.

That's not true and I would say that these are two separate issues. Matching hashes is one. empty tags is another.

Now about empty tags. As Michał explained the only proper way to solve this issue is to disallow empty tags. They are not needed, shouldn't be used. You answered "individual provider must make changes just before the release." I understand the QS MoveOn is that provider. OK, important argument. But what you propose instead is to change XSLT and make ALL providers do the change just before the release. Then anybody else will use the same argument. I do not think this is the best solution of the problem.

umesh-qs commented 5 months ago

But then this is totally different from what @mkurzydlowski is suggesting/thinking. There is no need to specify removal of empty tags if that can be handled by xsd transformation. We must make use of xsd transformation as far as possible.

That's not true and I would say that these are two separate issues. Matching hashes is one. empty tags is another.

Now about empty tags. As Michał explained the only proper way to solve this issue is to disallow empty tags. They are not needed, shouldn't be used. You answered "individual provider must make changes just before the release." I understand the QS MoveOn is that provider. OK, important argument. But what you propose instead is to change XSLT and make ALL providers do the change just before the release. Then anybody else will use the same argument. I do not think this is the best solution of the problem.

Unfortunately we are again thinking short term and picking individual provider. Our system does not have any issues with empty tags. We are and have always been compliant with the specifications. I am not sure where you got this impression. We are suggesting these changes since there are providers needing identical IIAs and moving towards identical hash is the best way to support that. It is up to DG EAC to take these suggests for long term benefits or discard these for short term justifications. I would leave this discussion here, to avoid any friction. This topic can be closed if DG EAC doesn't finds any merit. Thanks

janinamincer-daszkiewicz commented 5 months ago

I am not sure where you got this impression.

Apologies if I misinterpreted your previous post. I assume that nobody will be happy with any changes in the coming days. As for the long term, we can prepare both solutions:

  1. Changes prepared by Michał in https://github.com/erasmus-without-paper/ewp-specs-api-iias/compare/empty-field
  2. Changes in XSLT to remove empty tags (just to be on a safe side).

Providers should consider them as recommendations which will some day be introduced to the specification.

P.S. Thank you for raising this issue.

umesh-qs commented 5 months ago

I spent a couple of hours and came up with the attached xslt. This may not be complete as not tested extensively, but takes care of decimal values, empty tags and sorting with tag names and values with my simple test. We should work in this direction my_stylesheet_v7.txt

I can help make it closer to reality if someone can test and find issues.

janinamincer-daszkiewicz commented 5 months ago

Thank you Umesh, Michał is also working on that, let's join forces. He will continue on Monday.

demilatof commented 5 months ago

I spent a couple of hours and came up with the attached xslt. This may not be complete as not tested extensively, but takes care of decimal values, empty tags and sorting with tag names and values with my simple test. We should work in this direction my_stylesheet_v7.txt

I can help make it closer to reality if someone can test and find issues.

I really appreciate your effort, but I think there is no more time to make a change in the XSLT, this is the last week before version 7 and any change can bring errors without solve anything. And after the next week any change in the XSLT will make the IIA Hash incompatible again, voiding the approvals

All that said, with the current XSDs we cannot have two identical IIA hashes, a simple space in the clarification makes them different, a different order in a single mobility makes them different, two identical mobility inverted make them different and so on... Without a mobility ID, you cannot order them, but to this scope we have to add it and declare immutable; we can study a way to order inside the XSLT but, again, the clarification field can make the IIA hashes different. The same IIA IDs are in different order in my and your IIA xml document.

So, why should we over complicate everything without coming to a solution? Currently, we have to be sure that the IIA hash code that YOU compute for YOUR xml is the same I can compute for YOUR xml, nothing more. And this is what we have done for years.

To have two identical hashes we have to change our approach to the IIA structure and exchanges, with a deep analysis of the process, not introducing weak solutions a week before the deadline. I tried to make an analysis and introducing a solution almost a year ago, but it was not much considered. Now it's late for version 7; we can talk about in the next months.

umesh-qs commented 5 months ago

@demilatof no changes will be done for v7. This is something for future version. Please note that it is already an official business requirement that IIA content must match before approving. So why not make efforts to match the hash by overcoming the challenges that you have listed, using the transformation xslt.

demilatof commented 5 months ago

@demilatof no changes will be done for v7.

Good to know

This is something for future version. Please note that it is already an official business requirement that IIA content must match before approving.

Could you show me the above requirement? I'm not sure that exists this requirement in the terms you mean; please look here stessoContenutoIIA

We need to have the same number of mobilities and, possibly, the same fields, but not the same content

So why not make efforts to match the hash by overcoming the challenges that you have listed, using the transformation xslt.

Because you cannot have a definitive solution, the problem is not to check the two IIAs, the problem is the freedom given by the master master model of having two IIA slightly different. If we want two identical IIAs, and therefore two identical hashes, the way is changing the IIA structure and xml assembling, understanding that it is only problematic the freedom we have to write outgoing mobilities when the only one who can state the rules for his HEI is the partner (with his incoming mobilities).

In my opinion, a solution that facilitates even the approval is to split the cooperation-conditions in two parts:

  1. Incoming mobilities (written by the owner of the copy of the IIA; this fragment of XML is generated)
  2. Outgoing mobilities (incorporated as they are from the partner's copy of the IIA; this fragment of XML is inherited)

Doing so, if we don't generate our incoming mobilities in a different order from time to time, we can always have two identical IIAs when we accept partner's incoming mobilities incorporating them in our IIA. And this allows even an automatism in mutual approval (I can describe the process in the next weeks, now it's not urgent)

umesh-qs commented 5 months ago

@demilatof no changes will be done for v7.

Good to know

This is something for future version. Please note that it is already an official business requirement that IIA content must match before approving.

Could you show me the above requirement?

@demilatof please see https://github.com/erasmus-without-paper/ewp-specs-api-iias/issues/170#issuecomment-2014880505 This was also mention in the recent IF call. Yes currently specifications do not specify that but and that is why it is not enforced for v7.

I'm not sure that exists this requirement in the terms you mean; please look here stessoContenutoIIA

We need to have the same number of mobilities and, possibly, the same fields, but not the same content

So why not make efforts to match the hash by overcoming the challenges that you have listed, using the transformation xslt.

Because you cannot have a definitive solution, the problem is not to check the two IIAs, the problem is the freedom given by the master master model of having two IIA slightly different. If we want two identical IIAs, and therefore two identical hashes, the way is changing the IIA structure and xml assembling, understanding that it is only problematic the freedom we have to write outgoing mobilities when the only one who can state the rules for his HEI is the partner (with his incoming mobilities).

@demilatof my suggestion is guided by business requirement. If content is allowed to be different then there is no way hash can be matched.

In my opinion, a solution that facilitates even the approval is to split the cooperation-conditions in two parts:

  1. Incoming mobilities (written by the owner of the copy of the IIA; this fragment of XML is generated)
  2. Outgoing mobilities (incorporated as they are from the partner's copy of the IIA; this fragment of XML is inherited)

Doing so, if we don't generate our incoming mobilities in a different order from time to time, we can always have two identical IIAs when we accept partner's incoming mobilities incorporating them in our IIA. And this allows even an automatism in mutual approval (I can describe the process in the next weeks, now it's not urgent)

@demilatof if as per business requirement contents must match then easier solution would be to make use of xslt to sort and generate the same hash.

demilatof commented 5 months ago

This is something for future version. Please note that it is already an official business requirement that IIA content must match before approving.

Could you show me the above requirement?

@demilatof please see #170 (comment) This was also mention in the recent IF call. Yes currently specifications do not specify that but and that is why it is not enforced for v7.

As a matter of fact, from the last IF I understood exactly the opposite: the two IIAs don't need to be identical, but we cannot recall this aspect in the current specifications. That is, in future it will be specified that is forbidden to require that partner's IIA is identical to ours.

@demilatof my suggestion is guided by business requirement. If content is allowed to be different then there is no way hash can be matched.

They cannot impose to write identical content, it's easier to win the lottery. There are marginal aspects that a partner could not know, and it would be time consuming to stop the approval until they have collected all, exactly till the last comma, from the partner. With the possibility that seconds later, the partner decides to change a space in the clarification field, for example.

The xslt could not face this problem; obviously...

@demilatof if as per business requirement contents must match then easier solution would be to make use of xslt to sort and generate the same hash.

If as per business requirement contents must be identical, it's easier to edit only our incoming mobilities and incorporate the partner's incoming mobilities as our outgoing mobilities, there is no way that they could be different. And the XSLT remains lighter

umesh-qs commented 5 months ago

@demilatof I would let Janina or DG EAC confirm what will be mentioned in the specs in future. But my feeling is that your wish for this point "That is, in future it will be specified that is forbidden to require that partner's IIA is identical to ours" might never be fulfilled.

If in future it is mentioned that contents (contractual values) need to be same, namespace, order etc can vary. What issues do you see matching the hash?

demilatof commented 5 months ago

If in future it is mentioned that contents (contractual values) need to be same, namespace, order etc can vary. What issues do you see matching the hash?

Easy, if they must be the same as you describe above, the actual XSLT is enough. If we need to modify the XSLT to make them appear the same, even if they aren't, it means that they are not the same, as instead they requested. If we conclude that they could be almost equal, we fall again in the problem that every XSLT we will try do implement would fail managing values that are in the free form (as text, for example)

umesh-qs commented 5 months ago

If in future it is mentioned that contents (contractual values) need to be same, namespace, order etc can vary. What issues do you see matching the hash?

Easy, if they must be the same as you describe above, the actual XSLT is enough.

@demilatof only content must be same. xml might not be identical. Formatting can differ. In that case existing XSLT is not enough.

If we need to modify the XSLT to make them appear the same, even if they aren't, it means that they are not the same, as instead they requested. If we conclude that they could be almost equal, we fall again in the problem that every XSLT we will try do implement would fail managing values that are in the free form (as text, for example)

@demilatof I ask again, why can't xslt transform the same content in different order/format to generate identical hash. Can you provide examples of 2 xmls with same content (of cooperation condition) but different order/formatting that xslt transformation cannot produce identical hash?

demilatof commented 5 months ago

Easy, if they must be the same as you describe above, the actual XSLT is enough.

@demilatof only content must be same. xml might not be identical. Formatting can differ. In that case existing XSLT is not enough.

I've misread your sentence: "need to be same, namespace, order etc can vary. "; I've missed "can vary", so that they seemed a list of immutable things.

The XSLT may help, but it cannot intrude the free text, e.g. clarification:

<isced-clarification>For law student</isced-clarification>

should not be different from

<isced-clarification>For student in law</isced-clarification>

but for XSLT it is. I think that the idea of completely identical IIAs, brings no way; anyway, if they will decide so, I think that there will be a strong opposition by most officers. At the end, we will see how to deal, if it will need. Now it is useless discussing future scenarios.

umesh-qs commented 5 months ago

Easy, if they must be the same as you describe above, the actual XSLT is enough.

@demilatof only content must be same. xml might not be identical. Formatting can differ. In that case existing XSLT is not enough.

I've misread your sentence: "need to be same, namespace, order etc can vary. "; I've missed "can vary", so that they seemed a list of immutable things.

The XSLT may help, but it cannot intrude the free text, e.g. clarification:

<isced-clarification>For law student</isced-clarification>

should not be different from

<isced-clarification>For student in law</isced-clarification>

@demilatof content is not identical here. I assume that you are in agreement that xslt can be used generated identical hash, if content is same?

but for XSLT it is. I think that the idea of completely identical IIAs, brings no way; anyway, if they will decide so, I think that there will be a strong opposition by most officers. At the end, we will see how to deal, if it will need. Now it is useless discussing future scenarios.

@demilatof when do you think it is useful to discuss future changes/scenarios? 1 month from now, or 6 months from now or 1 year from now?

demilatof commented 5 months ago

@demilatof content is not identical here. I assume that you are in agreement that xslt can be used generated identical hash, if content is same?

Yes, but if they require to have the same content, they could also require to have the same order. Why not?

@demilatof when do you think it is useful to discuss future changes/scenarios? 1 month from now, or 6 months from now or 1 year from now?

Not now, not without a good analysis of the problem, and not without an explicit request that the two IIAs MUST BE identical everywhere. I really don't understand how do you think to discuss now, with an incoming deadline, about a problem that we don't even know if it exists and with what extents. I'm not used to work so, it only brings to a waste of time and unpredictable results.

When they decide that the two IIAs must be identical, let me know, thanks

umesh-qs commented 5 months ago

@demilatof content is not identical here. I assume that you are in agreement that xslt can be used generated identical hash, if content is same?

Yes, but if they require to have the same content, they could also require to have the same order. Why not?

@demilatof same order can also be achieved by using the xslt transformation, before responding to the get request. This is same as generating same hash with different ordering. Again, on same content on both IIAs, I would request Janina to confirm on the business requirement, else there is no point discussing further.

@demilatof when do you think it is useful to discuss future changes/scenarios? 1 month from now, or 6 months from now or 1 year from now?

Not now, not without a good analysis of the problem, and not without an explicit request that the two IIAs MUST BE identical everywhere. I really don't understand how do you think to discuss now, with an incoming deadline, about a problem that we don't even know if it exists and with what extents. I'm not used to work so, it only brings to a waste of time and unpredictable results.

When they decide that the two IIAs must be identical, let me know, thanks

@demilatof discussions cannot and must not stop because of unavailability of few folks. I am not sure what you mean by "explicit request", but if it is already done by some provider in v7, and allowed, and also mentioned above by Janina (requesting to confirm again) isn't it obvious that this (same content) is the direction DG EAC wants to take.

demilatof commented 5 months ago

@demilatof discussions cannot and must not stop because of unavailability of few folks. I am not sure what you mean by "explicit request", but if it is already done by some provider in v7, and allowed, and also mentioned above by Janina (requesting to confirm again) isn't it obvious that this (same content) is the direction DG EAC wants to take.

I'm glad to know that you have time and money to discuss about issues that are out of specifics and nobody know it they will be ever imposed. If some providers require that identity in v7 and may be not in v6, they are completely out of specifics, because nowhere is written that the two hash (iia hash or hash code) MUST be equals.

I've no mind to modify my code just to make EWP compliant bad implementations of some provider; if something has gone wrong in their code, it's up to them to find a solution

umesh-qs commented 5 months ago

@demilatof discussions cannot and must not stop because of unavailability of few folks. I am not sure what you mean by "explicit request", but if it is already done by some provider in v7, and allowed, and also mentioned above by Janina (requesting to confirm again) isn't it obvious that this (same content) is the direction DG EAC wants to take.

I'm glad to know that you have time and money to discuss about issues that are out of specifics and nobody know it they will be ever imposed. If some providers require that identity in v7 and may be not in v6, they are completely out of specifics, because nowhere is written that the two hash (iia hash or hash code) MUST be equals.

I've no mind to modify my code just to make EWP compliant bad implementations of some provider; if something has gone wrong in their code, it's up to them to find a solution

Discussion with you so difficult, because you do not want to listen. Isn't it mentioned multiple times that specification doesn't restrict anyone on match/not matching the IIA content? No one has said that someone is currently matching hash in v7. Some providers are matching the content and the same has been communicated multiple times.

I repeat again, there are providers whose implementation must match IIA content in v7, which is allowed by DG EAC. And if that is allowed then it makes sense to move towards matching hash using the xslt transformation.

I can clearly see that due to your busy schedule you are missing the discussion points when replying. I suggest you leave these less important issue with others and focus on something better.

demilatof commented 5 months ago

Discussion with you so difficult, because you do not want to listen. Isn't it mentioned multiple times that specification doesn't restrict anyone on match/not matching the IIA content? No one has said that someone is currently matching hash in v7. Some providers are matching the content and the same has been communicated multiple times.

I repeat again, there are providers whose implementation must match IIA content in v7, which is allowed by DG EAC. And if that is allowed then it makes sense to move towards matching hash using the xslt transformation.

I can clearly see that due to your busy schedule you are missing the discussion points when replying. I suggest you leave these less important issue with others and focus on something better.

No, you're not catching the point If it is allowed to match IIA content in v7 (as it was it v6), it doesn't mean that it is required. What point of the specifications would have introduced this news? And if it is not required, it is out of specifications who wants that the partner has the same content for the IIA (that is the problem raised during the last IF).

If you do not want to impose identical IIAs to everybody, then you're suggesting a solution that put in a mess the incoming v7, just to facilitate who has the same content in a different order, because a new XSLT would make incompatible the IIA hashes just created.

You can implement a second XSLT just for internal use or to be commonly adopted but in addition to the current one, eventually for automatic approval (if you like).

If we want to improve the EWP exchanges, we need to review the architecture from the basis. Before thinking about a new XSLT let see the impact of the current one in the already approved IIAs. Just to say...

umesh-qs commented 5 months ago

Discussion with you so difficult, because you do not want to listen. Isn't it mentioned multiple times that specification doesn't restrict anyone on match/not matching the IIA content? No one has said that someone is currently matching hash in v7. Some providers are matching the content and the same has been communicated multiple times. I repeat again, there are providers whose implementation must match IIA content in v7, which is allowed by DG EAC. And if that is allowed then it makes sense to move towards matching hash using the xslt transformation. I can clearly see that due to your busy schedule you are missing the discussion points when replying. I suggest you leave these less important issue with others and focus on something better.

No, you're not catching the point If it is allowed to match IIA content in v7 (as it was it v6), it doesn't mean that it is required. What point of the specifications would have introduced this news? And if it is not required, it is out of specifications who wants that the partner has the same content for the IIA (that is the problem raised during the last IF).

If you do not want to impose identical IIAs to everybody, then you're suggesting a solution that put in a mess the incoming v7, just to facilitate who has the same content in a different order, because a new XSLT would make incompatible the IIA hashes just created.

You can implement a second XSLT just for internal use or to be commonly adopted but in addition to the current one, eventually for automatic approval (if you like).

If we want to improve the EWP exchanges, we need to review the architecture from the basis. Before thinking about a new XSLT let see the impact of the current one in the already approved IIAs. Just to say...

Let me again point you to https://github.com/erasmus-without-paper/ewp-specs-api-iias/issues/170#issuecomment-2014876270 And how do you expect a particular client of provider to not to match the IIA content and still go through with mutual approval with any client of Dashboard? It might not be in the specification yet, but considering these points there is no reason to not to have it in the specification in future. @janinamincer-daszkiewicz please suggest. Your arguments keeps changing based on your convenience. You sometimes go by what Janina confirms irrespective of whether it is in the specification or not, and sometimes you want it in specification to even start the discussion. That is where it is so difficult to have discussion with you.

demilatof commented 5 months ago

et me again point you to #170 (comment)

You gave this assumption without any proof, point me to the BPO requirement that says this:

But for now business requirement is that the content of IIAs must match

And how do you expect a particular client of provider to not to match the IIA content and still go through with mutual approval with any client of Dashboard?

So now you think that if Dashboard is wrong, we all should make a wrong implementation just to communicate with it? If my IRO accept to modify our IIA to have the same content, he/she can already do. If my IRO doesn't accept to modify our IIA to have the same content, we can approve partner's IIA anyway; it it the partner that has to modify its IIA, if his/her system doesn't accept different contents.

In both of the cases, it's a problem of the partner's internal system, because the specifications explicitly say:

The digest is present in the IIAs API get response, but MUST be verified by the server before sending the approval. For this purpose, the server has to call the IIAs get API and compare the hash received in the response with the hash independently calculated from that response. If both hashes are identical, the agreement can be approved.

The specifications don't take into any consideration the need that the two content are the same

It might not be in the specification yet, but considering these points there is no reason to not to have it in the specification in future. @janinamincer-daszkiewicz please suggest.

Again: now, are you thinking that the specifications are written by Dashboard implementation?

Your arguments keeps changing based on your convenience. You sometimes go by what Janina confirms irrespective of whether it is in the specification or not, and sometimes you want it in specification to even start the discussion. That is where it is so difficult to have discussion with you.

You're wrong again, but this is not a problem for me. Show me where is written that business requirement is that the content of IIAs must match and we can talk about this question.

umesh-qs commented 5 months ago

et me again point you to #170 (comment)

You gave this assumption without any proof, point me to the BPO requirement that says this:

But for now business requirement is that the content of IIAs must match

And how do you expect a particular client of provider to not to match the IIA content and still go through with mutual approval with any client of Dashboard?

So now you think that if Dashboard is wrong, we all should make a wrong implementation just to communicate with it? If my IRO accept to modify our IIA to have the same content, he/she can already do. If my IRO doesn't accept to modify our IIA to have the same content, we can approve partner's IIA anyway; it it the partner that has to modify its IIA, if his/her system doesn't accept different contents.

@demilatof did I say Dashboard is wrong? Why do you think Dashboard is wrong? I you think Dashboard is not wrong please answer how other systems are supposed to deal with Dashboard insisting on IIA content matching?

In both of the cases, it's a problem of the partner's internal system, because the specifications explicitly say:

The digest is present in the IIAs API get response, but MUST be verified by the server before sending the approval. For this purpose, the server has to call the IIAs get API and compare the hash received in the response with the hash independently calculated from that response. If both hashes are identical, the agreement can be approved.

The specifications don't take into any consideration the need that the two content are the same

@demilatof you are really impossible. Please go through previous comments in this thread and think from a constructive mind. We are talking about future changes.

It might not be in the specification yet, but considering these points there is no reason to not to have it in the specification in future. @janinamincer-daszkiewicz please suggest.

Again: now, are you thinking that the specifications are written by Dashboard implementation?

@demilatof You might want to check with DG EAC how specifications are written. My suggestions are based on business requirement that I believe is confirmed twice. Once in the IF call and second time in this discussion.

Your arguments keeps changing based on your convenience. You sometimes go by what Janina confirms irrespective of whether it is in the specification or not, and sometimes you want it in specification to even start the discussion. That is where it is so difficult to have discussion with you.

You're wrong again, but this is not a problem for me. Show me where is written that business requirement is that the content of IIAs must match and we can talk about this question.

I know you will never acknowledge. Here is one of your comment that says specification could have been made better but clarification from Janina is enough https://github.com/erasmus-without-paper/ewp-specs-api-iias-approval/issues/15#issuecomment-1898401065

I will stop on this until there is some clarification from @janinamincer-daszkiewicz on the business requirement.

demilatof commented 5 months ago

I'm still waiting a proof for your claim:

But for now business requirement is that the content of IIAs must match

You don't have? Keep on speaking about nothing by yourself

umesh-qs commented 5 months ago

I'm still waiting a proof for your claim:

But for now business requirement is that the content of IIAs must match

You don't have? Keep on speaking about nothing by yourself

There is no need to answer. Because as I said your approach switches as per you convenience. And you can go to any extent to defend that. One more to refresh your memory https://github.com/erasmus-without-paper/ewp-specs-api-iias/issues/137#issuecomment-1740876127

demilatof commented 5 months ago

There is no need to answer. Because as I said your approach switches as per you convenience. And you can go to any extent to defend that. One more to refresh your memory #137 (comment)

What you call "my convenience" is the list of specifications you misunderstood (as empty approval or snapshot) and I have no time to look for among the github issues, but I see you have time and you're doing a good job.

Take a minute and show me where you have picked up that

But for now business requirement is that the content of IIAs must match

or should I suppose that you are incorrect and give fake news to misrepresent the BPO requirements on your convenience? You know, it could be a bit unfair to EWP community

umesh-qs commented 5 months ago

There is no need to answer. Because as I said your approach switches as per you convenience. And you can go to any extent to defend that. One more to refresh your memory #137 (comment)

What you call "my convenience" is the list of specifications you misunderstood (as empty approval or snapshot) and I have no time to look for among the github issues, but I see you have time and you're doing a good job.

@demilatof Unfortunately I have to do this because people tends to forget their past. You have all the time to go into unnecessary argument, but when shown the reality, you all of a sudden become busy. You were not able to point me to the specifications that justify your understanding and kept taking about how Janina has explained it. Now in the similar situation when, I am quoting Janina you conveniently changed your stance.

Take a minute and show me where you have picked up that

But for now business requirement is that the content of IIAs must match

or should I suppose that you are incorrect and give fake news to misrepresent the BPO requirements on your convenience? You know, it could be a bit unfair to EWP community

@demilatof I have already mentioned instances where you change track based on your convenience. Let community decide whose comments are fair/unfair/biased. And I have already replied multiple times only if you wish to understand. Please see again where I am quoting from https://github.com/erasmus-without-paper/ewp-specs-api-iias/issues/170#issuecomment-2014876270

Problem is that you want people to answer exactly what your mind is expecting and hence there is no discussion possible with you. If what @janinamincer-daszkiewicz replied in the above comment was a mistake then there is absolutely no point discussion further. So you might want to ask her where is the business requirement.

umesh-qs commented 5 months ago

@demilatof did I say Dashboard is wrong? Why do you think Dashboard is wrong? I you think Dashboard is not wrong please answer how other systems are supposed to deal with Dashboard insisting on IIA content matching?

@demilatof You missed this question. Can you can please take a few minutes from your busy schedule and answer this.

demilatof commented 5 months ago

@demilatof did I say Dashboard is wrong? Why do you think Dashboard is wrong? I you think Dashboard is not wrong please answer how other systems are supposed to deal with Dashboard insisting on IIA content matching?

@demilatof You missed this question. Can you can please take a few minutes from your busy schedule and answer this.

Here you are:

And how do you expect a particular client of provider to not to match the IIA content and still go through with mutual approval with any client of Dashboard? It might not be in the specification yet

Not in the specification yet = wrong to require an IIA match to go through with mutual approval with any client of Dashboard

I'm still waiting for your proof; you are mixing specification that requires to be understood (and then are questionable) with claims that you feel free to attribute to BPO without giving a proof. I don't discuss with incorrect people. Keep on crying with your blah blah

umesh-qs commented 5 months ago

@demilatof did I say Dashboard is wrong? Why do you think Dashboard is wrong? I you think Dashboard is not wrong please answer how other systems are supposed to deal with Dashboard insisting on IIA content matching?

@demilatof You missed this question. Can you can please take a few minutes from your busy schedule and answer this.

Here you are:

And how do you expect a particular client of provider to not to match the IIA content and still go through with mutual approval with any client of Dashboard? It might not be in the specification yet

Not in the specification yet = wrong to require an IIA match to go through with mutual approval with any client of Dashboard @demilatof thanks for confirming. So clients using your system will not match the IIA content, and complain to national agencies that they will not make changes to the IIA since it is Dashboard problem? Or they will match the IIA content and go for mutual approval?

I'm still waiting for your proof; you are mixing specification that requires to be understood (and then are questionable) with claims that you feel free to attribute to BPO without giving a proof. I don't discuss with incorrect people. Keep on crying with your blah blah

@demilatof You don't seem to be in your senses. This is already answered by me multiple times.

mkurzydlowski commented 5 months ago

I was asked by Janina to test the XSLT Umesh posted. Here are my findings.

What works:

What we still need:

What we still don't know how to handle:

umesh-qs commented 2 months ago

IIA hash match as long as content is same.txt Adding transformation xslt that takes care of sorting, decimal formatting, boolean, iia-id ordering,language element codes (en vs en-US, etc.). This will probably serve 99% of the cases. If the content is not same there is no way hash can match. This is what I mentioned in the last IF called and this is what I shared with Michal in march this year. I am posting it for reference if anyone wants to make use of it or improve it