kndm / iati-diff

Iati XML files differences check
GNU Affero General Public License v3.0
1 stars 1 forks source link

Output just showing elements that have changed - need to show the data #8

Closed matmaxgeds closed 5 years ago

matmaxgeds commented 5 years ago

I.e. need to show the content of the element that has changed i.e. need a new column for the content of that element so I can have an idea what was removed/added in terms of content

kndm commented 5 years ago

@matmaxgeds I played around with this a little bit but it proves to be very time extensive if I wanted to show the path's content (old value), given the way this library works and the way the parameters given/returned interact in the process itself.

I'd love to suggest the following solution instead:

For each activity found in activity-list(2).xml -> generate a new XML file per iati-identifier (split the big list into individual files) -> query the API using each iati-identifier -> compare the XML file obtained from the API to the one generated after splitting -> generate a new XML file with the differences being pointed out in CSS-like attributes -> now that the differences are CSS-friendly, decide on the best visual-formatting possible.

Following three files (an activity split from the activity-list(2).xml, the XML from same activity beign queried to the datastore, and the resulting XML with the diff attributes) for better visualization of this process.

Activity file from the split:

<result xmlns:iati-extra="http://datastore.iatistandard.org/ns">
    <iati-activity iati-extra:version="2.03" xml:lang="en" default-currency="CHF" last-updated-datetime="2019-03-11T00:00:00Z">
      <iati-identifier>CH-4-2008004812</iati-identifier>
      <reporting-org ref="CH-4" type="10">
         <narrative>Swiss Agency for Development and Cooperation (SDC)</narrative>
      </reporting-org>
      <title>
         <narrative>OCHA Humanitarian Response Fund</narrative>
      </title>
      <description type="1">
         <narrative>No description for this activity available at this point. Description indisponible pour cette activité.</narrative>
      </description>
      <participating-org ref="CH-4" role="1" type="10">
         <narrative>Swiss Agency for Development and Cooperation (SDC)</narrative>
      </participating-org>
      <participating-org ref="XM-DAC-41114" role="4" type="40">
         <narrative>Programme des Nations Unies pour le développement</narrative>
      </participating-org>
      <other-identifier ref="7F-06559.13.01" type="A1"/>
      <activity-status code="2"/>
      <activity-date iso-date="2018-03-15" type="1"/>
      <!--planned start-->
      <activity-date iso-date="2018-03-15" type="2"/>
      <!-- Actual start-->
      <activity-date iso-date="2018-12-31" type="3"/>
      <!-- planned end-->
      <activity-date iso-date="2018-12-31" type="4"/>
      <!-- Actual end-->
      <contact-info type="1">
         <!-- general enquiries-->
         <organisation>
            <narrative>Swiss Agency for Development and Cooperation SDC</narrative>
         </organisation>
         <department>
            <narrative>Statistics Unit</narrative>
         </department>
         <telephone>+41 58 464 92 35</telephone>
         <email>stats.sdc@eda.admin.ch</email>
         <website>http://www.sdc.admin.ch</website>
         <mailing-address>
            <narrative>Freiburgstrasse 130, 3003 Berne</narrative>
         </mailing-address>
      </contact-info>
      <recipient-country code="SO">
         <narrative>Somalie</narrative>
      </recipient-country>
      <sector vocabulary="1" code="72010" percentage="20"/>
      <sector vocabulary="1" code="72050" percentage="30"/>
      <sector vocabulary="1" code="72040" percentage="50"/>
      <policy-marker code="6" significance="0">
         <narrative>Aid Targeting the Objectives of the Framework Convention on Climate Change - Mitigation</narrative>
      </policy-marker>
      <policy-marker code="8" significance="0">
         <narrative>Aid Targeting the Objectives of the Convention to Combat Desertification</narrative>
      </policy-marker>
      <policy-marker code="7" significance="0">
         <narrative> Aid Targeting the Objectives of the Framework Convention on Climate Change - Adaptation </narrative>
      </policy-marker>
      <policy-marker code="2" significance="0">
         <narrative>Aid to Environment</narrative>
      </policy-marker>
      <policy-marker code="5" significance="0">
         <narrative>Aid Targeting the Objectives of the Convention on Biological Diversity</narrative>
      </policy-marker>
      <policy-marker code="9" significance="0">
         <narrative> Reproductive, Maternal, Newborn and Child Health (RMNCH)</narrative>
      </policy-marker>
      <collaboration-type code="1"/>
      <default-flow-type code="10"/>
      <default-finance-type code="110"/>
      <default-aid-type code="B03" vocabulary="1"/>
      <default-tied-status code="5"/>
      <transaction ref="7F-06559.13.01">
         <transaction-type code="2"/>
         <transaction-date iso-date="2018-12-15"/>
         <value currency="CHF" value-date="2018-12-15">1000000</value>
         <description>
            <narrative> Outgoing Commitment </narrative>
         </description>

      </transaction>
      <transaction ref="7F-06559.13.01">
         <transaction-type code="3"/>
         <transaction-date iso-date="2018-12-15"/>
         <value currency="CHF" value-date="2018-12-15">1000000</value>
         <description>
            <narrative> Disbursement</narrative>
         </description>

      </transaction>
      <conditions attached="0"/>
   </iati-activity></result>

Activity file from the Datastore:

<result xmlns:iati-extra="http://datastore.iatistandard.org/ns">  
  <iati-activity>
    <iati-identifier>CH-4-2008004812</iati-identifier>
    <reporting-org ref="CH-4" type="10" secondary-reporter="0">
      <narrative>Switzerland - Swiss Agency for Development and Cooperation (SDC)</narrative>
    </reporting-org>
    <title>
      <narrative>OCHA Humanitarian Response Fund</narrative>
    </title>
    <description type="1">
      <narrative>No description for this activity available at this point. Description indisponible pour cette activité.</narrative>
    </description>
    <participating-org ref="CH-4" type="10" role="1">
      <narrative>Swiss Agency for Development and Cooperation (SDC)</narrative>
    </participating-org>
    <participating-org ref="XM-DAC-41114" type="40" role="4">
      <narrative>Programme des Nations Unies pour le développement</narrative>
    </participating-org>
    <other-identifier ref="7F-06559.13.01" type="A1">
      <owner-org/>
    </other-identifier>
    <activity-status code="2"/>
    <activity-date iso-date="2018-03-15" type="1"/>
    <activity-date iso-date="2018-03-15" type="2"/>
    <activity-date iso-date="2018-12-31" type="3"/>
    <activity-date iso-date="2018-12-31" type="4"/>
    <contact-info type="1">
      <organisation>
        <narrative>Swiss Agency for Development and Cooperation SDC</narrative>
      </organisation>
      <department>
        <narrative>Statistics Unit</narrative>
      </department>
      <telephone>+41 58 464 92 35</telephone>
      <email>stats.sdc@eda.admin.ch</email>
      <website>http://www.sdc.admin.ch</website>
      <mailing-address>
        <narrative>Freiburgstrasse 130, 3003 Berne</narrative>
      </mailing-address>
    </contact-info>
    <recipient-country code="SO"/>
    <sector code="72010" vocabulary="1" percentage="20.00"/>
    <sector code="72050" vocabulary="1" percentage="30.00"/>
    <sector code="72040" vocabulary="1" percentage="50.00"/>
    <policy-marker vocabulary="1" code="6" significance="0">
      <narrative>Aid Targeting the Objectives of the Framework Convention on Climate Change - Mitigation</narrative>
    </policy-marker>
    <policy-marker vocabulary="1" code="8" significance="0">
      <narrative>Aid Targeting the Objectives of the Convention to Combat Desertification</narrative>
    </policy-marker>
    <policy-marker vocabulary="1" code="7" significance="0">
      <narrative> Aid Targeting the Objectives of the Framework Convention on Climate Change - Adaptation </narrative>
    </policy-marker>
    <policy-marker vocabulary="1" code="2" significance="0">
      <narrative>Aid to Environment</narrative>
    </policy-marker>
    <policy-marker vocabulary="1" code="5" significance="0">
      <narrative>Aid Targeting the Objectives of the Convention on Biological Diversity</narrative>
    </policy-marker>
    <policy-marker vocabulary="1" code="9" significance="0">
      <narrative> Reproductive, Maternal, Newborn and Child Health (RMNCH)</narrative>
    </policy-marker>
    <collaboration-type code="1"/>
    <default-flow-type code="10"/>
    <default-finance-type code="110"/>
    <default-tied-status code="5"/>
    <transaction ref="7F-06559.13.01">
      <transaction-type code="2"/>
      <transaction-date iso-date="2018-12-15"/>
      <value currency="CHF" value-date="2018-12-15">1000000.00</value>
      <description>
        <narrative> Outgoing Commitment </narrative>
      </description>
      <sector code="72040" vocabulary="1"/>
      <sector code="72050" vocabulary="1"/>
      <sector code="72010" vocabulary="1"/>
      <recipient-country code="SO"/>
      <flow-type code="10"/>
      <finance-type code="110"/>
      <tied-status code="5"/>
    </transaction>
    <transaction ref="7F-06559.13.01">
      <transaction-type code="3"/>
      <transaction-date iso-date="2018-12-15"/>
      <value currency="CHF" value-date="2018-12-15">1000000.00</value>
      <description>
        <narrative> Disbursement</narrative>
      </description>
      <sector code="72040" vocabulary="1"/>
      <sector code="72050" vocabulary="1"/>
      <sector code="72010" vocabulary="1"/>
      <recipient-country code="SO"/>
      <flow-type code="10"/>
      <finance-type code="110"/>
      <tied-status code="5"/>
    </transaction>
    <conditions attached="0"/>
  </iati-activity></result>

Difference between the two turned into a XML with css-friendly attributes:

<result xmlns:iati-extra="http://datastore.iatistandard.org/ns" xmlns:diff="http://namespaces.shoobx.com/diff">
  <iati-activity diff:insert="" default-currency="CHF" diff:add-attr="default-currency;last-updated-datetime;{http://datastore.iatistandard.org/ns}version;{http://www.w3.org/XML/1998/namespace}lang" last-updated-datetime="2019-03-11T00:00:00Z" iati-extra:version="2.03" xml:lang="en">
    <iati-identifier diff:insert="">CH-4-2008004812</iati-identifier>
    <reporting-org ref="CH-4" type="10" diff:insert="" diff:delete-attr="secondary-reporter">
      <narrative><diff:delete>Switzerland - </diff:delete>Swiss Agency for Development and Cooperation (SDC)</narrative>
    </reporting-org>
    <title diff:insert="">
      <narrative>OCHA Humanitarian Response Fund</narrative>
    </title>
    <description type="1" diff:insert="">
      <narrative>No description for this activity available at this point. Description indisponible pour cette activité.</narrative>
    </description>
    <participating-org ref="CH-4" type="10" role="1" diff:insert="">
      <narrative>Swiss Agency for Development and Cooperation (SDC)</narrative>
    </participating-org>
    <participating-org ref="XM-DAC-41114" type="40" role="4" diff:insert="">
      <narrative>Programme des Nations Unies pour le développement</narrative>
    </participating-org>
    <other-identifier ref="7F-06559.13.01" type="A1" diff:insert="">
      <owner-org diff:delete=""/>
    </other-identifier>
    <activity-status code="2" diff:insert=""/>
    <activity-date iso-date="2018-03-15" type="1" diff:insert=""/>
    <activity-date iso-date="2018-03-15" type="2" diff:insert=""/>
    <activity-date iso-date="2018-12-31" type="3" diff:insert=""/>
    <activity-date iso-date="2018-12-31" type="4" diff:insert=""/>
    <contact-info type="1" diff:insert="">
      <organisation diff:insert="" diff:rename="owner-org">
        <narrative diff:insert="">Swiss Agency for Development and Cooperation SDC</narrative>
      </organisation>
      <department diff:rename="organisation">
        <narrative diff:delete="">Swiss Agency for Development and Cooperation SDC</narrative>
        <narrative diff:insert="">Statistics Unit</narrative>
      </department>
      <department diff:delete="">
        <narrative>Statistics Unit</narrative>
      </department>
      <telephone>+41 58 464 92 35</telephone>
      <email>stats.sdc@eda.admin.ch</email>
      <website>http://www.sdc.admin.ch</website>
      <mailing-address diff:insert="" diff:rename="department">
        <narrative diff:delete="">Statistics Unit</narrative>
        <narrative diff:insert="">Freiburgstrasse 130, 3003 Berne</narrative>
      </mailing-address>
      <mailing-address diff:delete="">
        <narrative>Freiburgstrasse 130, 3003 Berne</narrative>
      </mailing-address>
    </contact-info>
    <recipient-country code="SO" diff:insert="">
      <narrative diff:insert="">Somalie</narrative>
    </recipient-country>
    <sector code="72010" vocabulary="1" percentage="20" diff:insert="" diff:update-attr="percentage:20.00"/>
    <sector code="72050" vocabulary="1" percentage="30" diff:insert="" diff:update-attr="percentage:30.00"/>
    <sector code="72040" vocabulary="1" percentage="50" diff:insert="" diff:update-attr="percentage:50.00"/>
    <policy-marker code="6" significance="0" diff:insert="" diff:delete-attr="vocabulary">
      <narrative>Aid Targeting the Objectives of the Framework Convention on Climate Change - Mitigation</narrative>
    </policy-marker>
    <policy-marker code="8" significance="0" diff:insert="" diff:delete-attr="vocabulary">
      <narrative>Aid Targeting the Objectives of the Convention to Combat Desertification</narrative>
    </policy-marker>
    <policy-marker code="7" significance="0" diff:insert="" diff:delete-attr="vocabulary">
      <narrative> Aid Targeting the Objectives of the Framework Convention on Climate Change - Adaptation </narrative>
    </policy-marker>
    <policy-marker code="2" significance="0" diff:insert="" diff:delete-attr="vocabulary">
      <narrative>Aid to Environment</narrative>
    </policy-marker>
    <policy-marker code="5" significance="0" diff:insert="" diff:delete-attr="vocabulary">
      <narrative>Aid Targeting the Objectives of the Convention on Biological Diversity</narrative>
    </policy-marker>
    <policy-marker code="9" significance="0" diff:insert="" diff:delete-attr="vocabulary">
      <narrative> Reproductive, Maternal, Newborn and Child Health (RMNCH)</narrative>
    </policy-marker>
    <collaboration-type code="1" diff:insert="" diff:rename="recipient-country" diff:update-attr="code:SO"/>
    <default-flow-type code="10" diff:insert="" diff:rename="collaboration-type" diff:update-attr="code:1"/>
    <default-finance-type code="110" diff:insert="" diff:rename="default-flow-type" diff:update-attr="code:10"/>
    <default-aid-type code="B03" vocabulary="1" diff:insert="" diff:rename="sector" diff:update-attr="code:72040"/>
    <default-tied-status code="5" diff:insert="" diff:rename="default-finance-type" diff:update-attr="code:110"/>
    <transaction ref="7F-06559.13.01" diff:insert="">
      <transaction-type code="2" diff:insert="" diff:rename="default-tied-status" diff:update-attr="code:5"/>
      <transaction-date iso-date="2018-12-15" diff:insert=""/>
      <value currency="CHF" value-date="2018-12-15" diff:insert="">1000000</value>
      <description diff:insert="" diff:rename="mailing-address">
        <narrative diff:delete="">Freiburgstrasse 130, 3003 Berne</narrative>
        <narrative diff:insert=""> Outgoing Commitment </narrative>
      </description>
      <transaction-type code="3" diff:delete=""/>
      <transaction-date iso-date="2018-12-15" diff:delete=""/>
      <value currency="CHF" value-date="2018-12-15" diff:delete="">1000000.00</value>
      <description diff:delete="">
        <narrative diff:delete=""> Disbursement</narrative>
      </description>
      <sector code="72040" vocabulary="1" diff:delete=""/>
      <sector code="72050" vocabulary="1" diff:delete=""/>
      <sector code="72010" vocabulary="1" diff:delete=""/>
      <recipient-country code="SO" diff:delete=""/>
      <flow-type code="10" diff:delete=""/>
      <finance-type code="110" diff:delete=""/>
      <tied-status code="5" diff:delete=""/>
    </transaction>
    <transaction ref="7F-06559.13.01" diff:insert="">
      <transaction-type code="3" diff:update-attr="code:2"/>
      <transaction-date iso-date="2018-12-15" diff:delete=""/>
      <value currency="CHF" value-date="2018-12-15" diff:delete="">1000000.00</value>
      <description diff:delete="">
        <narrative> Outgoing Commitment </narrative>
      </description>
      <sector code="72040" vocabulary="1" diff:delete=""/>
      <sector code="72050" vocabulary="1" diff:delete=""/>
      <sector code="72010" vocabulary="1" diff:delete=""/>
      <recipient-country code="SO" diff:delete=""/>
      <flow-type code="10" diff:delete=""/>
      <transaction-date diff:rename="finance-type" iso-date="2018-12-15" diff:add-attr="iso-date" diff:delete-attr="code"/>
      <value diff:insert="" diff:rename="sector" currency="CHF" diff:add-attr="currency;value-date" value-date="2018-12-15" diff:delete-attr="code;vocabulary">1000000</value>
      <description diff:insert="">
        <narrative diff:delete=""> Outgoing Commitment </narrative>
        <narrative diff:insert=""> Disbursement</narrative>
      </description>
      <tied-status code="5" diff:delete=""/>
    </transaction>
    <conditions diff:insert="" diff:rename="flow-type" attached="0" diff:add-attr="attached" diff:delete-attr="code"/>
  </iati-activity>
  <iati-activity diff:delete="">
    <iati-identifier diff:delete="">CH-4-2008004812</iati-identifier>
    <reporting-org ref="CH-4" type="10" secondary-reporter="0" diff:delete="">
      <narrative>Switzerland - Swiss Agency for Development and Cooperation (SDC)</narrative>
    </reporting-org>
    <title diff:delete="">
      <narrative>OCHA Humanitarian Response Fund</narrative>
    </title>
    <description type="1" diff:delete="">
      <narrative>No description for this activity available at this point. Description indisponible pour cette activité.</narrative>
    </description>
    <participating-org ref="CH-4" type="10" role="1" diff:delete="">
      <narrative>Swiss Agency for Development and Cooperation (SDC)</narrative>
    </participating-org>
    <participating-org ref="XM-DAC-41114" type="40" role="4" diff:delete="">
      <narrative>Programme des Nations Unies pour le développement</narrative>
    </participating-org>
    <other-identifier ref="7F-06559.13.01" type="A1" diff:delete="">
      <owner-org/>
    </other-identifier>
    <activity-status code="2" diff:delete=""/>
    <activity-date iso-date="2018-03-15" type="1" diff:delete=""/>
    <activity-date iso-date="2018-03-15" type="2" diff:delete=""/>
    <activity-date iso-date="2018-12-31" type="3" diff:delete=""/>
    <activity-date iso-date="2018-12-31" type="4" diff:delete=""/>
    <contact-info type="1" diff:delete="">
      <organisation>
        <narrative>Swiss Agency for Development and Cooperation SDC</narrative>
      </organisation>
      <department>
        <narrative>Statistics Unit</narrative>
      </department>
      <telephone>+41 58 464 92 35</telephone>
      <email>stats.sdc@eda.admin.ch</email>
      <website>http://www.sdc.admin.ch</website>
      <mailing-address>
        <narrative>Freiburgstrasse 130, 3003 Berne</narrative>
      </mailing-address>
    </contact-info>
    <recipient-country code="SO" diff:delete=""/>
    <sector code="72010" vocabulary="1" percentage="20.00" diff:delete=""/>
    <sector code="72050" vocabulary="1" percentage="30.00" diff:delete=""/>
    <sector code="72040" vocabulary="1" percentage="50.00" diff:delete=""/>
    <policy-marker vocabulary="1" code="6" significance="0" diff:delete="">
      <narrative>Aid Targeting the Objectives of the Framework Convention on Climate Change - Mitigation</narrative>
    </policy-marker>
    <policy-marker vocabulary="1" code="8" significance="0" diff:delete="">
      <narrative>Aid Targeting the Objectives of the Convention to Combat Desertification</narrative>
    </policy-marker>
    <policy-marker vocabulary="1" code="7" significance="0" diff:delete="">
      <narrative> Aid Targeting the Objectives of the Framework Convention on Climate Change - Adaptation </narrative>
    </policy-marker>
    <policy-marker vocabulary="1" code="2" significance="0" diff:delete="">
      <narrative>Aid to Environment</narrative>
    </policy-marker>
    <policy-marker vocabulary="1" code="5" significance="0" diff:delete="">
      <narrative>Aid Targeting the Objectives of the Convention on Biological Diversity</narrative>
    </policy-marker>
    <policy-marker vocabulary="1" code="9" significance="0" diff:delete="">
      <narrative> Reproductive, Maternal, Newborn and Child Health (RMNCH)</narrative>
    </policy-marker>
    <collaboration-type code="1" diff:delete=""/>
    <default-flow-type code="10" diff:delete=""/>
    <default-finance-type code="110" diff:delete=""/>
    <default-tied-status code="5" diff:delete=""/>
    <transaction ref="7F-06559.13.01" diff:delete="">
      <transaction-type code="2"/>
      <transaction-date iso-date="2018-12-15"/>
      <value currency="CHF" value-date="2018-12-15">1000000.00</value>
      <description>
        <narrative> Outgoing Commitment </narrative>
      </description>
      <sector code="72040" vocabulary="1" diff:delete=""/>
      <sector code="72050" vocabulary="1"/>
      <sector code="72010" vocabulary="1"/>
      <recipient-country code="SO" diff:delete=""/>
      <flow-type code="10"/>
      <finance-type code="110"/>
      <tied-status code="5"/>
    </transaction>
    <transaction ref="7F-06559.13.01" diff:delete="">
      <transaction-type code="3"/>
      <transaction-date iso-date="2018-12-15"/>
      <value currency="CHF" value-date="2018-12-15">1000000.00</value>
      <description>
        <narrative> Disbursement</narrative>
      </description>
      <sector code="72040" vocabulary="1"/>
      <sector code="72050" vocabulary="1"/>
      <sector code="72010" vocabulary="1"/>
      <recipient-country code="SO"/>
      <flow-type code="10"/>
      <finance-type code="110"/>
      <tied-status code="5"/>
    </transaction>
    <conditions attached="0" diff:delete=""/>
  </iati-activity>
</result>
matmaxgeds commented 5 years ago

So @kndm how can we display this in a way that a user can figure out what has changed?

matmaxgeds commented 5 years ago

closing here - will re-open as a separate issue on how to see the changes