IATI Publisher not publishing transactions to activity file XML

[emmajclegg]

Multiple IATI Publisher users have reported that all activity transactions are now missing from their published IATI activity file. The transactions appear saved and published in IATI Publisher, but are not visible in their activity file on the IATI Registry.

It looks like these users have published (or republished) their activity data using IATI Publisher recently, since the 31st January.

I have replicated this issue today on the IATI Publisher staging site with the following test account and activity:

• organisation name: emma-troubleshooting
• activity iati-identifier: CA-CC-XXX-999-TEST-1
• activity title: Troubleshooting IATI Publisher

This activity has 2 transactions showing as published within IATI Publisher, which are also present in the activity XLS download from Publisher. However, no transactions are visible in the XML file on the IATI Registry staging site.

(Screenshots from IATI Publisher:)

[praweshsth]

• [x] Fix the bug
• [x] change the XML file of the published activities and republish them with same published date rather than current date

[robredpath]

Thanks @praweshsth ! Could you let us know a little more about how this part of the system works, and what happened?

We were surprised by this because we expected the XML download in the Publisher UI to be the same as the XML that's listed in the Registry.

[PG-Momik]

Hi @robredpath ,

Bug

Context

On Dec 18, we merged the changes in branch: #1300. The changes were made in accordance with issue/discussion #1299 . A huge module of how xml's were generated for publishing had to be rewritten. This bug stems from those changes.

Problem

In its entirety the bug stems from a single line of code where transactions of activities are fetched using the code:

correct	current
$activity->transactions ?? []	$activity->transaction ?? []

Solution

Change $activity->transaction ?? [] to $activity->transactions ?? [].

---

Why xml downloaded in the system and in registry is not the same

" We were surprised by this because we expected the XML download in the Publisher UI to be the same as the XML that's listed in the Registry." - robredpath

Heres a brief summary of how xmls are published or downloaded.

Publish workflow

1. When publishing an activity we maintain an individual xml file with the name: <publisherId>-<activityId>.xml .

2. The xml value of that activity is also maintained in a parent xml (merged xml) with the name: <publisherId>-activities.xml.

   <xml>
   <iati-activities version="2.03" generated-datetime="timestamp">
     <iati-activity>...valid activity xml...</iati-activity>
     <iati-activity>...valid activity xml...</iati-activity>
   </iati-activities>

3. Before changes for #1299 was merged, in any kind of publish (whether single or bulk), the entire parent xml used to be regenerated from scratch.
4. In the current implementation we avoid regenerating the entire xml and instead just remove the activity or append it from the merged xml as needed.

Download workflow

1. When users download activity(s) we convert the current activity data (present in the system db) to the desired file format. And allows users to download accordingly.
2. We use the data from the database and NOT the activity xml for download. This is because:
   • Xml file will not exist unless an activity has been published.
   • User might chose to download an xml before publishing the activity (which cant be done, since xml file wont have existed to manipulate and download)
   • Using published xml as a source for download will not represent the current state of the data. ( If a user publishes an activity, then publishes it, then edits some data. The edited data will not be reflected in the downloaded file. )

Conclusion

• Because of how download and publish are implemented, the merged xml can have the same value as the xml listed in registry but, they are NOT the same 'object'.
• The registry points to a url of the merged xml (which has bugged data in this case).
• The downloaded xml is a freshly generated xml.

[emmajclegg]

Thanks for the information @PG-Momik and @praweshsth - let me know when the fix has been deployed and I'll check the affected publishers' transaction data.

[PG-Momik]

Hi @emmajclegg , The bug fix has been deployed to stage and is ready to be deployed to production. Activities will now be published with their transaction.

To fix broken data with, we can do any of the following:

1. Let users republish the affected activities.
2. Run a script to fix the activities.

The script is currently being tested by our QA. Once its approved, it can be executed on production.

[!NOTE] Do note that the script will only affect published activities and will not affect activities in republish state.

[emmajclegg]

To clarify @PG-Momik - has the first part of this (the bug fix to ensure activities publish with transactions) been deployed to production? Are we now focusing on the second part (the script to retrospectively republish affected publishers' activities)?

The retrospective republishing script will be needed as us in IATI Support won't be aware of all of the affected publishers, only those that have contacted us. But if this is holding up deploying the main bug fix, please do that first (from my testing on staging last week, it looked like transactions were publishing as expected)

[praweshsth]

@emmajclegg the bug fixing has not yet been deployed in the production but we can deploy it. and then fix the activities of the organisations that were affected by the bug. We can do both of these today.

[praweshsth]

The bug has been deployed but we are working on the script to fix the existing xml files of already published activities. @emmajclegg

[PG-Momik]

@emmajclegg , the script has been deployed and executed to production. Here's what the script does:

• Fetch assumed broken activities by fetching all activities that were published after Dec 01, 2023.
• Filter assumed broken activities by comparing number of transactions in database vs number of transactions in xml.
• Iterate over the filtered activities and, if the number of transactions in database and xml are not the same we add the said activity to a list of confirmed broken activities.
• The script iterates over all confirmed broken activities, and regenerates + replaces the broken <iati-activity> node in the merged xml that the registry points to.
• To confirm if the script successfully fixed an activity, we once again compare the number of transactions in database vs xml. We also check that the last-updated-datetime attribute of <iati-activity> node has not changed.

[emmajclegg]

Thank you @PG-Momik

[PG-Momik]

This has been completed and deployed to main. But it was reopened to mention the inner-working of the script. cc: @praweshsth @emmajclegg @robredpath