Archiver refresh

[PetyaKangalova]

@gtkChop I have just responded to a user who asked about their metadata update.

The data was last updated on 14-01-2021.

I did run the archiver manually to refresh the metadata, but wanted to check again at what time the Archiver runs on a daily basis, so that they know in the future when the latest change will be picked up. Many thanks!

[ss-bhat]

@PetyaKangalova

Please see Archiver status here:

https://iatiregistry.org/ckan-admin/

[PetyaKangalova]

Thanks @gtkChop ! Sharing screenshot for reference for others.

It is indeed 23:00, so maybe on that instance the data was refreshed just after the Archiver run.

[PetyaKangalova]

@gtkChop we should look into this as seems that issue is reoccurring. The latest file update has not yet been picked up.

[PetyaKangalova]

@gtkChop I think this is a bug. I just checked and the metadata has not refreshed. Another publisher also flagged the same issue- let's review today:

[andylolz]

313 relates to this (I didn’t want to go off-topic on this ticket!)

[ss-bhat]

@PetyaKangalova

Archiver cron job was not running before. Fixed the cron job

Archiver should run as usual from now on. Please let me know if the dataset (link to the dataset) is not updated by tomorrow. Thanks

[PetyaKangalova]

@gtkChop just checked and the metadata has been updated and archiver has run this morning, so looks good! It will probably be good to do a few tests tomorrow/ start of next week and if all still working well, happy to close the ticket before your next sprint review.

[andreaszenasidi]

Checked again this morning, the Archiver ran and the metadata was updated.

[andreaszenasidi]

@gtkChop the Archiver seems to still have an issue. The Oxfam Novid dataset (https://iatiregistry.org/dataset/onl-activity-old) was updated on 17 February 2021 but the metadata was not refreshed.

[ss-bhat]

@andreaszenasidi New version of archiver has been deployed today. It should pick the changes.

Did few tests and the archiver updates the dataset as expected. We need to observe tomorrow on the dataset status.

[PetyaKangalova]

@gtkChop I have just check both activity files for Oxfam Novib: https://iatiregistry.org/publisher/onl

See below that both files were updated on 2021-03-01.

<iati-activities generated-datetime="2021-03-01T23:10:28" version="2.03">
<iati-activity default-currency="EUR" hierarchy="1" humanitarian="0" last-updated-datetime="2021-03-01T00:00:00" 

<iati-activities generated-datetime="2021-03-01T23:40:28" version="2.03">
<iati-activity default-currency="EUR" hierarchy="1" humanitarian="0" last-updated-datetime="2021-03-01T00:00:00"

However, the archiver again didn't pick up the update for the Oxfam Novib activities old.

The archiver has last updated today so not sure why the change was not picked up.

[ss-bhat]

@PetyaKangalova

Thanks for reporting. I did a good amount of logging to find out what's going on.

There is an error in downloading the resource.

Package onl-activity updated with new extras
2021-03-02 06:19:48,171 INFO  [ckanext.iati.archiver] Processing package: onl-activity-old : Count 4397/8312
2021-03-02 06:19:48,397 INFO  [ckanext.iati.archiver] Archiving dataset: onl-activity-old (1 resources)
2021-03-02 06:19:51,281 ERROR [ckanext.iati.archiver] Error downloading resource for dataset onl-activity-old

Reason: Resource content length exceeded the max length.

Current length is : ckanext-archiver.max_content_length=60000000

But the file length is 50784987 it looks like the config value is not taken.

I have scheduled the archiver to run again to see the max length.

[PetyaKangalova]

@gtkChop many thanks for the response. Just to make sure I understand. What you are saying is that the file length is within the limit but it was not picked up, correct? Hence re-running the archiver to check if it gets picked up?

Current length is : ckanext-archiver.max_content_length=60000000

But the file length is 50784987 it looks like the config value is not taken.

I have scheduled the archiver to run again to see the max length.

[ss-bhat]

@PetyaKangalova Yes right.

I did fix the file length. But I need to make sure if that's the only problem.

May I know when would be next OXFAM file update? So that I can make sure its picked up.Thanks

[PetyaKangalova]

@gtkChop they have daily updates around 11pm- see below.

<iati-activities generated-datetime="2021-03-02T23:10:38" version="2.03"><iati-activity default-currency="EUR" hierarchy="1" humanitarian="0" last-updated-datetime="2021-03-02T00:00:00" xml:lang="en">

[ss-bhat]

@PetyaKangalova Thanks that's great. I will monitor this see if the archiver pickups this again today.

[ss-bhat]

@PetyaKangalova

Looks like the dataset is updated. Could you please verify? Thanks

[PetyaKangalova]

@gtkChop thanks. The file was updated yesterday at "2021-03-03T23:10:02" .

<iati-activities generated-datetime="2021-03-03T23:10:02" version="2.03"><iati-activity default-currency="EUR" hierarchy="1" humanitarian="0" last-updated-datetime="2021-03-03T00:00:00" xml:lang="en"

The archiver has last refreshed 8pm so that is why the latest update was not picked up? Is there a reason why it didn't run at 23:00 yesterday or today?

[ss-bhat]

@PetyaKangalova Yes it has been set to 20:00 to test archiver on the latest data.

It is set back to 23:00 again. Thanks

[andreaszenasidi]

@gtkChop @PetyaKangalova Monitored this in the past few days and it's updating fine. Closing it.