Missing SIDA activities: SE-0

[amy-silcock]

D-portal is currently missing c.20000 activities for SIDA (SE-0).

This is compared to the other IATI tools:

• Dashboard = 86344
• Datastore = 86347

There were previously issues on SIDA's end with file availability but we think that's now been fixed. Is there anything on the d-portal side that could be preventing the files being accessed/used?

Each file contains the correct IATI identifiers and passes validation.

[janerikohman]

Hi all, I work at SIDA with both OpenAid and the generation of the IATI-files. If you have any questions about the files I'd be happy to help. I can also be reached at jan-erik.ohman@sida.se Regards, Jan-Erik

[notshi]

Hi both, many thanks for raising this.

This is the error log for SIDA files during last night's import.

dstore/cache/sida-1031.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-1031.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-1032.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-1032.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-1032.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-1032.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-1032.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-289.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-289.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-298.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-298.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-298.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-589.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-589.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-589.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-589.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-589.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-679.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-679.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-679.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-789.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-789.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-789.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-798.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-798.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-798.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-88.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-88.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-88.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-998a.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-998a.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-998a.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-998a.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-998a.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-998c.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-998c.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-998c.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-998c.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-998c.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-998d.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-998d.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-998d.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-998d.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-998d.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-af.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-af.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-af.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-af.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-al.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-al.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-al.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-al.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-al.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ao.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ao.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ao.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ao.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bf.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-bf.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-bf.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bf.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bf.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bi.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bi.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bo.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bo.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bo.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bo.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bo.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bt.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-bt.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-bt.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-bt.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bt.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bw.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-bw.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-bw.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bw.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bw.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cd.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cd.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cd.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cd.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cd.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cl.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cl.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cl.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ec.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-ec.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-ec.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-ec.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ec.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-et.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-et.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-et.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-et.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-et.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-fj.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-fj.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-fj.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-fj.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-gy.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-gy.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-gy.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-gy.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-gy.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-hn.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-hn.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-hn.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-hu.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-hu.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-hu.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-iq.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-iq.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-iq.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-jm.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-jm.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-jm.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-jm.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-jm.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ke.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-ke.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ke.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ke.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ke.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kr.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-kr.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-kr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kw.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-kw.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-kw.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kw.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kw.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kz.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-kz.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-kz.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-kz.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kz.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lb.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-lb.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lb.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lb.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lb.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ls.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-ls.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-ls.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ls.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ls.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lv.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-lv.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-lv.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lv.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lv.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-me.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-me.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-me.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-me.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-me.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mk.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mk.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ml.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ml.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mn.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mn.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mn.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mn.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mn.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mx.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mx.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mx.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-my.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-my.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-my.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-my.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-na.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-na.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-np.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-np.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-pg.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-pg.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-pg.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ph.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-ph.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-ph.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ph.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ph.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-py.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-py.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-py.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-py.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-py.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ro.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ro.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ro.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-rs.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-rs.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-rs.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-rs.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-rs.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-rw.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-rw.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-rw.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-rw.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-rw.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-saintlucia.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-saintlucia.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-saintlucia.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-saintlucia.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sb.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sb.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sb.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sd.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-sd.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-sd.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sd.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sd.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sk.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sk.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sl.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-sl.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-sl.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sl.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sl.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-so.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-so.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-so.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-so.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-so.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ss.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ss.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ss.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ss.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-st.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-st.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-st.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sv.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-sv.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-sv.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sv.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sv.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sz.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sz.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-tg.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-tg.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-tg.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-tg.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-tg.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-tj.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-tj.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-tj.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-tj.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-tj.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-tn.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-tn.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-tn.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-tn.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-tn.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ua.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-vu.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-vu.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-vu.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-vu.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-zw.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-zw.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-zw.xml.curl.last.log:curl: (22) The requested URL returned error: 503 

Looks like there are errors concerning the SIDA server and access to it. Here is the list of errors according to the numbers:

502 Bad Gateway The server was acting as a gateway or proxy and received an invalid response from the upstream server. 503 Service Unavailable The server cannot handle the request (because it is overloaded or down for maintenance). Generally, this is a temporary state. 504 Gateway Timeout The server was acting as a gateway or proxy and did not receive a timely response from the upstream server.

[notshi]

Btw @janerikohman this happened at around 1:05 - 1:08am.

Could you check your server and see what is happening on you end? Thanks in advance.

Also, accessing https://iati.openaid.se/xml/998a.xml is taking a long time on a browser so this could partly explain the errors above.

[janerikohman]

Thanks for the input. I will contact the dev team and ask them to correlate their logs with your errors and times above. This is the same kind of errors that were produced earlier, I thought this was corrected,

I know that there were some issues when we used curl from command prompt and whether it could use compression or not - especially when opening the large files. The 998a.xml file is one of our largest files - it's 20Mb (the largest is 27Mb) - that's probably why a browser open takes a very long time.

[notshi]

Hi @amy-silcock @janerikohman, looks we didn't get an errors from SIDA's server last night and data has populated the portal.

Let me know if that's not the case.

[amy-silcock]

Thanks Shi, great to see them all there! I'll let @janerikohman follow up :)

[notshi]

Looks like we are getting the same errors during last night's import @janerikohman

dstore/cache/sida-1031.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-1031.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-1031.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-1031.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-1031.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-189.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-189.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-289.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-289.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-289.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-289.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-289.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-689.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-689.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-689.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-689.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-689.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-998b.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-998b.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-998e.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-998e.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-998e.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-998e.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-998e.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-al.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-al.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-al.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-al.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-al.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ao.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-ao.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ao.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ao.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ao.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bg.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bg.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bg.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bg.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bi.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bo.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-bo.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-bo.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bo.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bo.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bt.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bt.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-bt.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cd.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cf.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cf.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ci.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-ci.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ci.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ci.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ci.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cm.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-cm.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-cm.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cm.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cm.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cn.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cn.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cn.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cn.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cn.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cr.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-cr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cu.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cu.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-cu.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-dj.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-dj.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-dj.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-dj.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-dj.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-dm.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-dm.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-dm.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-dm.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-dz.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-dz.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-dz.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-dz.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-dz.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ee.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-ee.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-ee.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-ee.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ee.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ga.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ga.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-gm.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-gm.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-gm.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-gm.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-gn.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-gq.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-gq.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-gq.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-gq.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-gt.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-gt.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-gt.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-gt.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-gt.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-gy.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-gy.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-gy.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-gy.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-gy.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-hr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-hr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-hr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-hr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-hr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-id.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-id.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-id.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-id.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-id.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-in.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-in.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-in.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ir.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-ir.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ir.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ir.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ir.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-jm.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-jm.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-jm.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-jm.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-jm.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ke.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kg.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kg.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kg.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kg.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kh.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kh.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kh.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kiribati.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kiribati.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kiribati.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kiribati.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kw.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-kw.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-kw.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kw.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-kw.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lk.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-lk.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lk.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lk.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lk.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lr.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-lr.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-lr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ls.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ls.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ls.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ls.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lv.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-lv.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lv.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lv.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-lv.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ly.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-ly.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ly.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ly.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ly.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-me.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mg.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mg.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mh.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mk.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-mk.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mk.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mk.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mk.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mr.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mw.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mw.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mw.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mw.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mw.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-my.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-my.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-my.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-mz.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ne.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-ne.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ne.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ne.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ne.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ng.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ng.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ng.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ng.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ng.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-pk.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-pk.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-pk.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-pk.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-pk.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ps.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-ps.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-ps.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ps.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ps.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ro.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-rs.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-rs.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-rs.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-rs.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-rs.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-saintlucia.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-saintlucia.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sd.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sd.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sd.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sd.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-si.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-si.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-si.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-si.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-si.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-so.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-so.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-so.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-so.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-so.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-st.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-st.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-st.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-st.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-st.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sv.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-sv.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sv.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sv.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sv.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sy.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-sy.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-sy.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sy.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sy.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sz.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sz.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sz.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sz.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-sz.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-tg.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-tg.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-tg.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-tg.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-tj.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-tj.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-tj.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-tj.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-tj.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-tn.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-tn.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-tn.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-tn.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-tn.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-tz.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-tz.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-tz.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-tz.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-tz.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ug.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ug.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ug.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ug.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-uy.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-vn.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-vn.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-vn.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-vn.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-vn.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-xk.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-xk.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-xk.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-xk.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ye.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-ye.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ye.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ye.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-ye.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-za.xml.curl.last.log:curl: (22) The requested URL returned error: 502
dstore/cache/sida-za.xml.curl.last.log:curl: (22) The requested URL returned error: 504
dstore/cache/sida-za.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-za.xml.curl.last.log:curl: (22) The requested URL returned error: 503
dstore/cache/sida-za.xml.curl.last.log:curl: (22) The requested URL returned error: 503 

The above graph shows the number of activities going down starting from the 15th of October. Then it kept fluctuating, presumably due to the same errors.

Here's a more detailed error log for one of the files:

Mon Jan 18 01:05:39 UTC 2021

Downloading sida-so.xml from http://iati.openaid.se/xml/SO.xml

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 1675 0 --:--:-- --:--:-- --:--:-- 1675
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 502
Warning: Transient problem: HTTP error Will retry in 10 seconds. 4 retries
Warning: left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 3435 0 --:--:-- --:--:-- --:--:-- 3435
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0
curl: (22) The requested URL returned error: 504
Warning: Transient problem: HTTP error Will retry in 10 seconds. 3 retries
Warning: left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 3435 0 --:--:-- --:--:-- --:--:-- 3435
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 503
Warning: Transient problem: HTTP error Will retry in 10 seconds. 2 retries
Warning: left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 3435 0 --:--:-- --:--:-- --:--:-- 3435
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 503
Warning: Transient problem: HTTP error Will retry in 10 seconds. 1 retries
Warning: left.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 3435 0 --:--:-- --:--:-- --:--:-- 3435
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 503 

[janerikohman]

Hi, Thanks for the detailed errors. Do you have any pointers here at GitHub to the code your're running when you receive the error? I guess it's the code at https://github.com/devinit/D-Portal/blob/662f6940438aaebc36839c9c5d024edafd15d739/box/cron-download

Interesting that you receive that error on SO.xml - its 5,7Mb - we have a number of others that are 20+Mb. But it's definitely an error on our side - I will talk to our back-end people and ask them to investigate.

At mid-October we launched our new OpenAid-site - which also contains iati.openaid.se/xml - that's why the errors starts popping up then. Earlier the files were static and served from a web server - now they are being generated on the fly by an API. Which for some reasons returns errors now and then.

[notshi]

Thanks, @janerikohman - yes, the code is at https://github.com/devinit/D-Portal/blob/master/box/cron-download Do let us know of your findings and if we can provide you with anything else.

[notshi]

Hi @janerikohman

Wondering if you had considered using something like Cloudflare to serve cached data on your behalf? This could be an easy fix and an interim solution.

[janerikohman]

Thanks @notshi , I will be sure to mention that to them. They generate files on-the-fly to be able to do transformations if needed, but they should be able to be cached for like an hour or so.

[notshi]

We've been monitoring this issue and it looks like it's working better now after a few tweaks to the server on our end. We've slowed down the requests and we've changed the user-agent from Mozilla/5.0 to the default curl.

This solution was only possible due to some vigorous investigation into other issues that were facing downloading problems and it took numerous attempts to find the combination that worked.

Activities in d-portal http://d-portal.org/ctrack.html#view=search&reporting_ref=SE-0

This seems to be the same number on the Dashboard. http://dashboard.iatistandard.org/publisher/sida.html

[janerikohman]

I don't know why this hasn't been fixed on openaid serverside. But good to see that you don't receive any errors now. But I would expect that it would be 174 files. I will contact our IATI-admin and ask them to have a look.

[notshi]

Have been keeping an eye on the numbers on d-portal and the dashboard, and they seem currently in sync.

Are the number of files as expected now @janerikohman?

[janerikohman]

Hi, I have left SIDA and are no longer working with OpenAid.

I took a quick look at OpenAids page where all files are exposed and there I counted 179 files (https://iati.openaid.se/xml). I will ask Siri Rembe who works at SIDA to have a look at this.

Thanks for the update @notshi.

[notshi]

Thanks, @janerikohman, that would be much appreciated.

It looks like the missing file is https://iati.openaid.se/xml/389.xml

This file is not listed in the registry.

[stevieflow]

This seems to be resolved

d-porta: https://d-portal.org/ctrack.html#view=search&reporting_ref=SE-0 - Found 129,723 activities dashboard: http://dashboard.iatistandard.org/publisher/sida.html Activities 129723 datastore search: Found 130,350 activities - but think that is not related to this specific issue

Therefore: closing