Closed harryharrold closed 11 years ago
done
Works if end-point works and gives a .zip file with good data.
Fails if a successfully downloaded .zip file fails to uncompress - we retain the data in our triple store. Fails if we do not find a file at http://www.dfid.gov.uk/r4d/rdf/R4DOutputsData.zip - we retain the data in our triple store. Fails spectacularly if a successfully downloaded .zip file uncompresses but does not contain the type of content we expect - at this point, we've deleted the data in our triple store.
@practicalparticipation - could you confirm this is OK, and you've testing this successfully? (And then close this..)
Q: Does this mean it'll go wrong if we upload an empty .zip file or one which contains different formats?
"yes." "fair enough"
Advised that @practicalparticipation will do some backup work....
An RDF dump of the Research for Development data will be updated weekly on at http://www.dfid.gov.uk/r4d/rdf/R4DOutputsData.zip in a number of individual files (approximately 10k records per file). A scheduled process is needed which will:
Download these files; Load them into triple store in a named graph; Log and issues and errors; Report any errors to a defined set of e-mail address;
This file will contain the RDF data files for the R4D data. The file name format is:
R4DOutputsData-[SequenceNumber]-[yyyyMMddhhmmss].rdf
For example, R4DOutputsData-1-20130226052425.rdf.
Each file will contain a triple indicating the date and time that the data dump as a whole was run and the modifications serialised. This will appear in the form:
pointy bracket rdf:Description rdf:about="http://www.dfid.gov.uk/R4D/RDF/R4DOutputsData.zip" dcterms:modified="2013-02-26T05:24:24:004" close pointy bracket