Implement harvester client for PDN and GEM documents and media

[giohappy]

Both the remote sources are exposed as PostgREST API (GEM implementation still in progress). For the moment the priority is the metadata retrieval.

PDN

http://www.pacificdisaster.net/api/

Dcouments endpoints:

• document/
• document/id=eq.

Media endpoints

• media/

Files for PDN documents and media will be retrieved from the http://www.pacificdisaster.net/doc base url, by appending the filename reported inside the document/ and media/ responses.

GEM

TBD (waiting for news from SPC)

[ricardogsilva]

@giohappy

for /media I'm not sure how to proceed:

• The response includes a filename property and apparently all media files use the .flv extension, which is for the discontinued Flash Video format. Using PostgREST API to filter for other media types that may be in there:

  http://www.pacificdisaster.net/api/media?filename=not.like.*flv

  I get back zero results. This means all records under /media are flv files. AFAIK there is no native support for this format in modern web browsers, so users need some additional player (like VLC) or maybe a browser extension to be able to view them

• The API response does not include a full URL to the actual media file, and I've not been able to guess it - I've tried a couple of combinations of URLs and none of these allowed me to download one of these flv files

• The main website for PDN does not seem to expose these media files anywhere either, which suggests that they may not be too relevant, or at least not a priority for the time being

Should we continue to spend effort on these?

[ricardogsilva]

@giohappy

For GEM, are you aware of any progress on the availability of its planned PostgREST API?

[giohappy]

Sachindra has been informed about the problems reported by @ricardogsilva

[giohappy]

Sachindra replied as below:

The media files are quite old, and we can safely ignore them for a while. We will upload latest ones manually.

So we will skip the import of media fiels from PDN.