GBIFResultsReader

[niconoe]

IT seems GBIF has changed its export format, very that it still works (mainly row.source_metadata...)

Also new files included that we should support ?

[tucotuco]

I have been exercising GBIFResultsReader against an archive file downloaded from GBIF today. The downloaded file does not contain metadata.xml, so the GBIFResultsReader throws an exception. The stack trace is below. If there is no guarantee that the downloaded archive will have a metadata.xml file, the GBIFResultsReader should trap for that.

Traceback (most recent call last): File "dwca_utils.py", line 347, in main() File "dwca_utils.py", line 269, in main dwcareader = GBIFResultsReader(options.dwca_file) File "/Library/Python/2.7/site-packages/dwca/read.py", line 234, in init super(GBIFResultsReader, self).init(path) File "/Library/Python/2.7/site-packages/dwca/read.py", line 87, in init self.metadata = self._parse_metadata_file() File "/Library/Python/2.7/site-packages/dwca/read.py", line 168, in _parse_metadata_file return self._parse_xml_included_file(self.descriptor.metadata_filename) File "/Library/Python/2.7/site-packages/dwca/read.py", line 173, in _parse_xml_included_file return BeautifulSoup(self._read_additional_file(relative_path), "xml") File "/Library/Python/2.7/site-packages/dwca/read.py", line 162, in _read_additional_file return open(p).read() IOError: [Errno 2] No such file or directory: u'/Users/johnwieczorek/Projects/kurator-validation/src/main/python/org/kurator/validation/actors/dwca/v/metadata.xml'

[niconoe]

Thanks for the feedback John!

I can indeed fix this, but the more fundamental problem I have here is that the GBIF export format is changing quite often, and without notification. It's therefore a bit difficult to follow, especially if I want to keep compatibility with old downloads.

I sometimes had the opinion that given that given those difficulties and the fact that GBIFResultsReader doesn't add many things to DwCAReader, it could be more wise to deprecate it and instead make the base class more flexible (for example, to get access to any arbitrary additional file in an archive). In my opinion, the most interesting feature may be the per-row metadata...

What's your opinion about this? What specific features of GBIFResultsReader do you consider most useful? Any other recommandations?

thanks

[tucotuco]

I understand the difficulty. One would hope that some stability could be reached. I was most interested in the occurrence data within the archive at the moment, just as with other archives. I do think it is a service to be able to process GBIF downloads. Maybe an easy way forward for now would be simply to relax the requirement of the metadata.xml file. If it is not there, simply ignore that part. That would allow the Reader to be functional again.

[niconoe]

Hi John,

I was a bit confused after looking at the code, so I just generated a download from GBIF, and I suspect a bug on their side: the generated archive's metafile start with <archive xmlns="http://rs.tdwg.org/dwc/text/" metadata="metadata.xml">, but doesn't actually include any metadata.xml, hence the error...

According to http://rs.tdwg.org/dwc/terms/guides/text/index.htm, the metadata attribute is not required, so I guess they should simply remove it at generation time, and everything should work!

What do you think? I think the best is to fix it on their side than to circumvent it here, but what I could do to improve python-dwca-reader is throw an InvalidArchiveException (or something similar) in that case. Improving invalid archives handling was alrady on my to do list, this can be a first step!

[niconoe]

I reported the possible issue at GBIF:

http://dev.gbif.org/issues/browse/PF-2125

[tucotuco]

Good discovery, and good to report it. Throwing an exception seems the best thing to do under the circumstances.

[niconoe]

Hi John,

It's now implemented. I released a new 0.6.5 version (available at PyPI) that throws a dwca.exceptions.InvalidArchive exception when opening such an archive. I didn't try, but you should be able to catch and ignore this exception, and you still should be able to work with the archive's content. Hope this helps while GBIF fix the issue on their side!

[tucotuco]

Interestingly, I did a new GBIF download today and it contains the metadata.xml file. The exception handling is a good addition. This does everything I need it to now.