SOOP_AUSCPR harvester updates required - Githubissues

aodn / harvesters

Harvesters

GNU General Public License v3.0

0 stars 0 forks source link

SOOP_AUSCPR harvester updates required #67

Closed mhidas closed 10 years ago

mhidas commented 10 years ago

The following changes are required:

make timestamps consistent (with timezone)
harvest two additional tables from CMAR Geoserver:
- imos:AUSCPR_PHYTO_TAXON_CHANGELOG
- imos:AUSCPR_ZOOP_TAXON_CHANGELOG
join TAXON_CHANGELOG tables to data tables in the _data views using the TAXON_NAME column as a foreign key
Add component to update spatial extent in metadata record
Hope this all makes sense!

lbesnard commented 10 years ago

Everything done as requested, not committed yet.

However I don't quite get how I could join in a nice way the tables together. for example

select COUNT(distinct taxon_name) from soop_auscpr.soop_auscpr_zoop_taxon_changelog

returns 121

and

select COUNT(distinct taxon_name) from soop_auscpr.soop_auscpr_zoop_trajectory_map

returns 616.

So the join of the two table is not a full one since a lot of taxon names appear to be missing in the *_CHANGELOG table from CMAR geoserver

ggalibert commented 10 years ago

You could do a left join on the soop_auscpr_zoop_taxon_changelog table so that you always get a result from soop_auscpr_zoop_taxon_changelog (either matching or null) for each taxon_name in soop_auscpr_zoop_trajectory_map.

Would that be acceptable?

lbesnard commented 10 years ago

@ggalibert that's actually what I ended up doing, see https://github.com/aodn/harvesters/tree/auscpr_update just waiting for someone to review the PR

ggalibert commented 10 years ago

For the next time, it would have actually been easier to review if you had made distinct commits for each of Marty's point above.

ggalibert commented 10 years ago

Do you really need timestamp(6) ~ 6 digits in the fractional part of second? I can see that you also use the default timestamp type.

ggalibert commented 10 years ago

In your iUpdateSpatialExtent component, the field GeoNetwork URL should be set to context.Metadata_URL

https://docs.google.com/document/d/1-E05wDMvI_Yy2Zz9m3D8DCWUjjkhPDJFm_jr9TvbUgc/edit#heading=h.onsdd540oals

ggalibert commented 10 years ago

I'm not sure the updateMetadata component can be used with a tPostJob like you did.

Documentation mentioned above says "Add the subjob to run after harvesting in the main harvester job (making sure transmit whole context is ticked)", which tickbox you haven't ticked...

ggalibert commented 10 years ago

I can't find out how you did this left outer join.

lbesnard commented 10 years ago

@mhidas can we close this ?

mhidas commented 10 years ago

Yes, I just merged your pull request #69

mhidas commented 10 years ago

Thanks Loz for getting it all done!