search

aodn / content

Tracks AODN Portal content and configuration issues
0 stars 0 forks source link

talend srs failing #48

Closed danfruehauf closed 9 years ago

danfruehauf commented 10 years ago

Due to this file:

http://thredds.aodn.org.au/thredds/dodsC/IMOS/SRS/SRS-OC/SRS-OC-SOOP_Rad/VMQ9273_Solander/2014/IMOS_SRS-OC_F_20140312T030831Z_VMQ9273_FV01_DALEC_20140312T032541Z_C-20140629T231726Z.nc
lbesnard commented 10 years ago

@danfruehauf that's weird. Just ran the harvester locally on my machine with this file only, it works fine

danfruehauf commented 10 years ago

I guess you'll have to debug it then?

lbesnard commented 10 years ago

and /mnt/ebs/talend/jobs/srs_oc_soop_rad_rc-srs_oc_soop_rad_rc/log/console.log doesn't show anything

lbesnard commented 10 years ago

and same on prod nsp9

lbesnard commented 10 years ago

@danfruehauf well this is my 2cents :

This harvester shouldn't be ! 1) it doesn't exist in https://github.com/aodn/harvesters/tree/master/workspace so I don't even know how it is chef managed 2) all this is managed by this harvester https://github.com/aodn/harvesters/tree/master/workspace/SRS_OC_SOOP_RAD

BUT I have the feeling this is the way it is because of the AODN website which doesn't use the same layers.

Any ideas about this @smancini ?

jonescc commented 10 years ago

Its an old harvester written a long time ago to create layers for the old portals (now used by the AODN portal only) and for reporting. Rather than maintaining this old harvester, we should be sourcing the AODN layers and reporting data from the new harvester. This has been an outstanding task for some time.

There are a few other 'legacy harvesters' like this:

soop_asf_harvest-soop_asf_harvest-0-1 soop_ba_harvest-soop_ba_harvest-0-1 soop_frrf_harvest-soop_frrf_harvest-0-1 soop_sst_harvest-soop_sst_harvest-0-1 soop_tmv_harvest-soop_tmv_harvest-0-1 soop_trv_harvest-soop_trv_harvest-0-1 soop_xbt_harvest-soop_xbt_harvest-0-1 srs-srs_harvest-0-1 abos-abos_harvest-0-1
anmn_delayed-anmn_harvest-0-1 anmn_realtime-anmn_harvest-0-1

although the anmn and abos harvesters were/are? used for reporting only

jonescc commented 10 years ago

The source code for all these harvesters is in one Talend project NETCDFHARVESTER which is in a private repo due to embedded login credentials and hasn't been migrated to the harvesters repo because they should be deleted instead (once replaced).

xhoenner commented 10 years ago

@jonescc yes, still using the abos and both anmn harvesters for reporting.

danfruehauf commented 10 years ago

So the verdict? We fix it or chuck it?

julian1 commented 10 years ago

@xhoenner

Are you using the dw_ schemas for these. or reporting from these harvesters normal schemas ?

xhoenner commented 10 years ago

@julian1 only using the dw_anmn_realtime, dw_anmn, dw_abos, and dw_aatams_acoustic schemas

julian1 commented 10 years ago

@xhoenner

Are the reporting tables created by liquibase in the dw schemas, named differently/uniquely to the tables in the non dw schemas?

If so, then we could change the harvester config so the reporting talend connection is the same as the destination connection, and channel evertything into its correct schema.

Then we could delete all these specific dw schemas (as well as the unused dw schemas for the harvesters named above).

xhoenner commented 9 years ago

@julian1 the reporting tables in the dw_ schemas are basically a materialised view of a view in the corresponding non dw_schemas.

If so, then we could change the harvester config so the reporting talend connection is the same as the destination connection, and channel evertything into its correct schema.

Sounds like a good option to me, go ahead and I'll re-point my SQL queries for reporting once that's done.

danfruehauf commented 9 years ago

https://github.com/aodn/chef-private/pull/1458