Closed danfruehauf closed 9 years ago
@danfruehauf that's weird. Just ran the harvester locally on my machine with this file only, it works fine
I guess you'll have to debug it then?
and /mnt/ebs/talend/jobs/srs_oc_soop_rad_rc-srs_oc_soop_rad_rc/log/console.log doesn't show anything
and same on prod nsp9
@danfruehauf well this is my 2cents :
This harvester shouldn't be ! 1) it doesn't exist in https://github.com/aodn/harvesters/tree/master/workspace so I don't even know how it is chef managed 2) all this is managed by this harvester https://github.com/aodn/harvesters/tree/master/workspace/SRS_OC_SOOP_RAD
BUT I have the feeling this is the way it is because of the AODN website which doesn't use the same layers.
Any ideas about this @smancini ?
Its an old harvester written a long time ago to create layers for the old portals (now used by the AODN portal only) and for reporting. Rather than maintaining this old harvester, we should be sourcing the AODN layers and reporting data from the new harvester. This has been an outstanding task for some time.
There are a few other 'legacy harvesters' like this:
soop_asf_harvest-soop_asf_harvest-0-1
soop_ba_harvest-soop_ba_harvest-0-1
soop_frrf_harvest-soop_frrf_harvest-0-1
soop_sst_harvest-soop_sst_harvest-0-1
soop_tmv_harvest-soop_tmv_harvest-0-1
soop_trv_harvest-soop_trv_harvest-0-1
soop_xbt_harvest-soop_xbt_harvest-0-1
srs-srs_harvest-0-1
abos-abos_harvest-0-1
anmn_delayed-anmn_harvest-0-1
anmn_realtime-anmn_harvest-0-1
although the anmn and abos harvesters were/are? used for reporting only
The source code for all these harvesters is in one Talend project NETCDFHARVESTER which is in a private repo due to embedded login credentials and hasn't been migrated to the harvesters repo because they should be deleted instead (once replaced).
@jonescc yes, still using the abos and both anmn harvesters for reporting.
So the verdict? We fix it or chuck it?
@xhoenner
Are you using the dw_ schemas for these. or reporting from these harvesters normal schemas ?
@julian1 only using the dw_anmn_realtime, dw_anmn, dw_abos, and dw_aatams_acoustic schemas
@xhoenner
Are the reporting tables created by liquibase in the dw schemas, named differently/uniquely to the tables in the non dw schemas?
If so, then we could change the harvester config so the reporting talend connection is the same as the destination connection, and channel evertything into its correct schema.
Then we could delete all these specific dw schemas (as well as the unused dw schemas for the harvesters named above).
@julian1 the reporting tables in the dw_ schemas are basically a materialised view of a view in the corresponding non dw_schemas.
If so, then we could change the harvester config so the reporting talend connection is the same as the destination connection, and channel evertything into its correct schema.
Sounds like a good option to me, go ahead and I'll re-point my SQL queries for reporting once that's done.
Due to this file: