AUV_VIEWER harvester databags issue

aodn / harvesters

Harvesters

GNU General Public License v3.0

0 stars 0 forks source link

AUV_VIEWER harvester databags issue #108

Closed lbesnard closed 10 years ago

lbesnard commented 10 years ago

This harvester has been deployed on NSP14.

But the databags are incorrect. The sourceDir context should point to what is actually setup in the harvester, ie:

/mnt/ebs/wip/AUV/auv_viewer_process

instead of

sourceDir=>/mnt/imos-t3/IMOS/opendap/AUV

from /mnt/ebs/talend/jobs/auv_viewer_track_rc-auv_viewer_track_rc/etc/auv_viewer_track_rc-auv_viewer_track_rc.conf

However it seems like /mnt/ebs/wip/AUV/auv_viewer_process is only accessible through NSP10 so the harvester would have some difficulties to read the data anyway

@danfruehauf , @anguss00 ?

danfruehauf commented 10 years ago

Source dir should not be pointing to a temporary WIP directory. WIP directory is temporary and local.

lbesnard commented 10 years ago

@danfruehauf well that's what we talked about a few weeks ago, and that's why you created this 'special' WIP directory

I've sent your email back

danfruehauf commented 10 years ago

WIP is "Work In Progress", a synonym to /tmp. I never said it will be shared between VMs. Harvesters must not rely on that WIP directory ever.

lbesnard commented 10 years ago

@anguss00 databags can point now to : /mnt/imos-t4/IMOS/public/AUV/auv_viewer_data

thanks

danfruehauf commented 10 years ago

Is this the auv_viewer_track harvester?

danfruehauf commented 10 years ago

BTW, where are the thumbnails going to be?

lbesnard commented 10 years ago

yeap this is it. Thumbnails are in /mnt/imos-t4/IMOS/public/AUV/auv_viewer_data/thumbnails

But they aren't used by the harvester

lbesnard commented 10 years ago

@danfruehauf if Angus is not here today, could you (or someone else) change the databags to point to the right folder ?

lbesnard commented 4 days ago @anguss00 databags can point now to : /mnt/imos-t4/IMOS/public/AUV/auv_viewer_data

thanks

danfruehauf commented 10 years ago

Yeah, lets do it now?

lbesnard commented 10 years ago

whenever mate. I'd really like to move forward with this

danfruehauf commented 10 years ago

Try please with:

include_data=>.*TABLE_DATA.*\.csv
include_metadata=>.*TABLE_METADATA.*\.csv
include_reporting=>.*auvReporting\.csv

lbesnard commented 10 years ago

@danfruehauf please see https://github.com/aodn/harvesters/issues/108#issuecomment-61438214

also you got to remove excludeDir (and just use exclude) since this doesn't exist in the talend component. Already had a chat with @jonescc about this. The harvester might be slow going through millions of excluded files, so will have to see how this performs.

lbesnard commented 10 years ago

@anguss00 Bene merged a new version of my harvester so if you can put this one on RC it would be great. However even the current version should work with those databags. I tried on my machine with the exact same path as what is on prod and everything is ok. I don't get why the schema on RC is completely empty.