search

aodn / data-services

Scripts which are used to process incoming data in the data ingestion pipeline
GNU General Public License v3.0
1 stars 4 forks source link

Incoming files not detected? #440

Closed mhidas closed 6 years ago

mhidas commented 8 years ago

There are 112 files in /mnt/ebs/incoming/ANMN/QLD, uploaded between 16:30 and 18:30 last night, which have not triggered the incoming handler.

smancini commented 8 years ago

There are 935 files in /mnt/ebs/incoming/AOCRN/any, uploaded since yesterday afternoon which have also not triggered the incoming handler.

mhidas commented 8 years ago

I suspect none of the pipelines are being triggered. The last entry in /mnt/ebs/log/data-services/process.log.1 is Apr 28 16:12

mhidas commented 8 years ago

Look like the incoming directories were cleared out (i.e. files handled) yesterday, and pipelines running ok now.

@anguss00 Any ideas what went wrong? Could it be related to https://github.com/aodn/internal-discussions/issues/223 ?

julian1 commented 8 years ago

I cleared up space at 7.33am on the main partition / not /mnt/ebs/

root@10-aws-syd:~$ less /var/log/apt/history.log
Start-Date: 2016-05-02  07:33:19
Commandline: apt-get autoremove

It looks like the processing was immediately kicked off,

root@10-aws-syd:~$ head /mnt/ebs/log/data-services/process.log
May  2 07:33:19 10-aws-syd AATAMS_SATTAG_DM: Handling lftp file '/mnt/ebs/tmp/tmp.ric1YeQk97/aatams_sattag_dm_lftp.20160502-000044.log'
May  2 07:33:19 10-aws-syd AATAMS_SATTAG_DM: Handling '0' deletions
May  2 07:33:19 10-aws-syd AATAMS_SATTAG_DM: Handling '0' additions
May  2 07:33:19 10-aws-syd SOOP_XBT_NRT: Handling rsync file '/mnt/ebs/tmp/tmp.Q8qy1e2dme/IMOS_SOOP-XBT_NRT_fileList.csv'
May  2 07:33:19 10-aws-syd AATAMS_SATTAG_DM: Successfully handled all AATAMS_SATTAG_DM files!
May  2 07:33:19 10-aws-syd SOOP_XBT_NRT: Handling '10' additions
May  2 07:33:19 10-aws-syd SOOP_XBT_NRT: Bulk indexing/unindexing files from '/mnt/ebs/tmp/tmp.VL9NCyi142'
May  2 07:33:19 10-aws-syd AATAMS_SATTAG_NRT: Handling rsync file '/mnt/ebs/tmp/tmp.kI8NnSoLDN/manifest'
May  2 07:33:19 10-aws-syd AATAMS_SATTAG_NRT: Handling '0' additions
May  2 07:33:19 10-aws-syd AATAMS_SATTAG_NRT: Successfully handled all aatams nrt files!
julian1 commented 8 years ago

It looks like there's a lot of files in /tmp that should really be in /mnt/ebs/tmp

root@10-aws-syd:~$ ls -lht  $( find /tmp/ -type f   ) | grep \.nc

-rw-r--r-- 1 root           root             73K May  2 11:00 /tmp/sync_archive.log
-rw-rw-r-- 1 projectofficer projectofficer  3.7M May  1 20:14 /tmp/tmpay1Fdo.pid6817.ncwa.tmp
-rw-rw-r-- 1 projectofficer projectofficer  6.3M May  1 20:14 /tmp/tmpRDEzox/IMOS_SOOP-TRV_B_20160323T140000Z_VNCF_FV01_END-20160407T210727Z_C-20160501T101326Z.nc
-rw-rw-r-- 1 projectofficer projectofficer  7.3M May  1 20:13 /tmp/tmpbnQkHe/IMOS_SOOP-TRV_B_20160302T140000Z_VNCF_FV01_END-20160320T230800Z_C-20160501T101210Z.nc
-rw-rw-r-- 1 projectofficer projectofficer  3.4M May  1 20:11 /tmp/tmpM9bbwv/IMOS_SOOP-TRV_B_20160126T140000Z_VNCF_FV01_END-20160203T031335Z_C-20160501T101119Z.nc
-rw-rw-r-- 1 projectofficer projectofficer  7.8M May  1 20:11 /tmp/tmpYkV8P8/IMOS_SOOP-TRV_B_20151129T140000Z_VNCF_FV01_END-20151217T020450Z_C-20160501T101005Z.nc
-rw-rw-r-- 1 projectofficer projectofficer  3.9M May  1 20:10 /tmp/tmpvl3oNp/IMOS_SOOP-TRV_B_20151118T140000Z_VNCF_FV01_END-20151127T034052Z_C-20160501T100928Z.nc
-rw-rw-r-- 1 projectofficer projectofficer  4.4M May  1 20:09 /tmp/tmpcZbVVZ/IMOS_SOOP-TRV_B_20151104T140000Z_VNCF_FV01_END-20151116T003500Z_C-20160501T100847Z.nc
-rw-rw-r-- 1 projectofficer projectofficer  2.3M May  1 20:08 /tmp/tmpXijlc1/IMOS_SOOP-TRV_B_20151023T140000Z_VNCF_FV01_END-20151029T053942Z_C-20160501T100825Z.nc
julian1 commented 8 years ago

@mhidas list of current issues,

To note, on debian based systems like Ubuntu, it appears that the tmp dir is not cleaned up by cron but by reboot. see, http://serverfault.com/questions/377348/when-does-tmp-get-cleared

julian1 commented 8 years ago

Note that this was a serious issue. Files were not processed over the weekend (although none were lost).

bpasquer commented 8 years ago

No need to fix anfog_dm. I have to deal with the files in error_dir . However issue with file sitting in anfog_rt incoming dir

julian1 commented 8 years ago

Cleaned old dirs out of /tmp to free up more space

 mv $( find /tmp/ -ctime +2 -type d ) /mnt/ebs/tmp/from_root_tmp/
mhidas commented 8 years ago

@julian1 Please make /mnt/ebs/tmp/from_root_tmp/ read/writeable to project officers so that we can remove files once we've figured out why they were in /tmp (and made sure they don't end up there again). At the moment even just running du to see how much data is in there results in a bunch of "Permission denied" messages.

julian1 commented 8 years ago

@mhidas I've changed the owner to projectofficer. Let me know if that's not enough.

mhidas commented 8 years ago

:+1: Thanks @julian1

pblain commented 8 years ago

@julian1 , @mhidas - are you sure that's a good idea? Would read only access be sufficient? Pipeline processes shouldn't be writing to tmp right?

julian1 commented 8 years ago

@pblain, /mnt/ebs/tmp/from_root_tmp/ is a directory that contains files that were purged out of /tmp that should not have been there, and which were stale and lost to the system.

I gave project officers ownership of this subdir only - to enable them to manually evaluate and process, and then to remove that directory.

No pipeline jobs should be using the /tmp directory. Unfortunately all talend jobs appear to.

pblain commented 8 years ago

@julian1 - makes sense. Thanks!

julian1 commented 8 years ago

Summary for DOD

pmbohm commented 6 years ago

@mhidas Is this bug still relevant? Its on our board to do this iteration :)

lbesnard commented 6 years ago

euh , this is over 2 years old. I don't quite understand why it is on the board. This is the reason why we moved to pipeline 2