search

IKANOW / Aleph2

The IKANOW v2 meta-database and analytics platform
Apache License 2.0
2 stars 1 forks source link

Data governance - age out logs error on buckets that don't yet have data #50

Open Alex-Ikanow opened 8 years ago

Alex-Ikanow commented 8 years ago

Should just suppress these in HDFS Storage Service, since the dir isn't created until data is written for the first time

015-10-27 16:34:24 [ForkJoinPool.commonPool-worker-1] WARN  DataAgeOutSupervisor:162 - Bucket /bucket/test/analytics_test/timesliced/inputs:  processed error = [File /app/aleph2/data/bucket/test/analytics_test/timesliced/inputs/managed_bucket/import/stored/processed/current does not exist.: FileNotFoundException]:[Hdfs.java:214:org.apache.hadoop.fs.Hdfs$DirListingIterator:<init>][FileContext.java:1440:org.apache.hadoop.fs.FileContext:listStatus][HdfsStorageService.java:603:com.ikanow.aleph2.storage_service_hdfs.services.HdfsStorageService$HdfsDataService:handleAgeOutRequest_Worker][538:lambda$handleAgeOutRequest$61][-1:com.ikanow.aleph2.storage_service_hdfs.services.HdfsStorageService$HdfsDataService$$Lambda$621:apply][HdfsStorageService.java:538:com.ikanow.aleph2.storage_service_hdfs.services.HdfsStorageService$HdfsDataService:handleAgeOutRequest][518][DataAgeOutSupervisor.java:154:com.ikanow.aleph2.data_import_manager.governance.actors.DataAgeOutSupervisor:lambda$null$5][-1:com.ikanow.aleph2.data_import_manager.governance.actors.DataAgeOutSupervisor$$Lambda$241:accept][DataAgeOutSupervisor.java:153:com.ikanow.aleph2.data_import_manager.governance.actors.DataAgeOutSupervisor:lambda$null$6]
Alex-Ikanow commented 8 years ago

Example printout:

2015-11-24 08:27:41 [aleph2-akka.actor.default-dispatcher-2] INFO  DataAgeOutSupervisor:139 - DataAgeOutSupervisor checking age out on 16 bucket(s)
2015-11-24 08:27:41 [aleph2-akka.actor.default-dispatcher-2] WARN  DataAgeOutSupervisor:162 - Bucket /alerts/batch/test:  raw: deleted 0 directories; json error = [File /app/aleph2/data/alerts/batch/test/managed_bucket/import/stored/json/current does not exist.: FileNotFoundException]:[Hdfs.java:214:org.apache.hadoop.fs.Hdfs$DirListingIterator:<init>][FileContext.java:1440:org.apache.hadoop.fs.FileContext:listStatus][HdfsStorageService.java:641:com.ikanow.aleph2.storage_service_hdfs.services.HdfsStorageService$HdfsDataService:handleAgeOutRequest_Worker][572:lambda$handleAgeOutRequest$74][-1:com.ikanow.aleph2.storage_service_hdfs.services.HdfsStorageService$HdfsDataService$$Lambda$628:apply][HdfsStorageService.java:572:com.ikanow.aleph2.storage_service_hdfs.services.HdfsStorageService$HdfsDataService:handleAgeOutRequest][556][DataAgeOutSupervisor.java:154:com.ikanow.aleph2.data_import_manager.governance.actors.DataAgeOutSupervisor:lambda$null$5][-1:com.ikanow.aleph2.data_import_manager.governance.actors.DataAgeOutSupervisor$$Lambda$266:accept][DataAgeOutSupervisor.java:153:com.ikanow.aleph2.data_import_manager.governance.actors.DataAgeOutSupervisor:lambda$null$6]; processed: deleted 0 directories
2015-11-24 08:27:41 [aleph2-akka.actor.default-dispatcher-2] WARN  DataAgeOutSupervisor:162 - Bucket /alerts/streaming/test:  raw error = [File /app/aleph2/data/alerts/streaming/test/managed_bucket/import/stored/raw/current does not exist.: FileNotFoundException]:[Hdfs.java:214:org.apache.hadoop.fs.Hdfs$DirListingIterator:<init>][FileContext.java:1440:org.apache.hadoop.fs.FileContext:listStatus][HdfsStorageService.java:641:com.ikanow.aleph2.storage_service_hdfs.services.HdfsStorageService$HdfsDataService:handleAgeOutRequest_Worker][568:lambda$handleAgeOutRequest$72][-1:com.ikanow.aleph2.storage_service_hdfs.services.HdfsStorageService$HdfsDataService$$Lambda$626:apply][HdfsStorageService.java:568:com.ikanow.aleph2.storage_service_hdfs.services.HdfsStorageService$HdfsDataService:handleAgeOutRequest][556][DataAgeOutSupervisor.java:154:com.ikanow.aleph2.data_import_manager.governance.actors.DataAgeOutSupervisor:lambda$null$5][-1:com.ikanow.aleph2.data_import_manager.governance.actors.DataAgeOutSupervisor$$Lambda$266:accept][DataAgeOutSupervisor.java:153:com.ikanow.aleph2.data_import_manager.governance.actors.DataAgeOutSupervisor:lambda$null$6]; json error = [File /app/aleph2/data/alerts/streaming/test/managed_bucket/import/stored/json/current does not exist.: FileNotFoundException]:[Hdfs.java:214:org.apache.hadoop.fs.Hdfs$DirListingIterator:<init>][FileContext.java:1440:org.apache.hadoop.fs.FileContext:listStatus][HdfsStorageService.java:641:com.ikanow.aleph2.storage_service_hdfs.services.HdfsStorageService$HdfsDataService:handleAgeOutRequest_Worker][572:lambda$handleAgeOutRequest$74][-1:com.ikanow.aleph2.storage_service_hdfs.services.HdfsStorageService$HdfsDataService$$Lambda$628:apply][HdfsStorageService.java:572:com.ikanow.aleph2.storage_service_hdfs.services.HdfsStorageService$HdfsDataService:handleAgeOutRequest][556][DataAgeOutSupervisor.java:154:com.ikanow.aleph2.data_import_manager.governance.actors.DataAgeOutSupervisor:lambda$null$5][-1:com.ikanow.aleph2.data_import_manager.governance.actors.DataAgeOutSupervisor$$Lambda$266:accept][DataAgeOutSupervisor.java:153:com.ikanow.aleph2.data_import_manager.governance.actors.DataAgeOutSupervisor:lambda$null$6]; processed: deleted 0 directories
2015-11-24 08:27:41 [aleph2-akka.actor.default-dispatcher-2] WARN  DataAgeOutSupervisor:162 - Bucket /alex/test/netflow-batch/sample:  raw error = [File /app/aleph2/data/alex/test/netflow-batch/sample/managed_bucket/import/stored/raw/current does not exist.: FileNotFoundException]:[Hdfs.java:214:org.apache.hadoop.fs.Hdfs$DirListingIterator:<init>][FileContext.java:1440:org.apache.hadoop.fs.FileContext:listStatus][HdfsStorageService.java:641:com.ikanow.aleph2.storage_service_hdfs.services.HdfsStorageService$HdfsDataService:handleAgeOutRequest_Worker][568:lambda$handleAgeOutRequest$72][-1:com.ikanow.aleph2.storage_service_hdfs.services.HdfsStorageService$HdfsDataService$$Lambda$626:apply][HdfsStorageService.java:568:com.ikanow.aleph2.storage_service_hdfs.services.HdfsStorageService$HdfsDataService:handleAgeOutRequest][556][DataAgeOutSupervisor.java:154:com.ikanow.aleph2.data_import_manager.governance.actors.DataAgeOutSupervisor:lambda$null$5][-1:com.ikanow.aleph2.data_import_manager.governance.actors.DataAgeOutSupervisor$$Lambda$266:accept][DataAgeOutSupervisor.java:153:com.ikanow.aleph2.data_import_manager.governance.actors.DataAgeOutSupervisor:lambda$null$6]; json: deleted 0 directories; processed error = [File /app/aleph2/data/alex/test/netflow-batch/sample/managed_bucket/import/stored/processed/current does not exist.: FileNotFoundException]:[Hdfs.java:214:org.apache.hadoop.fs.Hdfs$DirListingIterator:<init>][FileContext.java:1440:org.apache.hadoop.fs.FileContext:listStatus][HdfsStorageService.java:641:com.ikanow.aleph2.storage_service_hdfs.services.HdfsStorageService$HdfsDataService:handleAgeOutRequest_Worker][576:lambda$handleAgeOutRequest$76][-1:com.ikanow.aleph2.storage_service_hdfs.services.HdfsStorageService$HdfsDataService$$Lambda$630:apply][HdfsStorageService.java:576:com.ikanow.aleph2.storage_service_hdfs.services.HdfsStorageService$HdfsDataService:handleAgeOutRequest][556][DataAgeOutSupervisor.java:154:com.ikanow.aleph2.data_import_manager.governance.actors.DataAgeOutSupervisor:lambda$null$5][-1:com.ikanow.aleph2.data_import_manager.governance.actors.DataAgeOutSupervisor$$Lambda$266:accept][DataAgeOutSupervisor.java:153:com.ikanow.aleph2.data_import_manager.governance.actors.DataAgeOutSupervisor:lambda$null$6]