search

dlr-eoc / prosEO

prosEO – A Processing System for Earth Observation Data
GNU General Public License v3.0
14 stars 1 forks source link

Storage Manager: No error result if file to ingest not found #95

Closed tangobravo62 closed 2 years ago

tangobravo62 commented 4 years ago

Currently the Storage Manager silently ignores files, which are not found on the input storage during ingestion:

2020-09-07 11:20:08.379  INFO 1 --- [nio-8080-exec-5] d.d.p.s.rest.ProductControllerImpl       : de.dlr.proseo.storagemgr.rest.model.RestProductFS@678893a3[productId=4927,sourceStorageType=POSIX,sourceFilePaths=[/mnt/./AUX/ICM_CKDSIR/2019/06/01/S5P_OPER_ICM_CKDSIR_20141001T000001_20501231T235959_00000_02_020000_20190601T080000.h5],targetStorageId=<null>,targetStorageType=POSIX,registeredFilePath=,registered=,registeredFilesCount=,registeredFilesList=[],deleted=,message=]
2020-09-07 11:20:08.379 TRACE 1 --- [nio-8080-exec-5] d.d.p.storagemgr.utils.ProseoFilePosix   : ProseoFilePosix created: de.dlr.proseo.storagemgr.utils.ProseoFilePosix@90d5a2c
2020-09-07 11:20:08.379 TRACE 1 --- [nio-8080-exec-5] d.d.p.storagemgr.utils.ProseoFilePosix   : ProseoFilePosix created: de.dlr.proseo.storagemgr.utils.ProseoFilePosix@49cb771d
2020-09-07 11:20:08.380 DEBUG 1 --- [nio-8080-exec-5] d.d.p.storagemgr.utils.ProseoFilePosix   : Copying from /mnt/./AUX/ICM_CKDSIR/2019/06/01/S5P_OPER_ICM_CKDSIR_20141001T000001_20501231T235959_00000_02_020000_20190601T080000.h5 to /proseo/data/4927/
2020-09-07 11:20:08.385 ERROR 1 --- [nio-8080-exec-5] d.d.p.storagemgr.utils.ProseoFilePosix   : Cannot find source file /mnt/./AUX/ICM_CKDSIR/2019/06/01/S5P_OPER_ICM_CKDSIR_20141001T000001_20501231T235959_00000_02_020000_20190601T080000.h5
2020-09-07 11:20:08.388 DEBUG 1 --- [nio-8080-exec-5] d.d.p.storagemgr.utils.ProseoFilePosix   : Files copied: []
2020-09-07 11:20:08.391 DEBUG 1 --- [nio-8080-exec-5] d.d.p.s.rest.ProductControllerImpl       : Files transferred: []
2020-09-07 11:20:08.392 DEBUG 1 --- [nio-8080-exec-5] d.d.p.s.rest.ProductControllerImpl       : Files registered: []
2020-09-07 11:20:08.393 DEBUG 1 --- [nio-8080-exec-5] d.d.p.s.rest.ProductControllerImpl       : Response created: de.dlr.proseo.storagemgr.rest.model.RestProductFS@1b602c56[productId=4927,sourceStorageType=POSIX,sourceFilePaths=[/mnt/./AUX/ICM_CKDSIR/2019/06/01/S5P_OPER_ICM_CKDSIR_20141001T000001_20501231T235959_00000_02_020000_20190601T080000.h5],targetStorageId=proseo/data,targetStorageType=POSIX,registeredFilePath=/proseo/data/4927//,registered=true,registeredFilesCount=0,registeredFilesList=[],deleted=false,message=registration executed on node docker-desktop]

Although the requested file was not found, the Storage Manager still returns with HTTP status code 201 (CREATED). This may be correct, if at least one file could be ingested, but not, if all files failed (as in the example above). Even in the case of a partial ingest the message should return an indication of what went wrong.

tangobravo62 commented 3 years ago

Not quite the case described above, but in the meantime the error handling was improved, and meaningful error messages are returned by the Storage Manager (from Ingestor log):

http-outgoing-11172 << HTTP/1.1 500 Internal Server Error
http-outgoing-11172 << Warning: 199 proseo-storage-mgr (E4001) Exception thrown: class com.amazonaws.AmazonClientException: Unable to complete transfer: Java heap space
tangobravo62 commented 2 years ago

Implemented with commit 2d82d9b.