search

hotosm / oam-uploader-api

DEPRECIATED - The OAM Uploader API server
BSD 3-Clause "New" or "Revised" License
3 stars 4 forks source link

Do workers clean up after themselves? #58

Closed mojodna closed 7 years ago

mojodna commented 7 years ago

43 is likely the root cause ("Image 0" is 6.8GB), but I was expecting subsequent files to succeed (2 and 3 are ~1GB).

(I'm actually happy that they didn't, as it means that I didn't end up with an incomplete dataset.)

https://upload.openaerialmap.org/#/status/58655b07f91c99bd00e9c7ab

image

smit1678 commented 7 years ago

@mojodna Yeah, I think the issue is that the workers don't clean up after each image. The problem on this upload was the first dataset filled up the temp storage and then the second dataset couldn't process. Regardless of any new deployment of OAM with bigger temp storage, this should be handled another probably.

nbumbarger commented 7 years ago

@mojodna @smit1678 The uploader API is supposed to clean up temporary files- they're explicitly deleted on either success or failure, and the processing is also cached in a special Linux directory where files are overwriteable when not locked.

Because of the new conversion step, though, processing uses at least 2x the disk space as previously, because gdal_translate needs to hold the input file while it writes the output file. If the input is a jp2 image, the output tiff will be much larger, so processing will actually use much more than double the original file size. I'm not familiar with gdal_translate's internal workings, but it's also very possible that the program creates a large processing swapfile when memory is exceeded.

I hadn't tested using large enough files to expose storage limitations, but I think that increasing the disk size to accommodate the new overhead should fix the issue- unless there's an error that's preventing proper cleanup, I can't think of a way to stretch the storage space more than we currently are while converting.

smit1678 commented 7 years ago

Related to #43, I redeployed the API with larger storage. I'm going to leave this open for a bit until we test with the new direct uploads and conversion process to make sure there isn't any edge issues that will cause problems in the future.

Next step here will be to watch the processing of new imagery and check if files are not cleaned at the appropriate time.

smit1678 commented 7 years ago

Workers are cleaning up after themselves and I don't think this is an issue at the moment. Closing.