lfcnassif
commented
10 months ago Hi @tc-wleite!
@lfcnassif, does my analysis make sense? Did you try/consider this approach (avoid deleting the parent temporary file before carved childs are processed)?
Yes and yes! Actually processing carved files immediately should be the default behavior to reuse the parent temp file and to take advantage of CPU internal caches, as far as I remember. But when the processing queue has a few items (likely to become empty soon), there is a rule in Worker class to enqueue carved files aiming to better use the CPU cores. Possibly that is allowing the parent temp file deletion. I think there is some protection, AFAIK in Item class dispose() method and other calling places I don't remember exactly (and I knew it isn't perfect, so the catch and comment you quoted above exists). Anyway I thought it was a very rare situation and I thought it wasn't causing such a large impact noticed and measured by you. So thank you very much for catching this, testing and debugging the issue! I'm curious about your solution!
PS: Original behavior of container expansion was also to process subitems immediately to take advantage of system/hardware caches if processing queue wasn't empty or almost empty. Unfortunatelly that was causing issues with JEP and I changed the behavior to always enqueue container subitems.
wladimirleite
While working on other issues, I noticed that processing most of a UFDR, the last items (mostly carved) took more time than I expected. I/O and CPU usage was low, which usually indicates that something could be improved.
After some investigation, I found that what was consuming extra time was the fact that carved items (from a few SQLite databases) couldn't access the temporary parent file because it was already deleted. Many items were falling in the exception shown below.
https://github.com/sepinf-inc/IPED/blob/5f5ab2d3f6a17fa4825830c62127095c1fe2c2b6/iped-engine/src/main/java/iped/engine/data/Item.java#L547-L557
So for each carved item, it seems that the parent file had to be unpacked (from the UFDR), so the carved piece can be accessed. There wasn't any disk activity because everything was in the SO cache, but still took an extra time per item.
In another part of the code, I noticed that if the processing queue is long (more than 100K items), carved files are processed as soon as possible, so usually their parent temporary file is available. Otherwise (which was my case), they are placed in the queue and there is no guarantee whether the parent temporary file will be there.
I made a code change to force the temporary file to be kept until its carved childs are processed and ran a comparison test with 7 UFDRs and 2 E01s.
The evidence that motivated the investigation was the first one, for which the processing time was reduced from ~26 minutes to ~4 minutes. It was the most extreme difference. For other evidence files (2, 4 and 7) there are no differences at all, but for the rest there was some noticeable improvement.
Tests had most of the common tasks enabled, except OCR and audio transcription. The column "Current Parent Temp Miss" is the number of times that the exception shown in the code snippet above was thrown. With the modified, no item threw exceptions at that function.
In some of the cases, the number of "Carved Corrupted" was reduced by a few items. Not sure if it is related, but I was wondering if it was possible that in very rare cases the parent file deletion happens (it is not blocked by "file in use") in a certain point of the carved child processing and causes some parsing exceptions. Not sure if this scenario is even possible...
This happens in a part of the code I was not familiar with... @lfcnassif, does my analysis make sense? Did you try/consider this approach (avoid deleting the parent temporary file before carved childs are processed)?