Error processing a forensic copy

[igiuseppem]

Good evening. I acquired an hard disk in E01 format. The original disk is an ext4 format and the operating system seems to be linux. I processed the forensic copy with IPED, but I get an error. Please find attached the log file.

Kind regards IPED-2024-06-06-01-12-45.log

[lfcnassif]

Hi @igiuseppem, seems a missing feature from sleuthkit library for some specific ext4 feature. It is not simple to fix. I suggest you to open the image with FTKImager and, if it opens fine, convert it to AD1 format, which should be processed fine by IPED, bypassing sleuthkit. You may need to create one AD1 per partition, I think creating one AD1 for the whole image is not possible within FTKImager.

[igiuseppem]

Hi @lfcnassif! Yes, it opens fine with FTK Imager (in this way I have seen it is an ext4 file system), but when I want to select the menu "Export Logical Image (AD1)", it is grayed.

[lfcnassif]

I think you need to right click on each partition with your mouse and create one AD1 for each of them, or just for the partitions of interest.

[igiuseppem]

When I right click on the partition I want to export in AD1, there is the "Export Disk Image" I already see in the "File" menu. Here I can't export the image in AD1, but in other kind of format (like dd, aff and so on).

[lfcnassif]

Sorry @igiuseppem, I gave you a wrong procedure, you should right click on the partition file system node, the tree node right below the partition, so the create AD1 option will be enabled and it will include allocated files, orphan files and unallocated space (useful for carving).

[lfcnassif]

[lfcnassif]

Wait, I just saw your log on the computer, earlier I was only able to read it from my mobile...

There are 1294 "java.io.IOException: Spazio su disco insufficiente" errors in the log coming from your temporary processing folder/disk. So, although there is some ext4 feature not supported by sleuthkit, seems the "Spazio su disco insufficiente" errors are the main cause of your issues. Try to use a bigger temp disk to process this image, the original E01 or the converted AD1 images.

[lfcnassif]

Hi @igiuseppem, did using a bigger temp disk help?

[lfcnassif]

Closing as the main issue seems related to temp disk full. The TSK ext4 unsupported feature issue should be reported and addressed in TSK project. If you are still having issues, please reopen.

[igiuseppem]

Hi @igiuseppem, did using a bigger temp disk help?

Hi @lfcnassif , sorry for the delay in the answer. I have more space on the temp disk right now, but there is another problem. The indexing with IPED now stops abruptely and restarts the computer. I tried it two times and both times I found the PC at the login screen, like it was restarted.

Please find attached one of the new log

IPED-2024-06-15-09-21-02.log

[lfcnassif]

Hi @lfcnassif , sorry for the delay in the answer. I have more space on the temp disk right now, but there is another problem. The indexing with IPED now stops abruptely and restarts the computer. I tried it two times and both times I found the PC at the login screen, like it was restarted.

Please find attached one of the new log

IPED-2024-06-15-09-21-02.log

Hi @igiuseppem. I took a look at the log and I didn't find any fatal errors. System reboot could be an operating system or hardware failure. Did you notice how was the memory usage? System reboot may also be caused by memory exhaustion. If IPED used all system memory (ideally this should be confirmed), I suggest you going to conf/FileSystemConfig.txt->numImageReaders and change it from auto (5 in your case) to 1 or 2. Then resume the processing with --continue.

[igiuseppem]

Hi @lfcnassif , sorry for the delay in the answer. I have more space on the temp disk right now, but there is another problem. The indexing with IPED now stops abruptely and restarts the computer. I tried it two times and both times I found the PC at the login screen, like it was restarted. Please find attached one of the new log IPED-2024-06-15-09-21-02.log

Hi @igiuseppem. I took a look at the log and I didn't find any fatal errors. System reboot could be an operating system or hardware failure. Did you notice how was the memory usage? System reboot may also be caused by memory exhaustion. If IPED used all system memory (ideally this should be confirmed), I suggest you going to conf/FileSystemConfig.txt->numImageReaders and change it from auto (5 in your case) to 1 or 2. Then resume the processing with --continue.

About the memory usage, I didn't notice it. I'll try what you suggest.

[igiuseppem]

Hi @lfcnassif. About 10 hours later (and right now), the usage of the RAM was at 98% / 99%. There is also a crash error window from firefox. The indexing seems to going on.

[lfcnassif]

Was a *.hprof memory dump generated in the terminal current folder? If yes, could you zip and share it? If not, try to create one running jstack -l pid

[igiuseppem]

No, there is not a *.hprof memory dump, neither in the main IPED folder, neither in the destination folder. The system is very slow. I will create the dump with the command you wrote, but I would like to stop the indexing, because the time is increasing instead of the decreasing, despite it has about 3000 files to completion. I want to stop it, but I would like to resume later. Can I stop closing only the progress window? I want to try resetting the numImageReaders parameter.

[lfcnassif]

No, there is not a *.hprof memory dump, neither in the main IPED folder, neither in the destination folder.

Usually it is created in the current user folder, the folder where your terminal is opened when the IPED command was run.

Can I stop closing only the progress window?

I suggest pausing the processing, waiting some seconds or minutes, then close the window.

[igiuseppem]

Usually it is created in the current user folder, the folder where your terminal is opened when the IPED command was run.

I confirm there is not a memory dump. The problem persists, despite I tried to change the numImageReaders parameter, setting it to 1 and 2. The RAM is always saturated. I runned jstack -l pid over 4 "java" process. Please find attached the dump. Note that IPED was paused when I runned jstack.

dump1.txt dump2.txt dump3.txt dump4.txt

[wladimirleite]

Hi @igiuseppem! Can you post a screenshot of IPED's processing window? It may be useful to help figuring out what is going on.

[paulobreim]

hi @igiuseppem ,

One suggestion, try reducing the number of threads in the LocalConfig.txt file, where it is numThreads = default. For example, half of the cores that you have available. Should use less memory.

[wladimirleite]

One suggestion, try reducing the number of threads in the LocalConfig.txt file, where it is numThreads = default. For example, half of the cores that you have available. Should use less memory.

In general, that is a good suggestion to reduce memory usage. However it is not clear (at least to me) if there are memory issues.

@igiuseppem, when you said RAM usage was very high (99%), you meant total RAM, right? (not JVM's memory usage) Which processes are using more memory (a screenshot of TaskManager "details" tab, like the one below, should help):

[lfcnassif]

However it is not clear (at least to me) if there are memory issues.

Neither to me, that was just a wild guess from me. A screenshot of the processing window, taken together with the thread dumps (jstack -l pid) would help a lot to.know why processing is taking to long.

[igiuseppem]

@igiuseppem, when you said RAM usage was very high (99%), you meant total RAM, right? (not JVM's memory usage)

@wladimirleite Yes, the total amount of RAM is 99%, but it happens only when I process this forensic copy. Never had this kind of problem.

One suggestion, try reducing the number of threads in the LocalConfig.txt file, where it is numThreads = default.

@paulobreim I tried to use only half the total threads. Same result: RAM at 99% in short time, as in the case I used the total amount of threads.

Neither to me, that was just a wild guess from me. A screenshot of the processing window, taken together with the thread dumps (jstack -l pid) would help a lot to.know why processing is taking to long.

@lfcnassif I have taken pictures (a screenshot was not possible, because the system was too slow and operating system didn't respond) together with the thread dumps. It is a new indexing, beacause I had to stop the previous, as I needed my workstation for other tasks.

dump1.txt dump2.txt dump3.txt dump4.txt dump5.txt dump6.txt dump7.txt

[wladimirleite]

Thanks @igiuseppem! A last request from my side: in the picture you took from the task manager, can you switch to "Detail" view (it is in the general "process" view) and sort by memory (not by CPU)?

[wladimirleite]

Complementing my previous comment, from the IPED processing window picture, it seems that most of JVM memory was free (6301 of 8098 MB). And the Task Manager shows 98% total RAM used, but java process was only using ~3 GB. Other visible processes are not using much memory. So something else in consuming the memory, possibly external processes started by IPED (e.g. ImageMagick). In the "Detail" window, sorted by memory used, we may see some useful information.

[igiuseppem]

Hi @wladimirleite. In the picture you see, the task manager is already sorted by memory (decreasing order). If you need the details, I can take a picture, but I have to restart again the indexing and doing again also the dumps.

[lfcnassif]

@lfcnassif I have taken pictures (a screenshot was not possible, because the system was too slow and operating system didn't respond) together with the thread dumps. It is a new indexing, beacause I had to stop the previous, as I needed my workstation for other tasks.

Thanks @igiuseppem, I took a look at the thread dumps, seems to me the processing was not frozen, but just slow because it was running OCR, which is a heavy task, it may take 4s per image and much more for multipage PDFs. Wasn't the number of processed files increasing slowly?

I don't see explicit memory issues here, so my initial wild guess about it should be wrong. I don't have tips why your machine was restarting, maybe the Windows event logs could help you to figure it out.

[lfcnassif]

So something else in consuming the memory, possibly external processes started by IPED (e.g. ImageMagick)

AFAIK, in the task manager screenshot sent by @igiuseppem, the memory used by all external processes called by IPED should be taken into account in the OpenJDK Platform processes group.

[igiuseppem]

@lfcnassif I have taken pictures (a screenshot was not possible, because the system was too slow and operating system didn't respond) together with the thread dumps. It is a new indexing, beacause I had to stop the previous, as I needed my workstation for other tasks.

Thanks @igiuseppem, I took a look at the thread dumps, seems to me the processing was not frozen, but just slow because it was running OCR, which is a heavy task, it may take 4s per image and much more for multipage PDFs. Wasn't the number of processed files increasing slowly?

I don't see explicit memory issues here, so my initial wild guess about it should be wrong. I don't have tips why your machine was restarting, maybe the Windows event logs could help you to figure it out.

Dear @lfcnassif, that was the problem. I disabled OCR and using all threads IPED finished in about 10 hours.

Wasn't the number of processed files increasing slowly?

Yes, the number of processed files was increasing very very very slowly. IPED stood still processing a file for a long time (maybe more than an hour), when there were around 3000 files left to finish. Now that I was able to finish the decoding, I saw that the files total around 1700000. With the previous indexing activities on this forensic copy, when IPED was stuck on that file, the total number of files was something like 1006989 and the hours per term increased instead of decreasing. The file on which the processing had focused the most (then I had to stop) was one of the "unallocs". The previous times I decoded the forensic copy with IPED, was generated a blue screen. One of these times, the error was related to the graphic card. I guess that it was because it was executing the OCR on that file it was stuck on. Am I right?

[lfcnassif]

Dear @lfcnassif, that was the problem. I disabled OCR and using all threads IPED finished in about 10 hours.

If you disabled OCR in IPED "installation" folder and then resumed processing with --continue, that doesn't work, all configurations are initially copied to output folder and loaded from there, you should change configs in output folder so they can take effect when resuming.

Yes, the number of processed files was increasing very very very slowly.

So processing was not frozen.

The file on which the processing had focused the most (then I had to stop) was one of the "unallocs". (...) I guess that it was because it was executing the OCR on that file it was stuck on. Am I right?

Actually IPED was recovering lots of deleted files from unallocated space files, according to your screenshot and thread dumps. Then it was running OCR on those thousands of recovered files (images and PDFs), so the slowness is expected in this situation.