search

sepinf-inc / IPED

IPED Digital Forensic Tool. It is an open source software that can be used to process and analyze digital evidence, often seized at crime scenes by law enforcement or in a corporate investigation by private examiners.
Other
967 stars 220 forks source link

ArrayIndexOutOfBoundsException: Index -65536 out of bounds for length 71428 at org.apache.lucene.index.TermsHashPerField.writeByte #2322

Open luizedum opened 1 month ago

luizedum commented 1 month ago

The following error was presented by IPED 4.1.6 during the processing of evidence in E01 format (parts of the file names was replaced as they contained personal information):

2024-09-11 11:44:34 [INFO] [task.index.IndexTask$DocumentsIterable$1] Worker-15 Splitting text of /ABC.E01/PASTAS_E_SUBPASTAS/ARQ_ZIP.zip>>PASTAS_E_SUBPASTAS_2/APPLE/Extração/conta@icloud.com/Apple iCloud (Backup)/conta@icloud.com.ufdr>>report.xml>>report.xml_0 2024-09-11 11:44:34 [INFO] [engine.core.Worker] Worker-15 interrupted on /ABC.E01/PASTAS_E_SUBPASTAS/ARQ_ZIP.zip>>PASTAS_E_SUBPASTAS_2/APPLE/Extração/conta@icloud.com/Apple iCloud (Backup)/conta@icloud.com.ufdr>>report.xml>>report.xml_0 (1950114112 bytes) 2024-09-11 11:44:35 [ERROR] [app.processing.Main] Processing Error: java.lang.Exception: Worker-15 Error while processing /ABC.E01/PASTAS_E_SUBPASTAS/ARQ_ZIP.zip>>PASTAS_E_SUBPASTAS_2/APPLE/Extração/conta@icloud.com/Apple iCloud (Backup)/conta@icloud.com.ufdr>>report.xml>>report.xml_0 (1950114112bytes) at iped.engine.core.Worker.process(Worker.java:186) ~[iped-engine-4.1.6.jar:?] at iped.engine.core.Worker.run(Worker.java:265) ~[iped-engine-4.1.6.jar:?] Caused by: java.lang.ArrayIndexOutOfBoundsException: Index -65536 out of bounds for length 71428 at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:219) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13] at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:241) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13] at org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:86) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13] at org.apache.lucene.index.FreqProxTermsWriterPerField.newTerm(FreqProxTermsWriterPerField.java:127) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13] at org.apache.lucene.index.TermsHashPerField.initStreamSlices(TermsHashPerField.java:175) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13] at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:198) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13] at org.apache.lucene.index.IndexingChain$PerField.invert(IndexingChain.java:1224) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13] at org.apache.lucene.index.IndexingChain.processField(IndexingChain.java:729) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13] at org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:620) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13] at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:241) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13] at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13] at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1532) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13] at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1503) ~[lucene-core-9.2.0.jar:9.2.0 ba8c3a806ada3d7b3c34d408e449a92376a8481b - romseygeek - 2022-05-19 15:10:13] at iped.engine.task.index.IndexTask.process(IndexTask.java:149) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processMonitorTimeout(AbstractTask.java:277) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:192) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.MinIOTask.sendToNextTask(MinIOTask.java:416) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.PythonTask.sendToNextTask(PythonTask.java:278) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.PythonTask.sendToNextTask(PythonTask.java:278) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.PythonTask.sendToNextTask(PythonTask.java:278) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.EmbeddedDiskProcessTask.sendToNextTask(EmbeddedDiskProcessTask.java:116) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.1.6.jar:?] at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.1.6.jar:?] at iped.engine.core.Worker.process(Worker.java:177) ~[iped-engine-4.1.6.jar:?] ... 1 more

lfcnassif commented 1 month ago

Thanks for reporting. Are you able to share the file: /ABC.E01/PASTAS_E_SUBPASTAS/ARQ_ZIP.zip>>PASTAS_E_SUBPASTAS_2/APPLE/Extração/conta@icloud.com/Apple iCloud (Backup)/conta@icloud.com.ufdr>>report.xml

which seems to have caused the exception, so we can try to reproduce the error and validate a possible fix?

luizedum commented 1 month ago

Do you prefer that I share the UFDR file or the 'report.xml' file inside it? What is the best way to share it while ensuring data confidentiality?

Is it possible to process an evidence in IPED ignoring a specific file so that the processing of the other files can be completed?

lfcnassif commented 1 month ago

Do you prefer that I share the UFDR file or the 'report.xml' file inside it?

At first, I think just the report.xml file could be enough.

What is the best way to share it while ensuring data confidentiality?

You can send it directly to my email in my profile.

Is it possible to process an evidence in IPED ignoring a specific file so that the processing of the other files can be completed?

Yes, the procedure below can be used to ignore specific files: https://github.com/sepinf-inc/IPED/wiki/Troubleshooting#processing-frozen

luizedum commented 1 month ago

Thank you, the file was shared via email.