search

sepinf-inc / IPED

IPED Digital Forensic Tool. It is an open source software that can be used to process and analyze digital evidence, often seized at crime scenes by law enforcement or in a corporate investigation by private examiners.
Other
962 stars 219 forks source link

'#1861 Workaround to overcome path of folders names with trailing spaces. #1888

Closed patrickdalla closed 1 year ago

patrickdalla commented 1 year ago

Workaround to overcome path of folders names with trailing spaces that lead to IPED processing interruption.

lfcnassif commented 1 year ago

Hi @patrickdalla, please test last commits I pushed with your test samples. If they are ok and you think they are enough, I'll merge this, thanks!

PS: Maybe we can refactor FolderTreeReader class to try to read skipped files in a separate ticket, to avoid introducing regressions in a minor fix.

lfcnassif commented 1 year ago

Hi @patrickdalla. About commit 506a808, have you tested current code (without it) with more recent JDK versions? We plan to move to java 17 soon (or maybe the next LTS), and maybe that commit wouldn't be needed, not sure...

lfcnassif commented 1 year ago

Hi @patrickdalla, thank you very much for your commits! Just one more question, the new FileAttributes class is still returning null in some methods, like those related to Dates and others, shouldn't it call the corresponding method from the wrapped class? Or was it intentional?

patrickdalla commented 1 year ago

@lfcnassif Sorry, I think I don't know how to use correctly the github code review. I have responded directly at each review you pointed out, but it appears to me as "Pending" and it seems you haven't received them as you asked again. Anyway, follows the explanation about FileAttributes class:

The forward to the wrapped path would raise Windows Exception, as they call the win APIs that checks path name conformity. This class was done exactly to avoid this calls.

With this "fake" BasicFilesAttributes, the exception is not raise, and the file walker continues inside the folder, allowing to process its children. Unfortunatelly, the file attributes are not read from nfts.

We could implement this bypassing windows api and java nio implementation for these specific files. Some considerable effort should be done so, maybe, it should not be the scope of this specific issue/PR, in my opinion.

lfcnassif commented 1 year ago

@lfcnassif Sorry, I think I don't know how to use correctly the github code review. I have responded directly at each review you pointed out, but it appears to me as "Pending" and it seems you haven't received them as you asked again.

Oh... yes, I didn't receive your answers. Don't worry, this also happened to me in the past. You should click on "Finish Review" button or something similar, so all your comments will be sent. I can't see them here, I hope they are not lost.

The forward to the wrapped path would raise Windows Exception, as they call the win APIs that checks path name conformity. This class was done exactly to avoid this calls.

With this "fake" BasicFilesAttributes, the exception is not raise, and the file walker continues inside the folder, allowing to process its children. Unfortunatelly, the file attributes are not read from nfts.

Right, got it now! But null is returned just for those problematic folders ending with spaces, right?

patrickdalla commented 1 year ago

Yes. The code to check is line 146 of FileSystemProviderWrapper:

public <A extends BasicFileAttributes> A readAttributes(Path path, Class<A> type, LinkOption... options)
        throws IOException {
    A result;
    try {
        if (path.toString().endsWith(" ")) {
            return (A) new FileAttributes((PathWrapper) path);
        }
        result = wrapped.readAttributes(((PathWrapper) path).getWrappedPath(), type, options);
    }catch (Exception e) {
        return (A) new FileAttributes((PathWrapper)path);
    }
    return  result;     
}
lfcnassif commented 1 year ago

Just got this error while testing last commit (just fixes indentation):

java.lang.Exception: Worker-18 Error while processing spaces/dir2 /s.txt     (33bytes)
        at iped.engine.core.Worker.process(Worker.java:186) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.core.Worker.run(Worker.java:265) ~[iped-engine-4.2-snapshot.jar:?]
Caused by: java.nio.file.InvalidPathException: Trailing char < > at index 26: f:\teste-files\spaces\dir2 \s.txt
        at sun.nio.fs.WindowsPathParser.normalize(Unknown Source) ~[?:?]
        at sun.nio.fs.WindowsPathParser.parse(Unknown Source) ~[?:?]
        at sun.nio.fs.WindowsPathParser.parse(Unknown Source) ~[?:?]
        at sun.nio.fs.WindowsPath.parse(Unknown Source) ~[?:?]
        at sun.nio.fs.WindowsFileSystem.getPath(Unknown Source) ~[?:?]
        at java.nio.file.Path.of(Unknown Source) ~[?:?]
        at iped.utils.FileInputStreamFactory.isDirectory(FileInputStreamFactory.java:61) ~[iped-utils-4.2-snapshot.jar:?]
        at iped.utils.FileInputStreamFactory.getPath(FileInputStreamFactory.java:37) ~[iped-utils-4.2-snapshot.jar:?]
        at iped.utils.IOUtil.getFile(IOUtil.java:416) ~[iped-utils-4.2-snapshot.jar:?]
        at iped.engine.data.Item.getTikaStream(Item.java:633) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.ParsingTask.safeProcess(ParsingTask.java:380) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.ParsingTask.process(ParsingTask.java:329) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processMonitorTimeout(AbstractTask.java:277) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:192) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.core.Worker.process(Worker.java:177) ~[iped-engine-4.2-snapshot.jar:?]
        ... 1 more
patrickdalla commented 1 year ago

@lfcnassif Commit 3476f36 solved the problem for me. Please retest.

Ps.: I made some mistakes when applying the commit. I hope I haven't undone the changes of your prevous commit 625420e.

lfcnassif commented 1 year ago

@lfcnassif Commit 3476f36 solved the problem for me. Please retest.

Thanks. Unfortunately it keeps failing in my test samples:

2023-10-26 10:48:48     [ERROR] [app.processing.Main]                   Processing Error:
java.lang.Exception: Worker-1 Error while processing spaces/mydirectory /b.txt  (0bytes)
        at iped.engine.core.Worker.process(Worker.java:186) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.core.Worker.run(Worker.java:265) ~[iped-engine-4.2-snapshot.jar:?]
Caused by: java.nio.file.InvalidPathException: Trailing char < > at index 33: f:\teste-files\spaces\mydirectory
        at sun.nio.fs.WindowsPathParser.normalize(Unknown Source) ~[?:?]
        at sun.nio.fs.WindowsPathParser.parse(Unknown Source) ~[?:?]
        at sun.nio.fs.WindowsPathParser.parse(Unknown Source) ~[?:?]
        at sun.nio.fs.WindowsPath.parse(Unknown Source) ~[?:?]
        at sun.nio.fs.WindowsFileSystem.getPath(Unknown Source) ~[?:?]
        at java.io.File.toPath(Unknown Source) ~[?:?]
        at iped.utils.FileInputStreamFactory.isDirectory(FileInputStreamFactory.java:63) ~[iped-utils-4.2-snapshot.jar:?]
        at iped.utils.FileInputStreamFactory.getPath(FileInputStreamFactory.java:38) ~[iped-utils-4.2-snapshot.jar:?]
        at iped.utils.IOUtil.getFile(IOUtil.java:416) ~[iped-utils-4.2-snapshot.jar:?]
        at iped.engine.data.Item.getTikaStream(Item.java:633) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.ParsingTask.safeProcess(ParsingTask.java:380) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.ParsingTask.process(ParsingTask.java:329) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processMonitorTimeout(AbstractTask.java:277) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:192) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.core.Worker.process(Worker.java:177) ~[iped-engine-4.2-snapshot.jar:?]
        ... 1 more

Ps.: I made some mistakes when applying the commit. I hope I haven't undone the changes of your prevous commit https://github.com/sepinf-inc/IPED/commit/625420e7ff1d79dbe274595fa30acd2c85e9cd04.

You haven't, it seem ok.

patrickdalla commented 1 year ago

It seems you have a file with trailing space name inside a dir with trailing name space, right? I will have to make more loops to overcome all these path parts.

lfcnassif commented 1 year ago

It seems you have a file with trailing space name inside a dir with trailing name space, right?

Yes.

I think this is becoming much more complex than expected. What do you think about reverting all NIO wrapping classes added, removing them, and just printing an error in the Console if a file or directory ending with space is found and can't be processed fine, warning the user explicitly about it, so he has a chance to fix the file/folder name?

If user wants a forensically sound processing, he should create an image (DD or AD1) instead of processing a mounted folder...

patrickdalla commented 1 year ago

yes. it can be a solution

Em qui., 26 de out. de 2023 10:30, Luis Filipe Nassif < @.***> escreveu:

It seems you have a file with trailing space name inside a dir with trailing name space, right?

Yes.

I think this is becoming much more complex than expected. What do you think about reverting all NIO wrapping classes added, removing them, and just printing an error in the Console if a file or directory ending with space is found and can't be processed fine, warning the user explicitly about it, so he has a chance to fix the file/folder name?

If user wants a forensically sound processing, he should create an image (DD or AD1) instead of processing a mounted folder...

— Reply to this email directly, view it on GitHub https://github.com/sepinf-inc/IPED/pull/1888#issuecomment-1781247880, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG247S2R5UT43QKSKKIVNQDYBJXWTAVCNFSM6AAAAAA45CKC5OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBRGI2DOOBYGA . You are receiving this because you were mentioned.Message ID: @.***>

patrickdalla commented 1 year ago

Anyway, I have implemented a filevisitor to get a PathWrapper to the path being tested by isDirectory. I have created a folder structure with two folders inside one another and with trailing space names, a a file with trailing space. It could process all the files in the folder structure except those that has trailing space names itself.

But, as you have said, it became complex for a workaround, that can be solved using IPED in a more forensic way.

image

lfcnassif commented 1 year ago

Thanks! Unfortunately it keeps aborting:

2023-10-26 14:14:20     [ERROR] [app.processing.Main]                   Processing Error:
java.lang.Exception: Worker-7 Error while processing spaces/mydirectory /b.txt  (0bytes)
        at iped.engine.core.Worker.process(Worker.java:186) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.core.Worker.run(Worker.java:265) ~[iped-engine-4.2-snapshot.jar:?]
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.io.FileNotFoundException: \\?\F:\teste-files\spaces\mydirectory \b.txt (The system cannot find the file specified)
        at iped.utils.FileInputStreamFactory.getPath(FileInputStreamFactory.java:61) ~[iped-utils-4.2-snapshot.jar:?]
        at iped.utils.IOUtil.getFile(IOUtil.java:416) ~[iped-utils-4.2-snapshot.jar:?]
        at iped.engine.data.Item.getTikaStream(Item.java:633) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.ParsingTask.safeProcess(ParsingTask.java:380) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.ParsingTask.process(ParsingTask.java:329) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processMonitorTimeout(AbstractTask.java:277) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:192) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.core.Worker.process(Worker.java:177) ~[iped-engine-4.2-snapshot.jar:?]
        ... 1 more
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: \\?\F:\teste-files\spaces\mydirectory \b.txt (The system cannot find the file specified)
        at iped.utils.FileInputStreamFactory.getPath(FileInputStreamFactory.java:57) ~[iped-utils-4.2-snapshot.jar:?]
        at iped.utils.IOUtil.getFile(IOUtil.java:416) ~[iped-utils-4.2-snapshot.jar:?]
        at iped.engine.data.Item.getTikaStream(Item.java:633) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.ParsingTask.safeProcess(ParsingTask.java:380) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.ParsingTask.process(ParsingTask.java:329) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processMonitorTimeout(AbstractTask.java:277) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:192) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.core.Worker.process(Worker.java:177) ~[iped-engine-4.2-snapshot.jar:?]
        ... 1 more
Caused by: java.io.FileNotFoundException: \\?\F:\teste-files\spaces\mydirectory \b.txt (The system cannot find the file specified)
        at java.io.FileInputStream.open0(Native Method) ~[?:?]
        at java.io.FileInputStream.open(Unknown Source) ~[?:?]
        at java.io.FileInputStream.<init>(Unknown Source) ~[?:?]
        at iped.utils.FileInputStreamFactory.getPath(FileInputStreamFactory.java:52) ~[iped-utils-4.2-snapshot.jar:?]
        at iped.utils.IOUtil.getFile(IOUtil.java:416) ~[iped-utils-4.2-snapshot.jar:?]
        at iped.engine.data.Item.getTikaStream(Item.java:633) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.ParsingTask.safeProcess(ParsingTask.java:380) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.ParsingTask.process(ParsingTask.java:329) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processMonitorTimeout(AbstractTask.java:277) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:192) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.sendToNextTask(AbstractTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:205) ~[iped-engine-4.2-snapshot.jar:?]
        at iped.engine.core.Worker.process(Worker.java:177) ~[iped-engine-4.2-snapshot.jar:?]
        ... 1 more

I'm thinking to revert all changes to commit 11123b350084dbd8fd69656a377afa60d0f3001a. Could you please check it out and test it on your test data and tell if the result is acceptable? Here the result is:

2023-10-26 14:17:16     [ERROR] [engine.datasource.FolderTreeReader$FolderVisitor]                      File ignored: f:\teste-files\spaces\dir\b.txt : java.nio.file.NoSuchFileException: f:\teste-files\spaces\dir\b.txt
2023-10-26 14:17:16     [ERROR] [engine.datasource.FolderTreeReader$FolderVisitor]                      File ignored: f:\teste-files\spaces\dir\b.txt : java.nio.file.NoSuchFileException: f:\teste-files\spaces\dir\b.txt
2023-10-26 14:17:16     [ERROR] [engine.datasource.FolderTreeReader$FolderVisitor]                      File ignored: f:\teste-files\spaces\dir2 : java.nio.file.NoSuchFileException: f:\teste-files\spaces\dir2
2023-10-26 14:17:16     [ERROR] [engine.datasource.FolderTreeReader$FolderVisitor]                      File ignored: f:\teste-files\spaces\dir2 : java.nio.file.NoSuchFileException: f:\teste-files\spaces\dir2
2023-10-26 14:17:16     [ERROR] [engine.datasource.FolderTreeReader$FolderVisitor]                      File ignored: f:\teste-files\spaces\dir3 : java.nio.file.NoSuchFileException: f:\teste-files\spaces\dir3
2023-10-26 14:17:16     [ERROR] [engine.datasource.FolderTreeReader$FolderVisitor]                      File ignored: f:\teste-files\spaces\dir3 : java.nio.file.NoSuchFileException: f:\teste-files\spaces\dir3
2023-10-26 14:17:16     [ERROR] [engine.datasource.FolderTreeReader$FolderVisitor]                      File ignored: f:\teste-files\spaces\mydirectory : java.nio.file.NoSuchFileException: f:\teste-files\spaces\mydirectory
2023-10-26 14:17:16     [ERROR] [engine.datasource.FolderTreeReader$FolderVisitor]                      File ignored: f:\teste-files\spaces\mydirectory : java.nio.file.NoSuchFileException: f:\teste-files\spaces\mydirectory

and processing finishes.

lfcnassif commented 1 year ago

Hi @patrickdalla, I put a workaround to fix last error above. However, after creating files with different content and equal names (except for the trailing space) in the same directory, the tool is returning the same hash values...

So, I will close this PR and create another one with initial commits until commit https://github.com/sepinf-inc/IPED/commit/11123b350084dbd8fd69656a377afa60d0f3001a. The tool will skip problematic files and folders and print an error in the console to warn users, like I posted above, and continue processing.

In this rare situation, if users want to process the files properly in a forensic way, they can create a forensic image from their data.

Thank you very much for your efforts here.

lfcnassif commented 1 year ago

Please, just test if PR https://github.com/sepinf-inc/IPED/pull/1952 finishes processing your test data instead of aborting, thanks.