search

markmckinnon / Autopsy-Plugins

Autopsy Python Plugins
332 stars 100 forks source link

Parse_USNJ sqlite error #44

Open vanhalessio opened 2 years ago

vanhalessio commented 2 years ago

Hi (and thanks for you great job on this famous plugins). I'm getting an error in executing, inside autopsy (latest version, but also by running the module parseusn.exe manually from cmd), the module on a E01 image of a relative small disk. The USNJ txt file is around 45GB.

image

vanhalessio commented 2 years ago

Let me add some info. When run as autopsy plugin, this is the log of the operation (the manual running of the parseusn.exe seemed to me more informational, that's why I pasted it first).

2021-11-02 17:31:50.312 ParseUsnJIngestModule process INFO: found 1 files 2021-11-02 17:31:50.313 ParseUsnJIngestModule process INFO: create Directory C:\Users\USER\AppData\Local\Temp\Autopsy\test3_20211030_095055\Temp 2021-11-02 17:32:54.524 ParseUsnJIngestModule process INFO: Saved File ==> C:\Users\USER\AppData\Local\Temp\Autopsy\test3_20211030_095055\Temp\usnj\usnj.txt 2021-11-02 17:32:54.524 ParseUsnJIngestModule process INFO: Running program ==> C:\Users\USER\AppData\Roaming\autopsy\python_modules\Parse_USNJ\parseusn.exe C:\Users\USER\AppData\Local\Temp\Autopsy\test3_20211030_095055\Temp\usnj\usnj.txt C:\Users\USER\AppData\Local\Temp\Autopsy\test3_20211030_095055\Temp\usnj.db3 2021-11-02 17:35:36.668 ParseUsnJIngestModule process INFO: Output from run is ==> usnj is C:\Users\USER\AppData\Local\Temp\Autopsy\test3_20211030_095055\Temp\usnj\usnj.txt DB file is C:\Users\USER\AppData\Local\Temp\Autopsy\test3_20211030_095055\Temp\usnj\usnj.db3 ('Unexpected error:', <class 'sqlite3.ProgrammingError'>)

2021-11-02 17:35:36.669 ParseUsnJIngestModule process INFO: Path the system database file created ==> C:\Users\USER\AppData\Local\Temp\Autopsy\test3_20211030_095055\Temp\usnj\usnj.db3 2021-11-02 17:35:36.672 ParseUsnJIngestModule process INFO: query SQLite Master table 2021-11-02 17:35:36.672 ParseUsnJIngestModule process INFO: Begin Create New Artifacts 2021-11-02 17:35:36.673 ParseUsnJIngestModule process INFO: Artifacts Creation Error, some artifacts may not exist now. ==> 2021-11-02 17:35:41.061 ParseUsnJIngestModule process INFO: removal of usnj directory failed C:\Users\USER\AppData\Local\Temp\Autopsy\test3_20211030_095055\Temp\usnj 2021-11-02 17:35:41.062 org.sleuthkit.autopsy.ingest.DataSourceIngestPipeline$DataSourcePipelineModule performTask INFO: USN Parser analysis of SOURCE.E01 finished 2021-11-02 17:35:41.062 org.sleuthkit.autopsy.ingest.IngestJobPipeline logInfoMessage INFO: Finished first stage analysis (data source = SOURCE.E01, objId = 1, pipeline id = 6, ingest job id = 18) 2021-11-02 17:35:41.062 org.sleuthkit.autopsy.ingest.IngestJobPipeline logInfoMessage INFO: Finished analysis (data source = SOURCE.E01, objId = 1, pipeline id = 6, ingest job id = 18) 2021-11-02 17:35:41.064 org.sleuthkit.autopsy.ingest.IngestManager finishIngestJob INFO: Ingest job 6 completed 2021-11-02 17:35:41.103 org.sleuthkit.autopsy.casemodule.IngestJobInfoPanel$1 done INFO: The refreshing of the IngestJobInfoPanel was cancelled

markmckinnon commented 2 years ago

I believe I know what the issue is and should have a fix for you to test shortly.

On Nov 2, 2021, at 1:08 PM, vanhalessio @.***> wrote:

 Let me add some info. When run as autopsy plugin, this is the log of the operation (the manual running of the parseusn.exe seemed to me more informational, that's why I pasted it first).

2021-11-02 17:31:50.312 ParseUsnJIngestModule process INFO: found 1 files 2021-11-02 17:31:50.313 ParseUsnJIngestModule process INFO: create Directory C:\Users\USER\AppData\Local\Temp\Autopsy\test3_20211030_095055\Temp 2021-11-02 17:32:54.524 ParseUsnJIngestModule process INFO: Saved File ==> C:\Users\USER\AppData\Local\Temp\Autopsy\test3_20211030_095055\Temp\usnj\usnj.txt 2021-11-02 17:32:54.524 ParseUsnJIngestModule process INFO: Running program ==> C:\Users\USER\AppData\Roaming\autopsy\python_modules\Parse_USNJ\parseusn.exe C:\Users\USER\AppData\Local\Temp\Autopsy\test3_20211030_095055\Temp\usnj\usnj.txt C:\Users\USER\AppData\Local\Temp\Autopsy\test3_20211030_095055\Temp\usnj.db3 2021-11-02 17:35:36.668 ParseUsnJIngestModule process INFO: Output from run is ==> usnj is C:\Users\USER\AppData\Local\Temp\Autopsy\test3_20211030_095055\Temp\usnj\usnj.txt DB file is C:\Users\USER\AppData\Local\Temp\Autopsy\test3_20211030_095055\Temp\usnj\usnj.db3 ('Unexpected error:', <class 'sqlite3.ProgrammingError'>)

2021-11-02 17:35:36.669 ParseUsnJIngestModule process INFO: Path the system database file created ==> C:\Users\USER\AppData\Local\Temp\Autopsy\test3_20211030_095055\Temp\usnj\usnj.db3 2021-11-02 17:35:36.672 ParseUsnJIngestModule process INFO: query SQLite Master table 2021-11-02 17:35:36.672 ParseUsnJIngestModule process INFO: Begin Create New Artifacts 2021-11-02 17:35:36.673 ParseUsnJIngestModule process INFO: Artifacts Creation Error, some artifacts may not exist now. ==> 2021-11-02 17:35:41.061 ParseUsnJIngestModule process INFO: removal of usnj directory failed C:\Users\USER\AppData\Local\Temp\Autopsy\test3_20211030_095055\Temp\usnj 2021-11-02 17:35:41.062 org.sleuthkit.autopsy.ingest.DataSourceIngestPipeline$DataSourcePipelineModule performTask INFO: USN Parser analysis of SOURCE.E01 finished 2021-11-02 17:35:41.062 org.sleuthkit.autopsy.ingest.IngestJobPipeline logInfoMessage INFO: Finished first stage analysis (data source = SOURCE.E01, objId = 1, pipeline id = 6, ingest job id = 18) 2021-11-02 17:35:41.062 org.sleuthkit.autopsy.ingest.IngestJobPipeline logInfoMessage INFO: Finished analysis (data source = SOURCE.E01, objId = 1, pipeline id = 6, ingest job id = 18) 2021-11-02 17:35:41.064 org.sleuthkit.autopsy.ingest.IngestManager finishIngestJob INFO: Ingest job 6 completed 2021-11-02 17:35:41.103 org.sleuthkit.autopsy.casemodule.IngestJobInfoPanel$1 done INFO: The refreshing of the IngestJobInfoPanel was cancelled

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

vanhalessio commented 2 years ago

thank you very much, waiting for the testing :)

shannaniggans commented 1 year ago

Running this plugin with Autopsy 4.20.0 and still getting the same error:

2023-05-01 06:34:27.818 org.sleuthkit.autopsy.ingest.IngestManager startIngestJob
INFO: Starting ingest job 7 at 1682922867818
2023-05-01 06:34:27.824 org.sleuthkit.autopsy.ingest.IngestJobExecutor logInfoMessage
INFO: Starting ingest job in file batch mode (data source = MUS-CTF-19-DESKTOP-001.E01, data source object ID = 1, job ID = 7)
2023-05-01 06:34:27.825 org.sleuthkit.autopsy.ingest.DataSourceIngestPipeline$DataSourcePipelineModule process
INFO: USN Parser analysis of MUS-CTF-19-DESKTOP-001.E01 starting
2023-05-01 06:34:27.925 org.sleuthkit.autopsy.casemodule.IngestJobInfoPanel$1 done
INFO: The refreshing of the IngestJobInfoPanel was cancelled
2023-05-01 06:34:27.987 ParseUsnJIngestModule process
INFO: found 1 files
2023-05-01 06:34:27.989 ParseUsnJIngestModule process
INFO: create Directory C:\Users\shanna\AppData\Local\Temp\Autopsy\test_20230428_045812\Temp
2023-05-01 06:34:27.99 ParseUsnJIngestModule process
INFO: Usnj Directory already exists C:\Users\shanna\AppData\Local\Temp\Autopsy\test_20230428_045812\Temp\usnj
2023-05-01 06:34:29.379 ParseUsnJIngestModule process
INFO: Saved File ==> C:\Users\shanna\AppData\Local\Temp\Autopsy\test_20230428_045812\Temp\usnj\usnj.txt
2023-05-01 06:34:29.38 ParseUsnJIngestModule process
INFO: Running program ==> C:\Users\shanna\AppData\Roaming\autopsy\python_modules\Parse_USNJ\parseusn.exe C:\Users\shanna\AppData\Local\Temp\Autopsy\test_20230428_045812\Temp\usnj\usnj.txt C:\Users\shanna\AppData\Local\Temp\Autopsy\test_20230428_045812\Temp\usnj.db3
2023-05-01 06:34:31.001 ParseUsnJIngestModule process
INFO: Output from run is ==> usnj is C:\Users\shanna\AppData\Local\Temp\Autopsy\test_20230428_045812\Temp\usnj\usnj.txt
DB file is C:\Users\shanna\AppData\Local\Temp\Autopsy\test_20230428_045812\Temp\usnj\usnj.db3

2023-05-01 06:34:31.001 ParseUsnJIngestModule process
INFO: Path the system database file created ==> C:\Users\shanna\AppData\Local\Temp\Autopsy\test_20230428_045812\Temp\usnj\usnj.db3
2023-05-01 06:34:31.002 ParseUsnJIngestModule process
INFO: query SQLite Master table
2023-05-01 06:34:31.004 ParseUsnJIngestModule process
INFO: Begin Create New Artifacts
2023-05-01 06:34:31.004 org.sleuthkit.autopsy.ingest.IngestJobExecutor logErrorMessage
SEVERE: USN Parser experienced an error during analysis (data source = MUS-CTF-19-DESKTOP-001.E01, data source object ID = 1, ingest job ID = 7)
    at org.sleuthkit.datamodel.Blackboard.getArtifactType(Blackboard.java:421)
    at org.sleuthkit.datamodel.AbstractContent.newArtifact(AbstractContent.java:353)
    at org.sleuthkit.datamodel.AbstractFile.newArtifact(AbstractFile.java:1570)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
org.sleuthkit.datamodel.TskCoreException: org.sleuthkit.datamodel.TskCoreException: No artifact type found matching id: -1

    org.python.core.Py.JavaError(Py.java:547)
    org.python.core.PyObject._jthrow(PyObject.java:3593)
    org.python.core.PyObject._jcall(PyObject.java:3600)
    org.python.proxies.Parse_Usnj$ParseUsnJIngestModule$1117.process(Unknown Source)
    org.sleuthkit.autopsy.ingest.DataSourceIngestPipeline$DataSourcePipelineModule.process(DataSourceIngestPipeline.java:95)
    org.sleuthkit.autopsy.ingest.DataSourceIngestPipeline$DataSourcePipelineModule.process(DataSourceIngestPipeline.java:74)
    org.sleuthkit.autopsy.ingest.IngestPipeline.performTask(IngestPipeline.java:217)
    org.sleuthkit.autopsy.ingest.IngestJobExecutor.execute(IngestJobExecutor.java:568)
    org.sleuthkit.autopsy.ingest.DataSourceIngestTask.execute(DataSourceIngestTask.java:41)
    org.sleuthkit.autopsy.ingest.IngestManager$ExecuteIngestJobTasksTask.run(IngestManager.java:1121)
    java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    java.util.concurrent.FutureTask.run(FutureTask.java:266)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    java.lang.Thread.run(Thread.java:748)
org.sleuthkit.datamodel.TskCoreException: No artifact type found matching id: -1
    org.python.core.Py.JavaError(Py.java:547)
    org.python.core.PyObject._jthrow(PyObject.java:3593)
    org.python.core.PyObject._jcall(PyObject.java:3600)
    org.python.proxies.Parse_Usnj$ParseUsnJIngestModule$1117.process(Unknown Source)
    org.sleuthkit.autopsy.ingest.DataSourceIngestPipeline$DataSourcePipelineModule.process(DataSourceIngestPipeline.java:95)
    org.sleuthkit.autopsy.ingest.DataSourceIngestPipeline$DataSourcePipelineModule.process(DataSourceIngestPipeline.java:74)
    org.sleuthkit.autopsy.ingest.IngestPipeline.performTask(IngestPipeline.java:217)
    org.sleuthkit.autopsy.ingest.IngestJobExecutor.execute(IngestJobExecutor.java:568)
    org.sleuthkit.autopsy.ingest.DataSourceIngestTask.execute(DataSourceIngestTask.java:41)
    org.sleuthkit.autopsy.ingest.IngestManager$ExecuteIngestJobTasksTask.run(IngestManager.java:1121)
    java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    java.util.concurrent.FutureTask.run(FutureTask.java:266)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    java.lang.Thread.run(Thread.java:748)
2023-05-01 06:34:31.004 org.sleuthkit.autopsy.ingest.IngestJobExecutor logInfoMessage
INFO: Finished all ingest tasks for tier 0 of ingest job (data source = MUS-CTF-19-DESKTOP-001.E01, data source object ID = 1, job ID = 7)
2023-05-01 06:34:31.004 org.sleuthkit.autopsy.ingest.IngestJobExecutor logInfoMessage
INFO: Scheduling ingest tasks for tier 1 of ingest job (data source = MUS-CTF-19-DESKTOP-001.E01, data source object ID = 1, job ID = 7)
2023-05-01 06:34:31.004 org.sleuthkit.autopsy.ingest.IngestJobExecutor logInfoMessage
INFO: Finished all ingest tasks for tier 1 of ingest job (data source = MUS-CTF-19-DESKTOP-001.E01, data source object ID = 1, job ID = 7)
2023-05-01 06:34:31.004 org.sleuthkit.autopsy.ingest.IngestJobExecutor logInfoMessage
INFO: Finished all ingest tasks for ingest job (data source = MUS-CTF-19-DESKTOP-001.E01, data source object ID = 1, job ID = 7)
2023-05-01 06:34:31.008 org.sleuthkit.autopsy.ingest.IngestManager finishIngestJob
INFO: Ingest job 7 completed at 1682922871008