search

sleuthkit / autopsy

Autopsy® is a digital forensics platform and graphical interface to The Sleuth Kit® and other digital forensics tools. It can be used by law enforcement, military, and corporate examiners to investigate what happened on a computer. You can even use it to recover photos from your camera's memory card.
http://www.sleuthkit.org/autopsy/
2.36k stars 590 forks source link

Linux - Timeline stalls in 4.5.1 #3247

Closed marshalla99 closed 6 years ago

marshalla99 commented 6 years ago

Trying to generate a timeline using the 4.5.1 release branch on Linux results in the program stalling after MAC time processing - at "Committing events database" - and becoming unresponsive. The progress bar does not show any progress.

Database has been created and can be inspected using sqlite3 from the command line.

No obvious errors displayed.

danpos commented 6 years ago

@marshalla99 hi, I have got similar behavior to process a 1 GB dd image from a pendrive, that has several jpeg files from scanned documents, which had called convert application in several threads (to work with Apache's Tika library in keyword search module). I had tried to limit vm's swapping, as well as java heap in bash, but I think that this has been superseded to run Autopsy (ant run).

marshalla99 commented 6 years ago

For info. (in case someone who knows the code far better than I can ever hope to do), this is what the shell window shows just before the program locks up.

     [exec] Dec 06, 2017 12:08:24 PM org.sleuthkit.autopsy.timeline.db.EventDB configureDB
     [exec] INFO: sqlite-jdbc version 3.21.0 loaded in native mode
     [exec] Dec 06, 2017 12:08:25 PM org.sleuthkit.autopsy.timeline.db.EventsRepository rebuildRepository
     [exec] INFO: (re)starting FULL db population task
     [exec] Dec 06, 2017 12:08:25 PM org.sleuthkit.autopsy.timeline.db.EventsRepository$DBPopulationWorker call
     [exec] INFO: Beginning population of timeline db.
     [exec] Dec 06, 2017 12:08:25 PM org.sleuthkit.autopsy.timeline.db.EventDB configureDB
     [exec] INFO: sqlite-jdbc version 3.21.0 loaded in native mode
     [exec] Dec 06, 2017 12:10:38 PM org.sleuthkit.autopsy.timeline.db.EventsRepository$DBPopulationWorker call
     [exec] INFO: updating content tags
     [exec] Dec 06, 2017 12:10:38 PM org.sleuthkit.autopsy.timeline.db.EventsRepository$DBPopulationWorker call
     [exec] INFO: updating artifact tags
     [exec] Dec 06, 2017 12:10:39 PM org.sleuthkit.autopsy.timeline.db.EventsRepository$DBPopulationWorker call
     [exec] INFO: committing db
     [exec] Dec 06, 2017 12:10:40 PM org.sleuthkit.autopsy.corecomponents.DataContentViewerHex <init>
     [exec] INFO: Created HexView instance: org.sleuthkit.autopsy.corecomponents.DataContentViewerHex[,0,0,0x0,invalid,layout=javax.swing.GroupLayout,alignmentX=0.0,alignmentY=0.0,border=javax.swing.plaf.synth.SynthBorder@1803201a,flags=9,maximumSize=,minimumSize=,preferredSize=java.awt.Dimension[width=610,height=58]]
     [exec] Dec 06, 2017 12:10:40 PM org.sleuthkit.autopsy.corecomponents.DataContentViewerString <init>
     [exec] INFO: Created StringView instance: org.sleuthkit.autopsy.corecomponents.DataContentViewerString[,0,0,0x0,invalid,layout=javax.swing.GroupLayout,alignmentX=0.0,alignmentY=0.0,border=javax.swing.plaf.synth.SynthBorder@7976a03a,flags=9,maximumSize=,minimumSize=java.awt.Dimension[width=5,height=5],preferredSize=]
     [exec] Dec 06, 2017 12:10:40 PM org.sleuthkit.autopsy.corecomponents.MediaViewVideoPanel createVideoPanel
     [exec] INFO: 64 bit JVM detected. Creating JavaFX Video Player.
     [exec] Dec 06, 2017 12:10:40 PM org.sleuthkit.autopsy.corecomponents.DataContentViewerMedia <init>
     [exec] INFO: Created MediaView instance: org.sleuthkit.autopsy.corecomponents.DataContentViewerMedia[,0,0,0x0,invalid,layout=java.awt.CardLayout,alignmentX=0.0,alignmentY=0.0,border=javax.swing.plaf.synth.SynthBorder@52bdfee0,flags=9,maximumSize=,minimumSize=,preferredSize=]
millmanorama commented 6 years ago

Sorry for the delay in responding to this issue. The log snippet looks normal to me...

Can you confirm if you get the pop up that tracks the progress of adding events to the database? Something like the attached [image: image.png]

This popup (and the rest of timeline) is implemented in JavaFX and I am wondering if it is a problem of JavaFX on linux?

On Wed, Dec 6, 2017 at 6:18 PM marshalla99 notifications@github.com wrote:

For info. (in case someone who knows the code far better than I can ever hope to do), this is what the shell window shows just before the program locks up.

 [exec] Dec 06, 2017 12:08:24 PM org.sleuthkit.autopsy.timeline.db.EventDB configureDB
 [exec] INFO: sqlite-jdbc version 3.21.0 loaded in native mode
 [exec] Dec 06, 2017 12:08:25 PM org.sleuthkit.autopsy.timeline.db.EventsRepository rebuildRepository
 [exec] INFO: (re)starting FULL db population task
 [exec] Dec 06, 2017 12:08:25 PM org.sleuthkit.autopsy.timeline.db.EventsRepository$DBPopulationWorker call
 [exec] INFO: Beginning population of timeline db.
 [exec] Dec 06, 2017 12:08:25 PM org.sleuthkit.autopsy.timeline.db.EventDB configureDB
 [exec] INFO: sqlite-jdbc version 3.21.0 loaded in native mode
 [exec] Dec 06, 2017 12:10:38 PM org.sleuthkit.autopsy.timeline.db.EventsRepository$DBPopulationWorker call
 [exec] INFO: updating content tags
 [exec] Dec 06, 2017 12:10:38 PM org.sleuthkit.autopsy.timeline.db.EventsRepository$DBPopulationWorker call
 [exec] INFO: updating artifact tags
 [exec] Dec 06, 2017 12:10:39 PM org.sleuthkit.autopsy.timeline.db.EventsRepository$DBPopulationWorker call
 [exec] INFO: committing db
 [exec] Dec 06, 2017 12:10:40 PM org.sleuthkit.autopsy.corecomponents.DataContentViewerHex <init>
 [exec] INFO: Created HexView instance: org.sleuthkit.autopsy.corecomponents.DataContentViewerHex[,0,0,0x0,invalid,layout=javax.swing.GroupLayout,alignmentX=0.0,alignmentY=0.0,border=javax.swing.plaf.synth.SynthBorder@1803201a,flags=9,maximumSize=,minimumSize=,preferredSize=java.awt.Dimension[width=610,height=58]]
 [exec] Dec 06, 2017 12:10:40 PM org.sleuthkit.autopsy.corecomponents.DataContentViewerString <init>
 [exec] INFO: Created StringView instance: org.sleuthkit.autopsy.corecomponents.DataContentViewerString[,0,0,0x0,invalid,layout=javax.swing.GroupLayout,alignmentX=0.0,alignmentY=0.0,border=javax.swing.plaf.synth.SynthBorder@7976a03a,flags=9,maximumSize=,minimumSize=java.awt.Dimension[width=5,height=5],preferredSize=]
 [exec] Dec 06, 2017 12:10:40 PM org.sleuthkit.autopsy.corecomponents.MediaViewVideoPanel createVideoPanel
 [exec] INFO: 64 bit JVM detected. Creating JavaFX Video Player.
 [exec] Dec 06, 2017 12:10:40 PM org.sleuthkit.autopsy.corecomponents.DataContentViewerMedia <init>
 [exec] INFO: Created MediaView instance: org.sleuthkit.autopsy.corecomponents.DataContentViewerMedia[,0,0,0x0,invalid,layout=java.awt.CardLayout,alignmentX=0.0,alignmentY=0.0,border=javax.swing.plaf.synth.SynthBorder@52bdfee0,flags=9,maximumSize=,minimumSize=,preferredSize=]

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/sleuthkit/autopsy/issues/3247#issuecomment-349710626, or mute the thread https://github.com/notifications/unsubscribe-auth/ABHTY7vFqmPi3k24cDm5xqK8nUAGD2N9ks5s9sxegaJpZM4QwlDS .

marshalla99 commented 6 years ago

Thanks for the reply. I appreciate that Linux is not a core platform for you, so any help is welcome.

Yes - the pop-up appears and then sometimes disappears, or sometimes stays on screen and frozen. The progress bar at bottom of the main window also freezes.

There seems to have been a long history of JavaFX problems on some linux distros. Thanks for confirming that this is the area I should probably concentrate on.

I'm trying some work-arounds.

millmanorama commented 6 years ago

I will keep thinking. It sounds more like a deadlock related to populating the db. Have you tried to reproduce it on windows? Let me know if you make any progress or have additional information that might help.

On Wed, Dec 6, 2017 at 6:55 PM marshalla99 notifications@github.com wrote:

Yes - the pop-up appears and then sometimes disappears, or sometimes stays on screen and frozen. The progress bar at bottom of the main window also freezes.

There seems to have been a long history of JavaFX problems on some linux distros.

I'm trying some work-arounds.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/sleuthkit/autopsy/issues/3247#issuecomment-349722142, or mute the thread https://github.com/notifications/unsubscribe-auth/ABHTY7tQLGc2Y9hPvc17Deau6kQ7bc2Fks5s9tUAgaJpZM4QwlDS .

marshalla99 commented 6 years ago

As far as I can see, the database seems to be populated OK (inspecting events.db using sqlite command line produces valid looking data). I dropped some extra logging into the timeline sources and, as far as I can see, it completes successfully. The freeze occurs just after the MediaView instance is created.

marshalla99 commented 6 years ago

Hmm - if I hit "Cancel" part-way through the MAC time processing, the system doesn't freeze - but there's no timeline visible either.

marshalla99 commented 6 years ago

This may help with problem identification. It's some output generated during a recent test run which created a new case, ran all ingest plugins available and then closed the case without attempting any TimeLine activity.

[exec] WARNING [org.netbeans.JarClassLoader]: Opening /home/osboxes/AutopsyInst/autopsy/build/cluster/modules/ext/sqlite-jdbc-3.7.8-SNAPSHOT.jar took 570 ms [exec] INFO [org.netbeans.core.windows.persistence]: [PersistenceManager.getTopComponentForID] Problem when deserializing TopComponent for tcID:'TimeLineTopComponent'. Reason: null [exec] Content: [exec] <?xml version="1.0" encoding="UTF-8"?> [exec] <!DOCTYPE settings PUBLIC "-//NetBeans//DTD Session settings 1.0//EN" "http://www.netbeans.org/dtds/sessionsettings-1_0.dtd"> [exec] <settings version="1.0"> [exec] <instance class="org.sleuthkit.autopsy.timeline.TimeLineTopComponent"/> [exec] </settings> [exec] Class: class org.sleuthkit.autopsy.timeline.TimeLineTopComponent [exec] Source: MultiFileObject@56ccea5b[Windows2Local/Components/TimeLineTopComponent.settings] [exec] Content: [exec] <?xml version="1.0" encoding="UTF-8"?> [exec] <!DOCTYPE settings PUBLIC "-//NetBeans//DTD Session settings 1.0//EN" "http://www.netbeans.org/dtds/sessionsettings-1_0.dtd"> [exec] <settings version="1.0"> [exec] <instance class="org.sleuthkit.autopsy.timeline.TimeLineTopComponent"/> [exec] </settings> [exec] Class: class org.sleuthkit.autopsy.timeline.TimeLineTopComponent [exec] Source: MultiFileObject@56ccea5b[Windows2Local/Components/TimeLineTopComponent.settings] [exec] Caused: java.lang.NoSuchMethodException: org.sleuthkit.autopsy.timeline.TimeLineTopComponent.<init>() [exec] at java.lang.Class.getConstructor0(Class.java:3082) [exec] at java.lang.Class.getDeclaredConstructor(Class.java:2178) [exec] at org.netbeans.modules.settings.convertors.XMLSettingsSupport.newInstance(XMLSettingsSupport.java:98) [exec] at org.netbeans.modules.settings.convertors.XMLSettingsSupport$SettingsRecognizer.instanceCreate(XMLSettingsSupport.java:628) [exec] Caused: java.io.IOException [exec] at org.netbeans.modules.settings.convertors.XMLSettingsSupport$SettingsRecognizer.instanceCreate(XMLSettingsSupport.java:630) [exec] at org.netbeans.modules.settings.convertors.SerialDataConvertor$SettingsInstance.instanceCreate(SerialDataConvertor.java:426) [exec] [catch] at org.netbeans.core.windows.persistence.PersistenceManager.getTopComponentPersistentForID(PersistenceManager.java:571) [exec] at org.netbeans.core.windows.persistence.PersistenceManager.getTopComponentForID(PersistenceManager.java:681) [exec] at org.netbeans.core.windows.PersistenceHandler.getTopComponentForID(PersistenceHandler.java:478) [exec] at org.netbeans.core.windows.WindowManagerImpl.getTopComponentForID(WindowManagerImpl.java:962) [exec] at org.netbeans.core.windows.model.TopComponentSubModel.getTopComponent(TopComponentSubModel.java:378) [exec] at org.netbeans.core.windows.model.TopComponentSubModel.getTopComponents(TopComponentSubModel.java:90) [exec] at org.netbeans.core.windows.model.DefaultModeModel.getTopComponents(DefaultModeModel.java:356) [exec] at org.netbeans.core.windows.model.DefaultModel.getModeTopComponents(DefaultModel.java:944) [exec] at org.netbeans.core.windows.Central.getModeTopComponents(Central.java:1577) [exec] at org.netbeans.core.windows.ModeImpl.getTopComponents(ModeImpl.java:220) [exec] at org.sleuthkit.autopsy.corecomponentinterfaces.CoreComponentControl.closeCoreWindows(CoreComponentControl.java:98) [exec] at org.sleuthkit.autopsy.casemodule.Case.lambda$updateGUIForCaseClosed$4(Case.java:1103) [exec] at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311) [exec] at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:756) [exec] at java.awt.EventQueue.access$500(EventQueue.java:97) [exec] at java.awt.EventQueue$3.run(EventQueue.java:709) [exec] at java.awt.EventQueue$3.run(EventQueue.java:703) [exec] at java.security.AccessController.doPrivileged(Native Method) [exec] at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:80) [exec] at java.awt.EventQueue.dispatchEvent(EventQueue.java:726) [exec] at org.netbeans.core.TimableEventQueue.dispatchEvent(TimableEventQueue.java:159) [exec] at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201) [exec] at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116) [exec] at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105) [exec] at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101) [exec] at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93) [exec] at java.awt.EventDispatchThread.run(EventDispatchThread.java:82) [exec] Content: [exec] <?xml version="1.0" encoding="UTF-8"?> [exec] <!DOCTYPE settings PUBLIC "-//NetBeans//DTD Session settings 1.0//EN" "http://www.netbeans.org/dtds/sessionsettings-1_0.dtd"> [exec] <settings version="1.0"> [exec] <instance class="org.sleuthkit.autopsy.timeline.TimeLineTopComponent"/> [exec] </settings> [exec] ALL [null]: Class: class org.sleuthkit.autopsy.timeline.TimeLineTopComponent [exec] ALL [null]: Source: MultiFileObject@56ccea5b[Windows2Local/Components/TimeLineTopComponent.settings] [exec] WARNING [org.netbeans.core.TimableEventQueue]: too much time in AWT thread org.netbeans.modules.sampler.InternalSampler@311dc74e [exec] WARNING [org.netbeans.core.TimableEventQueue]: no snapshot taken

marshalla99 commented 6 years ago

Just dropping a thought in here - does the JavaFX animation require gstreamer?

If so - I think that could be where the problem really lies. Linux distros don't seem to have a java bindings package available for gstreamer, although it should be possible to do something with gst1-java-core if I can figure out where it needs to be.

rcordovano commented 6 years ago

I asked one of our JavaFX gurus in the Digital Forensics group here at Basis Technology about this. His reply: "JavaFX uses gstreamer under the hood for media playback, however, I don't think gstreamer is involved in general JavaFX GUI animation."

marshalla99 commented 6 years ago

OK - well, we're probably going to hit video playback as an issue at some point in the future then ;)

I'm still no further forward with this issue, despite adding more packages to the pre-requisites list in V1.4 of the Ubuntu script. It's a biggie for me as I'd really like to get this working for a class I'm teaching at the end of the month.

marshalla99 commented 6 years ago

Closing this.

It's a build environment problem. Something in the Ubuntu live image I was using must have been causing the problem. I've just done a test build on CAINE 9.0 and Timeline is working there.

Great news for me as it means I can add Autopsy 4.x to the distro and use it with a class in a few weeks.

millmanorama commented 6 years ago

Thanks for your patience with this. I'm sorry for not being more helpful, but I am glad you have a resolution to your problem. (Although I would like to understand the root cause!)

On Thu, Jan 4, 2018 at 1:58 PM marshalla99 notifications@github.com wrote:

Closed #3247 https://github.com/sleuthkit/autopsy/issues/3247.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/sleuthkit/autopsy/issues/3247#event-1409490804, or mute the thread https://github.com/notifications/unsubscribe-auth/ABHTY32kADDWYp3qZVF3Ao2HAQGfbV-Aks5tHMrdgaJpZM4QwlDS .

marshalla99 commented 6 years ago

Thanks. It's useful just to know that there's someone out there reading what I type ;)

The good news is that it's no longer a showstopper, in my book, and since we now know it works in at least one distro, it should be possible in others.

I've had to modify my build script to work in the CAINE environment, adding a few more installs, so I'm going to backport it to the Ubuntu system and see if the Timeline works there too with the new process.

marshalla99 commented 6 years ago

Got it! I think it's related to TSK and the use of libboost-dev. The Ubuntu distro didn't have it installed, but CAINE almost certainly does. Having installed it into Ubuntu and recompiled everything, Timeline now works.