autopsy 4.21 Linux via snap - issues with Solr

miguel-negrao commented 1 year ago

Hi

I run Debian 12, and installed Autopsy 4.21 from github via snap. I've enabled all connections for the snap using the command line suggested in the readme:

miguel@legion:~$ snap connections autopsy 
Interface               Plug                            Slot                            Notes
audio-playback          autopsy:audio-playback          :audio-playback                 -
block-devices           autopsy:block-devices           :block-devices                  manual
browser-support         autopsy:browser-sandbox         :browser-support                manual
content[gnome-42-2204]  autopsy:gnome-42-2204           gnome-42-2204:gnome-42-2204     -
content[gtk-3-themes]   autopsy:gtk-3-themes            gtk-common-themes:gtk-3-themes  -
content[icon-themes]    autopsy:icon-themes             gtk-common-themes:icon-themes   -
content[sound-themes]   autopsy:sound-themes            gtk-common-themes:sound-themes  -
dbus                    -                               autopsy:dbus-daemon             -
desktop                 autopsy:desktop                 :desktop                        -
desktop-launch          autopsy:desktop-launch          :desktop-launch                 manual
desktop-legacy          autopsy:desktop-legacy          :desktop-legacy                 -
dm-crypt                autopsy:dm-crypt                :dm-crypt                       manual
fuse-support            autopsy:fuse-support            :fuse-support                   manual
gsettings               autopsy:gsettings               :gsettings                      -
hardware-observe        autopsy:hardware-observe        :hardware-observe               manual
home                    autopsy:home                    :home                           -
hugepages-control       autopsy:hugepages-control       :hugepages-control              manual
kernel-crypto-api       autopsy:kernel-crypto-api       :kernel-crypto-api              manual
mount-observe           autopsy:mount-observe           :mount-observe                  manual
network                 autopsy:network                 :network                        -
network-bind            autopsy:network-bind            :network-bind                   -
network-observe         autopsy:network-observe         :network-observe                manual
network-setup-observe   autopsy:network-setup-observe   :network-setup-observe          manual
network-status          autopsy:network-status          :network-status                 -
opengl                  autopsy:opengl                  :opengl                         -
optical-drive           autopsy:optical-drive           :optical-drive                  -
removable-media         autopsy:removable-media         :removable-media                manual
system-files            autopsy:system-files-autopsy    :system-files                   manual
system-files            autopsy:system-files-hugepages  :system-files                   manual
system-observe          autopsy:system-observe          :system-observe                 manual
wayland                 autopsy:wayland                 :wayland                        -
x11                     autopsy:x11                     :x11                            -

When creating a new case or opening an old one, in almost 90% of the times (but not 100%) I get an error related to solr:

2023-09-14 10:23:28.506 org.sleuthkit.autopsy.centralrepository.datamodel.RdbmsCentralRepo upgradeSchema
INFO: Central Repository is up to date
2023-09-14 10:23:28.519 org.sleuthkit.autopsy.keywordsearch.Server startLocalSolr
INFO: Starting local Solr SOLR8 server
2023-09-14 10:23:28.52 org.sleuthkit.autopsy.keywordsearch.Server startLocalSolr
INFO: Port [23 232] available, starting Solr
2023-09-14 10:23:28.52 org.sleuthkit.autopsy.keywordsearch.Server startLocalSolr
INFO: Starting Solr 8 server
2023-09-14 10:23:28.52 org.sleuthkit.autopsy.keywordsearch.Server runLocalSolr8ControlCommand
INFO: Setting Solr 8 directory: /snap/autopsy/x1/autopsy/autopsy/solr
2023-09-14 10:23:28.52 org.sleuthkit.autopsy.keywordsearch.Server runLocalSolr8ControlCommand
INFO: Running Solr 8 command: [/snap/autopsy/x1/autopsy/autopsy/solr/bin/autopsy-solr, start, -p, 23232] from /snap/autopsy/x1/autopsy/autopsy/solr
2023-09-14 10:23:28.523 org.sleuthkit.autopsy.keywordsearch.Server runLocalSolr8ControlCommand
INFO: Finished running Solr 8 command
2023-09-14 10:23:33.597 org.sleuthkit.autopsy.keywordsearch.Server isLocalSolrRunning
INFO: Solr server is running
2023-09-14 10:23:33.623 org.sleuthkit.autopsy.casemodule.Case openAppServiceCaseResources
SEVERE: Solr Keyword Search Service failed to open case resources for test_autopsy21_2
java.util.concurrent.ExecutionException: org.sleuthkit.autopsy.appservices.AutopsyService$AutopsyServiceException: Failed to open or create core for /home/miguel/tmp/test_autopsy21_2
    java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
    java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
    org.sleuthkit.autopsy.casemodule.Case.openAppServiceCaseResources(Case.java:2897)
    org.sleuthkit.autopsy.casemodule.Case.create(Case.java:2243)
    org.sleuthkit.autopsy.casemodule.Case.lambda$doOpenCaseAction$6(Case.java:2160)
    java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    java.base/java.lang.Thread.run(Thread.java:833)
org.sleuthkit.autopsy.appservices.AutopsyService$AutopsyServiceException: Failed to open or create core for /home/miguel/tmp/test_autopsy21_2
    java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
    java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
    org.sleuthkit.autopsy.casemodule.Case.openAppServiceCaseResources(Case.java:2897)
    org.sleuthkit.autopsy.casemodule.Case.create(Case.java:2243)
    org.sleuthkit.autopsy.casemodule.Case.lambda$doOpenCaseAction$6(Case.java:2160)
    java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    java.base/java.lang.Thread.run(Thread.java:833)
org.sleuthkit.autopsy.keywordsearch.KeywordSearchModuleException: Unable to connect to Solr server null
    java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
    java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
    org.sleuthkit.autopsy.casemodule.Case.openAppServiceCaseResources(Case.java:2897)
    org.sleuthkit.autopsy.casemodule.Case.create(Case.java:2243)
    org.sleuthkit.autopsy.casemodule.Case.lambda$doOpenCaseAction$6(Case.java:2160)
    java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    java.base/java.lang.Thread.run(Thread.java:833)
java.lang.UnsupportedOperationException
    java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
    java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
    org.sleuthkit.autopsy.casemodule.Case.openAppServiceCaseResources(Case.java:2897)
    org.sleuthkit.autopsy.casemodule.Case.create(Case.java:2243)
    org.sleuthkit.autopsy.casemodule.Case.lambda$doOpenCaseAction$6(Case.java:2160)
    java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    java.base/java.lang.Thread.run(Thread.java:833)
2023-09-14 10:23:33.629 org.sleuthkit.autopsy.imagegallery.PerCaseProperties getConfigSetting
INFO: File did not exist. Created file [Image Gallery.properties]
2023-09-14 10:23:33.631 org.sleuthkit.autopsy.imagegallery.datamodel.DrawableDB setPragmas
INFO: sqlite-jdbc version 3.42.0.0 loaded in native mode
2023-09-14 10:23:33.643 org.sleuthkit.autopsy.casemodule.Case openAsCurrentCase
INFO: Opened test_autopsy21_2 (test_autopsy21_2_20230914_102326) in /home/miguel/tmp/test_autopsy21_2 as the current case
2023-09-14 10:23:33.663 org.sleuthkit.autopsy.ingest.IngestMonitor$MonitorTimerAction logMonitoredRootDirectory
INFO: Monitoring disk space of /
2023-09-14 10:23:33.684 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Cyber Triage Malware Scanner, version = 1.0.0
2023-09-14 10:23:33.684 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Central Repository, version = 4.21.0
2023-09-14 10:23:33.684 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Data Source Integrity, version = 4.21.0
2023-09-14 10:23:33.684 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = DJI Drone Analyzer, version = 4.21.0
2023-09-14 10:23:33.685 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Embedded File Extractor, version = 4.21.0
2023-09-14 10:23:33.685 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Encryption Detection, version = 4.21.0
2023-09-14 10:23:33.685 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Extension Mismatch Detector, version = 4.21.0
2023-09-14 10:23:33.685 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = File Type Identification, version = 4.21.0
2023-09-14 10:23:33.685 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Hash Lookup, version = 4.21.0
2023-09-14 10:23:33.685 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Interesting Files Identifier, version = 4.21.0
2023-09-14 10:23:33.685 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Android Analyzer (aLEAPP), version = 4.21.0
2023-09-14 10:23:33.685 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = iOS Analyzer (iLEAPP), version = 4.21.0
2023-09-14 10:23:33.685 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = PhotoRec Carver, version = 7.0
2023-09-14 10:23:33.685 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Picture Analyzer, version = 4.21.0
2023-09-14 10:23:33.685 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Plaso, version = 4.21.0
2023-09-14 10:23:33.685 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Virtual Machine Extractor, version = 4.21.0
2023-09-14 10:23:33.685 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = YARA Analyzer, version = 4.21.0
2023-09-14 10:23:33.685 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Keyword Search, version = 4.21.0
2023-09-14 10:23:33.685 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Email Parser, version = 4.21.0
2023-09-14 10:23:33.685 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader addFactory
INFO: Found ingest module factory: name = Recent Activity, version = 4.21.0
2023-09-14 10:23:33.712 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader getIngestModuleFactories
INFO: Found ingest module factory: name = GPX Parser, version = 1.2
2023-09-14 10:23:33.712 org.sleuthkit.autopsy.ingest.IngestModuleFactoryLoader getIngestModuleFactories
INFO: Found ingest module factory: name = Android Analyzer, version = 4.21.0
2023-09-14 10:23:33.728 org.sleuthkit.autopsy.keywordsearch.KeywordSearchGlobalSearchSettingsPanel customizeComponents
WARNING: Could not get number of indexed files/chunks

The only time when I don't get the error is when creating a new case in a specific directory in a specif volume, which is quite misterious.

miguel-negrao commented 1 year ago

@gdicristofaro I think you might be the right person to look at this one. Thanks.

gdicristofaro commented 1 year ago

Hi @miguel-negrao , could you please also include the logs in ~/snap/autopsy/common/.autopsy/dev/var/log/ for the files: autopsy.log.0, messages.log, solr.log.stdout, and solr.log.stderr?

miguel-negrao commented 1 year ago

I'm getting the following behaviour:

If I delete all files in ~/snap/autopsy/common/.autopsy/dev/var/log/ I no longer get an error now when creating a new case. No ideia if it is related or not, could just be coincidence.

If I try to open an old case, I still get the same error. See the files you requested in this gist.

miguel-negrao commented 1 year ago

What seems to happen is that If I open an old case Solr fails, and from then on if I create a new case it also fails to start solr. But If I close autopsy and open again and create a new case then solr starts ok. It doesn't seem to be related to deleting the log files. It could just be random also...

gdicristofaro commented 1 year ago

Hello again @miguel-negrao , I'm not seeing anything readily apparent. Did you happen to have a solr.log.stdout file by any chance? I'll try to circle back to this soon to see if I can debug, but my initial thoughts are these:

It is possibly a permissions issue. For instance, if autopsy was run with different permissions or the case files (particularly the solr core in the case or possibly some of the solr data files in the snap common directory) were created with elevated permissions, solr may not be able to create/edit the files it needs to do in order to do its job. That seems less likely, but it may be a possibility.
It may be an issue where a solr instance created from a previous run, especially a failed run, is somehow interfering with your current autopsy/solr run. After autopsy closes, autopsy should close all solr instances. You should be able to check with something like ps aux | grep solr.
Looking at the solr.log.stderr, I see a usage error, which makes me think that the autopsy solr startup or stop script are somehow providing illegal arguments. If there is a solr.log.stdout, that would be helpful in this situation.

miguel-negrao commented 1 year ago

Every file in "~/snap/autopsy/common/.autopsy/" has the correct permissions "miguel:miguel" which is my username. All files of the autopsy case also have the same owner (miguel:miguel).

There are no solr instances left running, ps aux | grep solr gives nothing.

The content of solr.log.stdout is:

Sending stop command to Solr running on port 23232 ... waiting up to 180 seconds to allow Jetty process 382234 to stop gracefully.
 [|]   [/]   [-]   [\]

gdicristofaro commented 1 year ago

Hi @miguel-negrao , would you be willing to share the case at /home/miguel/tmp/test_autopsy21_2? When did you make that case by the way if you happen to remember? Short of that or in addition, could you share:

/home/miguel/tmp/test_autopsy21_2/SolrCore.properties
~/snap/autopsy/common/.autopsy/dev/solr/logs
~/snap/autopsy/common/.autopsy/dev/solr4/logs if you have any

miguel-negrao commented 1 year ago

Hi @gdicristofaro, I've sent you the case in a zip file to your gmail.

gdicristofaro commented 1 year ago

Hi @miguel-negrao , I believe the issue is that the older cases likely use Solr 4. You should be able to verify this by looking in SolrCore.properties and you should see something like: <SolrVersion>4</SolrVersion>. Autopsy moved on to Solr 8 a few years ago. I think what is happening is that when Autopsy opens the older cases, it has issues running Solr 4 when we are now running with Solr 8. Then, when you go to open a new case, Solr had issues starting up previously and the new case also fails.

Here are some workarounds:

If you opened an older case in Autopsy and Solr encounters an error, close Autopsy and reopen so that Solr can reset itself to run Solr 8.
To open older cases in Autopsy, delete or rename SolrCore.properties. This will cause Autopsy to generate a new SolrCore.properties file using Solr 8. If you need to do any text indexing (i.e. ad hoc searches), you will first need to rerun Keyword Search on the images in the old case to re-index the case in Solr 8. You can do this by right clicking on the image in the tree, going to 'Run Ingest Modules', and selecting 'Keyword Search' with the 'Add text to Solr index' selected.

miguel-negrao commented 1 year ago

Hi, indeed the case I was using for testing was using solr4. I've tested again with an old case but with solr 8 and everything works fine. I suggest that perhaps you could check SolrCore.properties at startup and display a warning saying that this case uses solr4 and ask weather to update to solr4 invalidating the keyword search (deleting the SolrCore.properties file), or close the case.

sleuthkit / autopsy

autopsy 4.21 Linux via snap - issues with Solr #7864