Feature Request: Add multiple evidence items at once, run ingest items once

[Fetchered]

I'm not sure if this is something already being worked on, however I would like the ability to add several evidence items at once, then process multiple ingest filters on all items. Currently, I have to select a single image, select or deselect all filters, then open and add. Then I have to wait several minutes to hours for the first image to load, then process the next and repeat. Unfortunately, if I'm not at the computer when the first one finishes, I could lose several hours due to 'down-time' whereby Autopsy is waiting for me to add the next one and start again.

I'm fine with the length of time it will take to process all ingest filters on all items, but at least it would only take minutes to add all images, then select all ingest items and click 'Start'. Then I could walk away knowing it will complete it's tasks, and check back every so often to look for errors or completion.

Cheers

[bcarrier]

Hello, We can look into adding that for single-user cases. The way that we end up doing it though is using the Auto-ingest feature for multi-user cases. It is an undocumented feature that is about to be documented with Pull Request #4610. It allows you to have nodes that are scanning folders looking for media and start to analyze them once they are found. You can then do multiple images in parallel or start a new analysis as soon as the first completes.

[markmckinnon]

You can also setup multi-user on a spare machine and use the Auto-ingest feature or you can setup multi-user on the same machine you are using and do it that way. In my testing and use I have seen a performance increase in using PostgreSQL (multi-user) on the same machine vs using SQLite (single-user). It usually runs in approximately 1/2 the time because of the journaling method in SQLite.

[Fetchered]

Primarily my ask is so that I can, in one sitting, click Add Data Source, Disk Image, and select all of my images at once. Even if I can't ingest them all at once, having to click on one image and wait 40 minutes before I can add the next is bothersome and time-consuming. The longest part of the wait is the 'adding it to a local database' part. It would be easier if I could select all images, then click start and have it do all of them at once.

Right now I have 6 images from a single case I'd like to add and start processing. But I can only add one (even without running the ingest filters), wait 40 minutes, then add another, wait 40 etc etc. As far as I can tell, Multi-User won't resolve this issue, only require me to have another machine and a server supplying Postgres et al, then I just have to repeat the tasking from both. If I'm the only user on this case, this becomes a bit more inconvenient.

[APriestman]

The auto ingest feature (which is in the experimental package) allows you to create a manifest file next to every data source you want to process. You tell it what case you want the data source to go in - they can either all go into the same case or different cases. An Autopsy node in auto-ingest mode will scan for these manifest files and process each data source automatically. You can do everything on a single system if desired - you just have to switch the auto ingest node back into normal mode when processing is complete.

On Mon, Mar 25, 2019 at 11:14 AM Fetchered notifications@github.com wrote:

Primarily my ask is so that I can, in one sitting, click Add Data Source, Disk Image, and select all of my images at once. Even if I can't ingest them all at once, having to click on one image and wait 40 minutes before I can add the next is bothersome and time-consuming. The longest part of the wait is the 'adding it to a local database' part. It would be easier if I could select all images, then click start and have it do all of them at once.

Right now I have 6 images from a single case I'd like to add and start processing. But I can only add one (even without running the ingest filters), wait 40 minutes, then add another, wait 40 etc etc. As far as I can tell, Multi-User won't resolve this issue, only require me to have another machine and a server supplying Postgres et al, then I just have to repeat the tasking from both. If I'm the only user on this case, this becomes a bit more inconvenient.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/sleuthkit/autopsy/issues/4649#issuecomment-476242170, or mute the thread https://github.com/notifications/unsubscribe-auth/ADFgIPWbWbD0UHpMJkm56rc_pnydZdvbks5vaOfwgaJpZM4cF7S- .

[markmckinnon]

Actually this is a perfect use for auto-ingest. You set up the auto-ingest case and it will process thru all the images adding each one to the case and running the ingest modules that you want to use. I have tested this feature out and it will work in your scenario.

[Fetchered]

Do I have to click 'Add Data Source' first? If so, then this is still the bottleneck and outside of the 'ingest'. The issue is actually adding the images before any processing gets done. In every other forensic tool currently out there, I can browse to add all of the images without processing a single module, filter, or application of any sort against them. Then, I can choose the items to process, and start the auto-ingest, walking away to continue other work while it processes.

This is the feature I'm asking for. Simply, Add All Images prior to doing any processing.

[markmckinnon]

As Ann mentioned above if you have multi-user mode set up then you can create your manifest files using the manifest tool. Start Autopsy in auto-ingest mode and it will start to process each image it finds a manifest for. If you set up ingest to run at the same time then it will run each ingest modules you select on the images. Once you start Auto-ingest walk away and come back later and your images will be processed in the same case (barring any errors). It is perfect for your scenario. I have tested this out and it does work well. It does take a little bit to set up but once you do it is really easy to use.

[APriestman]

I'll attach the main docs since they aren't available online yet. They're targeted more at the situation where you'd have multiple machines and analysts, but you can install all the services and Autopsy on on machine.

Autopsy User Documentation_Automated Ingest.pdf Autopsy User Documentation_Auto Ingest Configuration.pdf

[Fetchered]

Excellent, thank you both. I'll give this a shot now and reach back if I have any issues.

Cheers.

[markmckinnon]

If you want to reach out to me off-list feel free to do so.

[rcordovano]

Primarily my ask is so that I can, in one sitting, click Add Data Source, Disk Image, and select all of my images at once. Even if I can't ingest them all at once, having to click on one image and wait 40 minutes before I can add the next is bothersome and time-consuming.

Although auto ingest may fit the bill here, it should be noted that you do not have to add data sources one at a time. Once you start an "ingest job" you can go ahead and click/select the "Add Data Source" toolbar button/menu item to start another ingest job, and so on. Each job can have its own filter, module settings, etc. Depending on the data sources and the resources available on your machine, things may run slower than they would if you add and analyze the data sources serially, but this is a way to analyze multiple data sources at one time if you want to stick to single-user cases.

On Mon, Mar 25, 2019 at 11:39 AM Mark McKinnon notifications@github.com wrote:

If you want to reach out to me off-list feel free to do so.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/sleuthkit/autopsy/issues/4649#issuecomment-476253260, or mute the thread https://github.com/notifications/unsubscribe-auth/ABolxZdnV9j5PjYsA2B8qp8WsSiCsp5-ks5vaO3MgaJpZM4cF7S- .

[Fetchered]

Hello again, To start, I've configured everything the way it should be with multi-user mode, and everything works, with a caveat: There is no auto-ingest button in 4.10.0 to test that the configuration options you provided would work. I will wait until this feature is rolled-out and test again to see if it will do what I'm hoping it will accomplish.

@rcordovano , unfortunately at the current time, you cannot go ahead and click/select the "Add Data Source" toolbar button to start another ingest job until the 'Add Data Source' is completed. This is what I am hoping to rectify. Right now, I create a new case, select Add Data Source -> select Disk Image or VM File -> Next -> Browse to the path, select the image (I can only select one, can't CTRL or SHIFT select multiple), click Open -> click Next -> Choose no ingest modules (just to save me time for this example) -> Next -> Then I have to wait for this process to finish:

This can take between 10 - 40 minutes, or longer if the data source is large (1TB +). So now I have to wait for this to finish, because I can't click on anything else other than Cancel.

@markmckinnon @bcarrier , you can mark this as closed if you wish as it's possible Auto-Ingest will solve this problem. However, it would be great to have this feature for Single-User without the extra requirements (services, servers, etc).

Thank you for the help!

[APriestman]

If you mean the auto-ingest tab on the options panel is missing, you'll need to enable the experimental package to see it. There are instructions here: http://sleuthkit.org/autopsy/docs/user-docs/4.10.0/experimental_page.html

[Fetchered]

@APriestman I do mean that tab, but I want to wait until it's no longer experimental before I decide to build a use-case for it.

I would like to use Autopsy as a replacement tool for teaching students the basics of Digital Forensics (not trying to teach the tool, but use the tool to get the data). Currently, we're using another tool which students find quite confusing, an somewhat unreliable. However, it is capable of adding all data sources at once, then ingesting the data in a timely fashion.

Autopsy is very easy to use, provides detailed results and is very configurable. But adding the images to the training case (6 images in total), takes nearly a day to complete with all ingest modules required. Since the students will be doing this on PC's we provide them, it increases our overhead by nearly 200% to have to add three additional services to each box, add configurations for storage, create new credentials etc. Doing this over the network is not an option, as our network is shared with several other classes at the same time.

I will still contribute to the testing of this version and see that it does what I hope. And I also appreciate your assistance.

Cheers!

[bcarrier]

FWIW, things in the Experimental module are stable. They run in our customer environments. But:

• Their API is in flux and we don't like to change the API of things once they are in "Core Autopsy". So, things stay in Experimental for a while as their features evolve.
• The current functionality may have some assumptions for a specific customer that it was built for and not quite general purpose yet. So, they hang out in Experimental for a while until they are more general purpose.

But, I agree that we should be able to queue these things up in a single-user case too.

Thanks.