search

Velocidex / velociraptor

Digging Deeper....
https://docs.velociraptor.app/
Other
2.89k stars 480 forks source link

no actions appears to take place when using deaddisk function on raw image using windows binary #2016

Closed bcvilnrotter closed 1 year ago

bcvilnrotter commented 2 years ago

First attempt at getting remapped config file

PS <REDACTED>> .\velociraptor-v0.6.5-2-windows-amd64.exe -v deaddisk --add_windows_disk='<REDACTED>\Windows-10-test-flat.001' test.yaml
[INFO] 2022-08-23T23:22:35-04:00  _    __     __           _                  __
[INFO] 2022-08-23T23:22:35-04:00 | |  / /__  / /___  _____(_)________ _____  / /_____  _____
[INFO] 2022-08-23T23:22:35-04:00 | | / / _ \/ / __ \/ ___/ / ___/ __ `/ __ \/ __/ __ \/ ___/
[INFO] 2022-08-23T23:22:35-04:00 | |/ /  __/ / /_/ / /__/ / /  / /_/ / /_/ / /_/ /_/ / /
[INFO] 2022-08-23T23:22:35-04:00 |___/\___/_/\____/\___/_/_/   \__,_/ .___/\__/\____/_/
[INFO] 2022-08-23T23:22:35-04:00                                   /_/
[INFO] 2022-08-23T23:22:35-04:00 Digging deeper!                  https://www.velocidex.com
[INFO] 2022-08-23T23:22:35-04:00 This is Velociraptor 0.6.5-2 built on 2022-07-27T02:38:20+10:00 (795f9339)
[INFO] 2022-08-23T23:22:35-04:00 Env var VELOCIRAPTOR_API_CONFIG is not set
[INFO] 2022-08-23T23:22:35-04:00 No embedded config - you can pack one with the `config repack` command
[INFO] 2022-08-23T23:22:35-04:00 Env var VELOCIRAPTOR_CONFIG is not set
[INFO] 2022-08-23T23:22:35-04:00 Setting empty config
[INFO] 2022-08-23T23:22:35-04:00 Starting Journal service.
[INFO] 2022-08-23T23:22:35-04:00 Starting the notification service.
[INFO] 2022-08-23T23:22:35-04:00 Installing Dummy inventory_service. Will download tools to temp directory.
[INFO] 2022-08-23T23:22:35-04:00 Loaded 333 built in artifacts in 208.7707ms
[INFO] 2022-08-23T23:22:35-04:00 Enumerating partitions using Windows.Forensics.PartitionTable
[DEBUG] 2022-08-23T23:22:35-04:00 Query Stats: {"RowsScanned":0,"PluginsCalled":1,"FunctionsCalled":0,"ProtocolSearch":9,"ScopeCopy":2}
[INFO] 2022-08-23T23:22:35-04:00 Exiting Launcher Service

Second attempt at creating remapping config yaml

PS <REDACTED>> .\velociraptor-v0.6.5-2-windows-amd64.exe -v deaddisk --add_windows_disk '<REDACTED>\Windows-10-test-flat.001' test.yaml
[INFO] 2022-08-23T23:22:24-04:00  _    __     __           _                  __
[INFO] 2022-08-23T23:22:24-04:00 | |  / /__  / /___  _____(_)________ _____  / /_____  _____
[INFO] 2022-08-23T23:22:24-04:00 | | / / _ \/ / __ \/ ___/ / ___/ __ `/ __ \/ __/ __ \/ ___/
[INFO] 2022-08-23T23:22:24-04:00 | |/ /  __/ / /_/ / /__/ / /  / /_/ / /_/ / /_/ /_/ / /
[INFO] 2022-08-23T23:22:24-04:00 |___/\___/_/\____/\___/_/_/   \__,_/ .___/\__/\____/_/
[INFO] 2022-08-23T23:22:24-04:00                                   /_/
[INFO] 2022-08-23T23:22:24-04:00 Digging deeper!                  https://www.velocidex.com
[INFO] 2022-08-23T23:22:24-04:00 This is Velociraptor 0.6.5-2 built on 2022-07-27T02:38:20+10:00 (795f9339)
[INFO] 2022-08-23T23:22:24-04:00 Env var VELOCIRAPTOR_API_CONFIG is not set
[INFO] 2022-08-23T23:22:24-04:00 No embedded config - you can pack one with the `config repack` command
[INFO] 2022-08-23T23:22:24-04:00 Env var VELOCIRAPTOR_CONFIG is not set
[INFO] 2022-08-23T23:22:24-04:00 Setting empty config
[INFO] 2022-08-23T23:22:24-04:00 Starting Journal service.
[INFO] 2022-08-23T23:22:24-04:00 Starting the notification service.
[INFO] 2022-08-23T23:22:24-04:00 Installing Dummy inventory_service. Will download tools to temp directory.
[INFO] 2022-08-23T23:22:24-04:00 Loaded 333 built in artifacts in 232.0047ms
[INFO] 2022-08-23T23:22:24-04:00 Enumerating partitions using Windows.Forensics.PartitionTable
[DEBUG] 2022-08-23T23:22:24-04:00 Query Stats: {"RowsScanned":0,"PluginsCalled":1,"FunctionsCalled":0,"ProtocolSearch":9,"ScopeCopy":2}
[INFO] 2022-08-23T23:22:24-04:00 Exiting notification service!
[INFO] 2022-08-23T23:22:24-04:00 Exiting Launcher Service

Contents of test.yaml

remappings:
- type: shadow
  from:
    accessor: zip
  "on":
    accessor: zip
- type: shadow
  from:
    accessor: raw_reg
  "on":
    accessor: raw_reg
- type: shadow
  from:
    accessor: data
  "on":
    accessor: data

Testing raw image with mmls.exe and fls.exe on windows

PS <REDACTED>> .\mmls.exe '<REDACTED>\Windows-10-test-flat.001'
DOS Partition Table
Offset Sector: 0
Units are in 512-byte sectors

      Slot      Start        End          Length       Description
000:  Meta      0000000000   0000000000   0000000001   Primary Table (#0)
001:  -------   0000000000   0000002047   0000002048   Unallocated
002:  000:000   0000002048   0000104447   0000102400   NTFS / exFAT (0x07)
003:  000:001   0000104448   0103809676   0103705229   NTFS / exFAT (0x07)
004:  -------   0103809677   0103811071   0000001395   Unallocated
005:  000:002   0103811072   0104853503   0001042432   Unknown Type (0x27)
006:  -------   0104853504   0104857599   0000004096   Unallocated
PS <REDACTED>> .\fls.exe -o 0000104448 '<REDACTED>\Windows-10-test-flat.001'
d/d 100413-144-1:       Documents and Settings
d/d 1362-144-6: ProgramData
d/d 1493-144-5: Users
r/r 4-128-1:    $AttrDef
r/r 8-128-2:    $BadClus
r/r 8-128-1:    $BadClus:$Bad
r/r 6-128-4:    $Bitmap
r/r 6-128-5:    $Bitmap:$SRAT
r/r 7-128-1:    $Boot
d/d 11-144-4:   $Extend
r/r 2-128-1:    $LogFile
r/r 0-128-6:    $MFT
r/r 1-128-1:    $MFTMirr
d/d 58-144-1:   $Recycle.Bin
r/r 9-144-17:   $Secure:$SDH
r/r 9-144-16:   $Secure:$SII
r/r 9-128-18:   $Secure:$SDS
r/r 10-128-1:   $UpCase
r/r 10-128-4:   $UpCase:$Info
r/r 3-128-3:    $Volume
r/r 98419-128-1:        DumpStack.log.tmp
d/d 103716-144-1:       OneDriveTemp
d/d 59-144-1:   PerfLogs
d/d 60-144-6:   Program Files
d/d 1219-144-6: Program Files (x86)
d/d 99113-144-1:        Recovery
r/r 98420-128-1:        swapfile.sys
d/d 57085-144-6:        System Volume Information
d/d 1550-144-5: Windows
V/V 107776:     $OrphanFiles
r/r 98418-128-1:        pagefile.sys
scudette commented 2 years ago

The deaddisk command tries to guess a remapping config by enumerating the partitions and looking for a 'windows' directory (to make it a C: drive). It is pretty simplistic at the moment and is really a best effort kind of thing.

It uses the Windows.Forensics.PartitionTable artifact which currently only supports GPT GUID partition tables - it looks from mmls that this drive has an old DOS style partition table (most modern disks use GPT). I think this is the reason it is not able to see any paritions .

Anyway it is probably easier to just adapt the remapping by hand - you can see here

https://gist.github.com/scudette/97152f6b7427ebbb924308954ab06d39

for example a remapping file from another drive - all you need to do is to adjust the offsets into the disk (note: they are in bytes and mmls gives in blocks so you need to convert).

bcvilnrotter commented 2 years ago

Thanks for this response, this helps me conceptualize how I can navigate some automation projects I'm working on.

Is there any indication that the Windows.Forensics.PartitionTable will be expanded to utilize old DOS style partition tables in the future?

I've created this test VMDK from VirtualBox, and converted the VMDK to raw image using FTK Imager to conduct this test. I've also used velociraptor 0.6.4-2 on Client raw images with very similar results, and I would like to have an efficient way to conduct deaddisk analysis using velociraptor.

scudette commented 2 years ago

Sure you can file a bug to support dos partition style in that artifact. Maybe also attach an example partition table for testing. (Just dd the first few sectors should be fine)

I normally just mount the vmdk directly for example https://docs.velociraptor.app/blog/2022/2022-03-22-deaddisk/#mounting-the-image

It saves having to image it again.

bcvilnrotter commented 2 years ago

Thanks for the suggestion, I'll see if I can file a bug to support dos partition style for Windows.Forensics.PartitionTable.

Lastly, I use that blog post mentioned directly to get deaddisk remapping to work using linux binaries. Is the recommendation to just not use the windows binary of velociraptor to produce remapped yamls?

scudette commented 2 years ago

It should not matter if you use linux or windows binaries - are you saying that the deaddisk command did not work on the windows version? Or are you saying that after editing the remapping file example by hand you used the linux binary to analyze it?

bcvilnrotter commented 2 years ago

It does kind of matter if I use linux or windows binaries to create the remapping yaml, because whichever binary I use dictates the binary I have to use to parse the deaddisk image in, and therefore the environment....unless I'm missing something, which is highly possible.

Currently I'm working on a project to build a standalone tool that can parse through deaddisk images on a windows environment seamlessly without arbitrary dependencies like incorporating a *unix environment such as WSL.

Windows also doesn't allow me to mount an image as a flat file like how it is described in the blog (in linux). If I mount an image in Windows then run --add_windows_directory, previous testing (in 0.6.4-2) shows that it cannot pull crucial running artifacts such as $MFT. So the only way I perceive I can go forward is to potentially build a script that can take the template remapped yaml you provided and alter it per image based on offset values from mmls.exe, and changing out the file location of the image.

(editted) this is only for images that have old DOS style partition tables.

scudette commented 2 years ago

Does vmware-mount not work on windows? I am not sure? I think affuse can also expose vmdk images as flat images.

If not this will be the biggest challenge - building the actual remapping file is not a big deal but currently Velociraptor only supports parsing raw dd style images, so there has to be a way to present a flat file.

We do have an issue open #1840 that aims to support ewf files directly, perhaps we need to do vmdk as well.

bcvilnrotter commented 2 years ago

I haven't used vmware-mount on windows so I will definitely dive into that. Thanks a lot for the support and suggestions, I'll close out this ticket, and start a new one focusing on the Windows partition table artifact since that appears to be the crux of the issue.

One last item. I am a bit new to mounting technologies so this might be a place of ignorance for me, are the flat images that affuse exposes from VMDKs inherently different than taking a flat VMDK created from VirtualBox, and converting it to a raw dd using FTK Imager?

bcvilnrotter commented 2 years ago

As another test, I took another image and tried to get a remapped yaml using the deaddisk function in multiple environments using both Windows and Linux distro. My results were, I was unable to get a remapped yaml on Windows/Linux environments, but WAS able to successfully get a remapped yaml on windows using WSL.

Windows/not WSL

PS <REDACTED>> .\velociraptor-v0.6.4-2-windows-amd64.exe -v deaddisk --add_windows_disk <REDACTED>\Windows_10_test-flat.001 test.yaml
[INFO] 2022-08-24T17:34:13Z  _    __     __           _                  __
[INFO] 2022-08-24T17:34:13Z | |  / /__  / /___  _____(_)________ _____  / /_____  _____
[INFO] 2022-08-24T17:34:13Z | | / / _ \/ / __ \/ ___/ / ___/ __ `/ __ \/ __/ __ \/ ___/
[INFO] 2022-08-24T17:34:13Z | |/ /  __/ / /_/ / /__/ / /  / /_/ / /_/ / /_/ /_/ / /
[INFO] 2022-08-24T17:34:13Z |___/\___/_/\____/\___/_/_/   \__,_/ .___/\__/\____/_/
[INFO] 2022-08-24T17:34:13Z                                   /_/
[INFO] 2022-08-24T17:34:13Z Digging deeper!                  https://www.velocidex.com
[INFO] 2022-08-24T17:34:13Z This is Velociraptor 0.6.4-2 built on 2022-05-08T21:40:03+10:00 (b6c5764a)
[INFO] 2022-08-24T17:34:13Z Env var VELOCIRAPTOR_API_CONFIG is not set
[INFO] 2022-08-24T17:34:13Z No embedded config - you can pack one with the `config repack` command
[INFO] 2022-08-24T17:34:13Z Env var VELOCIRAPTOR_CONFIG is not set
[INFO] 2022-08-24T17:34:13Z Setting empty config
[INFO] 2022-08-24T17:34:13Z Starting Journal service.
[INFO] 2022-08-24T17:34:13Z Starting the notification service.
[INFO] 2022-08-24T17:34:13Z Installing Dummy inventory_service. Will download tools to temp directory.
[INFO] 2022-08-24T17:34:14Z Loaded 330 built in artifacts in 250.9156ms
velociraptor: Enumerating partitions using Windows.Forensics.PartitionTable
velociraptor: Query Stats: {"RowsScanned":0,"PluginsCalled":1,"FunctionsCalled":0,"ProtocolSearch":42,"ScopeCopy":2}
[INFO] 2022-08-24T17:34:14Z Exiting Launcher Service
[INFO] 2022-08-24T17:34:14Z Exiting notification service!
PS (EDIT: <REDACTED>)> cat .\test.yaml
remappings:
- type: shadow
  from:
    accessor: zip
  "on":
    accessor: zip
- type: shadow
  from:
    accessor: raw_reg
  "on":
    accessor: raw_reg
- type: shadow
  from:
    accessor: data
  "on":
    accessor: data

Linux

<REDACTED>$ sudo velociraptor -v deaddisk --add_windows_disk='<REDACTED>/Windows_10_test-flat.001' <REDACTED>/remapping.yaml
[INFO] 2022-08-24T20:26:07Z  _    __     __           _                  __ 
[INFO] 2022-08-24T20:26:07Z | |  / /__  / /___  _____(_)________ _____  / /_____  _____ 
[INFO] 2022-08-24T20:26:07Z | | / / _ \/ / __ \/ ___/ / ___/ __ `/ __ \/ __/ __ \/ ___/ 
[INFO] 2022-08-24T20:26:07Z | |/ /  __/ / /_/ / /__/ / /  / /_/ / /_/ / /_/ /_/ / / 
[INFO] 2022-08-24T20:26:07Z |___/\___/_/\____/\___/_/_/   \__,_/ .___/\__/\____/_/ 
[INFO] 2022-08-24T20:26:07Z                                   /_/ 
[INFO] 2022-08-24T20:26:07Z Digging deeper!                  https://www.velocidex.com 
[INFO] 2022-08-24T20:26:07Z This is Velociraptor 0.6.4-2 built on 2022-05-08T23:09:48+10:00 (b6c5764a) 
[INFO] 2022-08-24T20:26:07Z Env var VELOCIRAPTOR_API_CONFIG is not set 
[INFO] 2022-08-24T20:26:07Z No embedded config - you can pack one with the `config repack` command 
[INFO] 2022-08-24T20:26:07Z Loading config from env VELOCIRAPTOR_CONFIG (<REDACTED>) 
[INFO] 2022-08-24T20:26:07Z Starting Journal service. 
[INFO] 2022-08-24T20:26:07Z Starting the notification service. 
[INFO] 2022-08-24T20:26:07Z NotificationService: Watching for events from Server.Internal.Ping 
[INFO] 2022-08-24T20:26:07Z NotificationService: Watching for events from Server.Internal.Pong 
[INFO] 2022-08-24T20:26:07Z NotificationService: Watching for events from Server.Internal.Notifications 
[INFO] 2022-08-24T20:26:07Z Starting Inventory Service 
[INFO] 2022-08-24T20:26:07Z Loaded 330 built in artifacts in 178.539113ms 
[INFO] 2022-08-24T20:26:07Z RepositoryManager: Watching for events from Server.Internal.ArtifactModification 
[INFO] 2022-08-24T20:26:07Z Labeler: Watching for events from Server.Internal.Label 
[INFO] 2022-08-24T20:26:07Z Starting Label service. 
velociraptor: Enumerating partitions using Windows.Forensics.PartitionTable
[INFO] 2022-08-24T20:26:08Z Compiled all artifacts. 
velociraptor: Query Stats: {"RowsScanned":0,"PluginsCalled":1,"FunctionsCalled":0,"ProtocolSearch":45,"ScopeCopy":2}
[INFO] 2022-08-24T20:26:09Z Exiting Launcher Service 
[INFO] 2022-08-24T20:26:09Z Exiting notification service!
<REDACTED>$ cat remapping.yaml
remappings:
- type: shadow
  from:
    accessor: zip
  "on":
    accessor: zip
- type: shadow
  from:
    accessor: raw_reg
  "on":
    accessor: raw_reg
- type: shadow
  from:
    accessor: data
  "on":
    accessor: data

Windows / With WSL

<REDACTED>$ ./velociraptor-v0.6.4-2-linux-amd64 -v deaddisk --add_windows_disk <REDACTED>/Windows_10_test-flat.001 test.yaml
[INFO] 2022-08-24T17:36:48Z  _    __     __           _                  __
[INFO] 2022-08-24T17:36:48Z | |  / /__  / /___  _____(_)________ _____  / /_____  _____
[INFO] 2022-08-24T17:36:48Z | | / / _ \/ / __ \/ ___/ / ___/ __ `/ __ \/ __/ __ \/ ___/
[INFO] 2022-08-24T17:36:48Z | |/ /  __/ / /_/ / /__/ / /  / /_/ / /_/ / /_/ /_/ / /
[INFO] 2022-08-24T17:36:48Z |___/\___/_/\____/\___/_/_/   \__,_/ .___/\__/\____/_/
[INFO] 2022-08-24T17:36:48Z                                   /_/
[INFO] 2022-08-24T17:36:48Z Digging deeper!                  https://www.velocidex.com
[INFO] 2022-08-24T17:36:48Z This is Velociraptor 0.6.4-2 built on 2022-05-08T23:09:48+10:00 (b6c5764a)
[INFO] 2022-08-24T17:36:48Z Env var VELOCIRAPTOR_API_CONFIG is not set
[INFO] 2022-08-24T17:36:48Z No embedded config - you can pack one with the `config repack` command
[INFO] 2022-08-24T17:36:48Z Env var VELOCIRAPTOR_CONFIG is not set
[INFO] 2022-08-24T17:36:48Z Setting empty config
[INFO] 2022-08-24T17:36:48Z Starting Journal service.
[INFO] 2022-08-24T17:36:48Z Starting the notification service.
[INFO] 2022-08-24T17:36:48Z Installing Dummy inventory_service. Will download tools to temp directory.
[INFO] 2022-08-24T17:36:49Z Loaded 330 built in artifacts in 250.8982ms
velociraptor: Enumerating partitions using Windows.Forensics.PartitionTable
velociraptor: Searching for a Windows directory at the top level
velociraptor: Searching for a Windows directory at the top level
velociraptor: Adding windows partition at offset 53477376
velociraptor: Searching for a Windows directory at the top level
velociraptor: Query Stats: {"RowsScanned":3,"PluginsCalled":1,"FunctionsCalled":0,"ProtocolSearch":45,"ScopeCopy":8}
[INFO] 2022-08-24T17:36:49Z Exiting Launcher Service
[INFO] 2022-08-24T17:36:49Z Exiting notification service!

MMLS of new image

<REDACTED>> .\mmls.exe <REDACTED>\Windows_10_test-flat.001
DOS Partition Table
Offset Sector: 0
Units are in 512-byte sectors

      Slot      Start        End          Length       Description
000:  Meta      0000000000   0000000000   0000000001   Primary Table (#0)
001:  -------   0000000000   0000002047   0000002048   Unallocated
002:  000:000   0000002048   0000104447   0000102400   NTFS / exFAT (0x07)
003:  000:001   0000104448   0103809676   0103705229   NTFS / exFAT (0x07)
004:  -------   0103809677   0103811071   0000001395   Unallocated
005:  000:002   0103811072   0104853503   0001042432   Unknown Type (0x27)
006:  -------   0104853504   0104857599   0000004096   Unallocated
scudette commented 2 years ago

Thank you for testing this thoroughly - I think you may have hit a bug in the way raw file access is done on windows being different from Linux. I just submitted #2018 to address this issue.

Behavior on both linux and windows should be identical but this was not the case here.

scudette commented 2 years ago

You should be able to test this with one of the latest CI binaries (https://github.com/Velocidex/velociraptor#getting-the-latest-version)

bcvilnrotter commented 1 year ago

Thanks for the help! The prebuild has helped me tremendously with the progression of my velociraptor projects. Closing this out just to help clear out this space, and keep things somewhat tidy.