search

log2timeline / plaso

Super timeline all the things
https://plaso.readthedocs.io
Apache License 2.0
1.73k stars 352 forks source link

issue with source scanner #229

Closed joachimmetz closed 9 years ago

joachimmetz commented 9 years ago 
[WARNING] Unable to scan for a supported filesystem with error: Unable to scan source, with error:
Unable to process source path specification with error: 'pyewf_handle_read_buffer: unable to read data.
libewf_chunk_data_initialize: invalid chunk data.
libewf_read_io_handle_read_chunk_data: unable to create chunk data.
libewf_handle_read_buffer: unable to read chunk data: 6966691.'
FS_Info_Con: (tsk3.c:207) Unable to open the image as a filesystem: Cannot determine file system type
Traceback (most recent call last):
File "/usr/bin/log2timeline.py", line 4, in <module>
 __import__('pkg_resources').run_script('plaso==1.2.1.post20150509', 'log2timeline.py')
  File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 729, in run_script
  File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 1642, in run_script
  File "/usr/lib/python2.7/site-packages/plaso-1.2.1.post20150509-py2.7.egg/EGG-INFO/scripts/log2timeline.py", line 576, in <module>
    if not Main():
  File "/usr/lib/python2.7/site-packages/plaso-1.2.1.post20150509-py2.7.egg/EGG-INFO/scripts/log2timeline.py", line 559, in Main
    tool.ProcessSource()
  File "/usr/lib/python2.7/site-packages/plaso-1.2.1.post20150509-py2.7.egg/EGG-INFO/scripts/log2timeline.py", line 504, in ProcessSource
    vss_stores=self._vss_stores)
  File "/usr/lib/python2.7/site-packages/plaso-1.2.1.post20150509-py2.7.egg/plaso/frontend/storage_media_frontend.py", line 450, in ScanSource
    self._scan_context, scan_path_spec=scan_path_spec)
  File "build/bdist.linux-x86_64/egg/dfvfs/helpers/source_scanner.py", line 406, in Scan
  File "build/bdist.linux-x86_64/egg/dfvfs/helpers/source_scanner.py", line 265, in _ScanNode
  File "build/bdist.linux-x86_64/egg/dfvfs/helpers/source_scanner.py", line 513, in ScanForVolumeSystem
  File "build/bdist.linux-x86_64/egg/dfvfs/analyzer/analyzer.py", line 342, in GetVolumeSystemTypeIndicators
  File "build/bdist.linux-x86_64/egg/dfvfs/analyzer/analyzer.py", line 184, in _GetTypeIndicators
IOError: pysigscan_scanner_scan_file_object: unable to scan file.
pysigscan_file_object_read_buffer: unable to read from file object with error:
pyewf_handle_read_buffer: unable to read data.
libewf_chunk_data_initialize: invalid chunk data.
libewf_read_io_handle_read_chunk_data: unable to create chunk data.
libewf_handle_read_buffer: unable to read chunk data: 0.
pysigscan_file_object_io_handle_read: unable to read from file object.
libbfio_handle_read_buffer: unable to read from handle.
libsigscan_scanner_scan_file_io_handle: unable to read buffer.

The image is loaded from a VMWare shared folder and mounted to /mnt/hgfs/

Also see: https://groups.google.com/forum/#!topic/log2timeline-discuss/TtrBoKfopOA

joachimmetz commented 9 years ago 
PYTHONPATH=. python examples/source_analyzer.py --no-auto-recurse /mnt/hgfs/Kali_Image/11052015_SYSTEMX.E01
Scan level: 3
Source type     : storage media image

OS: location: /mnt/hgfs/Kali_Image/11052015_SYSTEMX.E01
  EWF: 
    TSK_PARTITION: location: /
      TSK_PARTITION: 0, start offset: 0 (0x00000000)
      TSK_PARTITION: 1, start offset: 0 (0x00000000)
      TSK_PARTITION: 2, start offset: 1048576 (0x00100000), location: /p1
        TSK: location: /
      TSK_PARTITION: 3, start offset: 105906176 (0x06500000), location: /p2
        VSHADOW: location: /
          TSK: location: /
      TSK_PARTITION: 4, start offset: 225051672576 (0x3466200000)

^CTraceback (most recent call last):
  File "examples/source_analyzer.py", line 250, in <module>
    if not Main():
  File "examples/source_analyzer.py", line 235, in Main
    source_analyzer.Analyze(options.source, output_writer)
  File "examples/source_analyzer.py", line 62, in Analyze
    scan_path_spec=scan_path_spec)
  File "build/bdist.linux-x86_64/egg/dfvfs/helpers/source_scanner.py", line 486, in Scan
  File "build/bdist.linux-x86_64/egg/dfvfs/helpers/source_scanner.py", line 349, in _ScanNode
  File "build/bdist.linux-x86_64/egg/dfvfs/helpers/source_scanner.py", line 404, in _ScanNode
  File "build/bdist.linux-x86_64/egg/dfvfs/helpers/source_scanner.py", line 509, in ScanForFileSystem
dfvfs.lib.errors.BackEndError: Unable to process source path specification with error: 'pyewf_handle_read_buffer: unable to read data. libewf_chunk_data_initialize: invalid chunk data. libewf_read_io_handle_read_chunk_data: unable to create chunk data. libewf_handle_read_buffer: unable to read chunk data: 6914818.'
FS_Info_Con: (tsk3.c:207) Unable to open the image as a filesystem: Cannot determine file system type
joachimmetz commented 9 years ago

Seems to be stuck in scanning the VSS: https://github.com/log2timeline/dfvfs/blob/caaf8c9b9de97ac692253017df9e2dbd7f563727/dfvfs/helpers/source_scanner.py#L349 https://github.com/log2timeline/dfvfs/blob/caaf8c9b9de97ac692253017df9e2dbd7f563727/dfvfs/helpers/source_scanner.py#L404 https://github.com/log2timeline/dfvfs/blob/caaf8c9b9de97ac692253017df9e2dbd7f563727/dfvfs/helpers/source_scanner.py#L509

https://github.com/log2timeline/dfvfs/blob/caaf8c9b9de97ac692253017df9e2dbd7f563727/dfvfs/analyzer/analyzer.py#L259

joachimmetz commented 9 years ago

source_analyzer does currently not sub scan VSS volumes when --no-auto-recurse is provided. The volume scan loop terminate because no new volume is found and the file system scan stops because it cannot scan the root of a volume system for a file system.

joachimmetz commented 9 years ago

The last issue has been fixed in https://github.com/joachimmetz/dfvfs/tree/source_scanner

Note to self, think about handling VSS current volume and BDE ToGo unencrypted volume. How should they be represented in terms of the scan node hierarchy? What about location name clashes?

joachimmetz commented 9 years ago

After adding some debug output the issue seems to happen in p4. Where it looks like that the system scan loops in the "remainder" file systems analyzers which is the TSKAnalyzerHelper that tries to run pytsk3.FS_Info() on the volume. Manually analysis with TSK does not show the looping behavior, maybe something with the interaction between TSK and dfvfs? Seeing that TSK is not strictly following the POSIX file behavior this can lead to subtle issues.

joachimmetz commented 9 years ago

https://codereview.appspot.com/247350043/