search

Lazza / RecuperaBit

A tool for forensic file system reconstruction.
GNU General Public License v3.0
517 stars 73 forks source link

ValueError: cannot fit 'int' into an offset-sized integer (python3) - OverflowError: integer too large (pypy3) #88

Closed schulmenueplaner-de closed 3 years ago

schulmenueplaner-de commented 3 years ago

Hi,

I have a raw 750GB image of a drive which used to have one ntfs partition of that size, but now its showing only a 449gb ntf partition and windows can not mount the partition ("unformatted").

First run of RecuperaBit works fine, then it throws following error: (64bit debian 10 (buster) pypy3 / python3)

pypy3:

...
ERROR:root:Cannot handle multiple attribute $STANDARD_INFORMATION
ERROR:root:Cannot handle multiple attribute $STANDARD_INFORMATION
INFO:root:Parsing INDX records
INFO:root:Reading boot sectors
INFO:root:Repaired MFT entry #0 - $MFT in partition at offset 2048 from backup
INFO:root:Repaired MFT entry #1 - $MFTMirr in partition at offset 2048 from backup
INFO:root:Repaired MFT entry #2 - $LogFile in partition at offset 2048 from backup
DEBUG:root:Dropping bogus NTFS partition with MFT position 913644690 generated by MFT mirror of partition at offset 913639528
DEBUG:root:Dropping bogus NTFS partition with MFT position 913508696 generated by MFT mirror of partition at offset 913508680
DEBUG:root:Dropping bogus NTFS partition with MFT position 915820154 generated by MFT mirror of partition at offset 915814992
DEBUG:root:Dropping bogus NTFS partition with MFT position 915826346 generated by MFT mirror of partition at offset 915821184
DEBUG:root:Dropping bogus NTFS partition with MFT position 915832538 generated by MFT mirror of partition at offset 915827376
DEBUG:root:Dropping bogus NTFS partition with MFT position 915868434 generated by MFT mirror of partition at offset 915863272
DEBUG:root:Dropping bogus NTFS partition with MFT position 914665816 generated by MFT mirror of partition at offset 914665800
DEBUG:root:Dropping bogus NTFS partition with MFT position 977548198 generated by MFT mirror of partition at offset 977543036
DEBUG:root:Dropping bogus NTFS partition with MFT position 1357141790 generated by MFT mirror of partition at offset 1357136628
INFO:root:Finding partition geometry
INFO:root:Finalizing MFT reconstruction of partition at offset 2048
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
Traceback (most recent call last):
  File "main.py", line 374, in <module>
    main()
  File "main.py", line 357, in main
    parts.update(scanner.get_partitions())
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 837, in get_partitions
    self.finalize_reconstruction(part)
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 673, in finalize_reconstruction
    self.add_from_attribute_list(parsed, part, node.offset)
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 628, in add_from_attribute_list
    _integrate_attribute_list(parsed, part, image)
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 219, in _integrate_attribute_list
    dump = sectors(image, real_pos, length, 1)
  File "/root/RecuperaBit/recuperabit/utils.py", line 44, in sectors
    image.seek(offset * bsize)
OverflowError: integer too large

python3:

...
ERROR:root:Cannot handle multiple attribute $STANDARD_INFORMATION
ERROR:root:Cannot handle multiple attribute $STANDARD_INFORMATION
INFO:root:Parsing INDX records
INFO:root:Reading boot sectors
DEBUG:root:Dropping bogus NTFS partition with MFT position 914665816 generated by MFT mirror of partition at offset 914665800
INFO:root:Repaired MFT entry #0 - $MFT in partition at offset 2048 from backup
INFO:root:Repaired MFT entry #1 - $MFTMirr in partition at offset 2048 from backup
INFO:root:Repaired MFT entry #2 - $LogFile in partition at offset 2048 from backup
DEBUG:root:Dropping bogus NTFS partition with MFT position 977548198 generated by MFT mirror of partition at offset 977543036
DEBUG:root:Dropping bogus NTFS partition with MFT position 1357141790 generated by MFT mirror of partition at offset 1357136628
DEBUG:root:Dropping bogus NTFS partition with MFT position 913644690 generated by MFT mirror of partition at offset 913639528
DEBUG:root:Dropping bogus NTFS partition with MFT position 915820154 generated by MFT mirror of partition at offset 915814992
DEBUG:root:Dropping bogus NTFS partition with MFT position 915826346 generated by MFT mirror of partition at offset 915821184
DEBUG:root:Dropping bogus NTFS partition with MFT position 915832538 generated by MFT mirror of partition at offset 915827376
DEBUG:root:Dropping bogus NTFS partition with MFT position 915868434 generated by MFT mirror of partition at offset 915863272
DEBUG:root:Dropping bogus NTFS partition with MFT position 913508696 generated by MFT mirror of partition at offset 913508680
INFO:root:Finding partition geometry
INFO:root:Finalizing MFT reconstruction of partition at offset 914665800
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 2048
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
Traceback (most recent call last):
  File "main.py", line 374, in <module>
    main()
  File "main.py", line 357, in main
    parts.update(scanner.get_partitions())
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 837, in get_partitions
    self.finalize_reconstruction(part)
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 673, in finalize_reconstruction
    self.add_from_attribute_list(parsed, part, node.offset)
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 628, in add_from_attribute_list
    _integrate_attribute_list(parsed, part, image)
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 219, in _integrate_attribute_list
    dump = sectors(image, real_pos, length, 1)
  File "/root/RecuperaBit/recuperabit/utils.py", line 44, in sectors
    image.seek(offset * bsize)
ValueError: cannot fit 'int' into an offset-sized integer

I can provide the image and logfile in a VM if needed.

Regards

Ralph

Lazza commented 3 years ago

Could you please check the latest commit and try again with both?

Thank you.

schulmenueplaner-de commented 3 years ago

Hi Lazza, that worked, thank you !

I got into the CLI (pypy3), but unfortunately it only restores about 20 Gb from partition 29 and not much from partition 33 (which should be identical with 29)

> recoverable
Partition #29 -> Partition (NTFS, 698.63 GB, 10774 files, Recoverable, Offset: 2048, Offset (b): 1048576, Sec/Clus: 8, MFT offset: 6293504, MFT mirror offset: 2064)
Partition #33 -> Partition (NTFS, ??? b, 48962 files, Recoverable, Offset: 2048, Offset (b): 1048576, Sec/Clus: 8, MFT offset: 706320448, MFT mirror offset: None)
Partition #38 -> Partition (NTFS, 3.01 MB, 19 files, Recoverable, Offset: 913639528, Offset (b): 467783438336, Sec/Clus: 1, MFT offset: 913641586, MFT mirror offset: 913644690)
Partition #41 -> Partition (NTFS, 450.00 MB, 37 files, Recoverable, Offset: 913508680, Offset (b): 467716444160, Sec/Clus: 8, MFT offset: 913815880, MFT mirror offset: 913508696)
Partition #42 -> Partition (NTFS, 3.01 MB, 19 files, Recoverable, Offset: 915814992, Offset (b): 468897275904, Sec/Clus: 1, MFT offset: 915817050, MFT mirror offset: 915820154)
Partition #43 -> Partition (NTFS, 3.01 MB, 19 files, Recoverable, Offset: 915821184, Offset (b): 468900446208, Sec/Clus: 1, MFT offset: 915823242, MFT mirror offset: 915826346)
Partition #44 -> Partition (NTFS, 3.01 MB, 19 files, Recoverable, Offset: 915827376, Offset (b): 468903616512, Sec/Clus: 1, MFT offset: 915829434, MFT mirror offset: 915832538)
Partition #45 -> Partition (NTFS, 3.01 MB, 19 files, Recoverable, Offset: 915863272, Offset (b): 468921995264, Sec/Clus: 1, MFT offset: 915865330, MFT mirror offset: 915868434)
Partition #118 -> Partition (NTFS, 19.45 GB, 128499 files, Recoverable, Offset: 914665800, Offset (b): 468308889600, Sec/Clus: 8, MFT offset: 920957256, MFT mirror offset: 914665816)
Partition #119 -> Partition (NTFS, 3.01 MB, 19 files, Recoverable, Offset: 977543036, Offset (b): 500502034432, Sec/Clus: 1, MFT offset: 977545094, MFT mirror offset: 977548198)
Partition #120 -> Partition (NTFS, ??? b, 4555 files, Recoverable, Offset: 2048, Offset (b): 1048576, Sec/Clus: 8, MFT offset: 1339275104, MFT mirror offset: None)
Partition #121 -> Partition (NTFS, 3.01 MB, 19 files, Recoverable, Offset: 1357136628, Offset (b): 694853953536, Sec/Clus: 1, MFT offset: 1357138686, MFT mirror offset: 1357141790)
Partition #122 -> Partition (NTFS, ??? b, 513 files, Recoverable, Offset: 2048, Offset (b): 1048576, Sec/Clus: 8, MFT offset: 1447197184, MFT mirror offset: None)
Partition #123 -> Partition (NTFS, ??? b, 4608 files, Recoverable, Offset: 2048, Offset (b): 1048576, Sec/Clus: 8, MFT offset: 1456092400, MFT mirror offset: None)

I do have a backup for most pictures but the pictures on this drive were sorted / renamed / adjusted ...

Thanks again for RecuperaBit and you quick reply.

Lazza commented 3 years ago

I would first try to export a CSV file of the interesting partitions (e.g. 29, 33 and 118 that do seem they have a lot of files). Then study them and verify their contents.

When you say it restores "not much" what do you mean? Did you simply try to restore from the Root node or did you check the Lost Files as well?

Lazza commented 3 years ago

PS:

I got into the CLI (pypy3)

Just to be sure, is the fix also working on Python 3 rather than Pypy3? Thank you!

schulmenueplaner-de commented 3 years ago

Unfortunately your fix does not work with python3

ERROR:root:Cannot handle multiple attribute $STANDARD_INFORMATION
INFO:root:Parsing INDX records
INFO:root:Reading boot sectors
DEBUG:root:Dropping bogus NTFS partition with MFT position 914665816 generated by MFT mirror of partition at offset 914665800
INFO:root:Repaired MFT entry #0 - $MFT in partition at offset 2048 from backup
INFO:root:Repaired MFT entry #1 - $MFTMirr in partition at offset 2048 from backup
INFO:root:Repaired MFT entry #2 - $LogFile in partition at offset 2048 from backup
DEBUG:root:Dropping bogus NTFS partition with MFT position 977548198 generated by MFT mirror of partition at offset 977543036
DEBUG:root:Dropping bogus NTFS partition with MFT position 1357141790 generated by MFT mirror of partition at offset 1357136628
DEBUG:root:Dropping bogus NTFS partition with MFT position 913644690 generated by MFT mirror of partition at offset 913639528
DEBUG:root:Dropping bogus NTFS partition with MFT position 915820154 generated by MFT mirror of partition at offset 915814992
DEBUG:root:Dropping bogus NTFS partition with MFT position 915826346 generated by MFT mirror of partition at offset 915821184
DEBUG:root:Dropping bogus NTFS partition with MFT position 915832538 generated by MFT mirror of partition at offset 915827376
DEBUG:root:Dropping bogus NTFS partition with MFT position 915868434 generated by MFT mirror of partition at offset 915863272
DEBUG:root:Dropping bogus NTFS partition with MFT position 913508696 generated by MFT mirror of partition at offset 913508680
INFO:root:Finding partition geometry
INFO:root:Finalizing MFT reconstruction of partition at offset 914665800
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 2048
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
Traceback (most recent call last):
  File "main.py", line 374, in <module>
    main()
  File "main.py", line 357, in main
    parts.update(scanner.get_partitions())
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 837, in get_partitions
    self.finalize_reconstruction(part)
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 673, in finalize_reconstruction
    self.add_from_attribute_list(parsed, part, node.offset)
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 628, in add_from_attribute_list
    _integrate_attribute_list(parsed, part, image)
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 219, in _integrate_attribute_list
    dump = sectors(image, real_pos, length, 1)
  File "/root/RecuperaBit/recuperabit/utils.py", line 44, in sectors
    image.seek(offset * bsize)
ValueError: cannot fit 'int' into an offset-sized integer

in comparison to pypy3

ERROR:root:Cannot handle multiple attribute $STANDARD_INFORMATION
INFO:root:Parsing INDX records
INFO:root:Reading boot sectors
INFO:root:Repaired MFT entry #0 - $MFT in partition at offset 2048 from backup
INFO:root:Repaired MFT entry #1 - $MFTMirr in partition at offset 2048 from backup
INFO:root:Repaired MFT entry #2 - $LogFile in partition at offset 2048 from backup
DEBUG:root:Dropping bogus NTFS partition with MFT position 913644690 generated by MFT mirror of partition at offset 913639528
DEBUG:root:Dropping bogus NTFS partition with MFT position 913508696 generated by MFT mirror of partition at offset 913508680
DEBUG:root:Dropping bogus NTFS partition with MFT position 915820154 generated by MFT mirror of partition at offset 915814992
DEBUG:root:Dropping bogus NTFS partition with MFT position 915826346 generated by MFT mirror of partition at offset 915821184
DEBUG:root:Dropping bogus NTFS partition with MFT position 915832538 generated by MFT mirror of partition at offset 915827376
DEBUG:root:Dropping bogus NTFS partition with MFT position 915868434 generated by MFT mirror of partition at offset 915863272
DEBUG:root:Dropping bogus NTFS partition with MFT position 914665816 generated by MFT mirror of partition at offset 914665800
DEBUG:root:Dropping bogus NTFS partition with MFT position 977548198 generated by MFT mirror of partition at offset 977543036
DEBUG:root:Dropping bogus NTFS partition with MFT position 1357141790 generated by MFT mirror of partition at offset 1357136628
INFO:root:Finding partition geometry
INFO:root:Finalizing MFT reconstruction of partition at offset 2048
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
DEBUG:root:Found MATCH in positions {44512280} with weight 9 (100.0%)
INFO:root:Finalizing MFT reconstruction of partition at offset 2048
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 913639528
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 913508680
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 915814992
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 915821184
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 915827376
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 915863272
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 914665800
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 977543036
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
DEBUG:root:Found MATCH in positions {661972816} with weight 142 (97.93103448275862%)
INFO:root:Finalizing MFT reconstruction of partition at offset 2048
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:Finalizing MFT reconstruction of partition at offset 1357136628
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
DEBUG:root:Found MATCH in positions {722298656} with weight 14 (100.0%)
INFO:root:Finalizing MFT reconstruction of partition at offset 2048
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
DEBUG:root:Found MATCH in positions {630871856} with weight 197 (99.4949494949495%)
INFO:root:Finalizing MFT reconstruction of partition at offset 2048
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
INFO:root:Adding ghost entries from $INDEX_ALLOCATION
INFO:root:MFT for partition at offset 913639528 is fragmented. Trying to merge 2 parts...
DEBUG:root:Merging partition with MFT offset 913639512 into Partition (NTFS, 3.01 MB, 14 files, Recoverable, Offset: 913639528, Offset (b): 467783438336, Sec/Clus: 1, MFT offset: 913641586, MFT mirror offset: 913644690) (fragmented MFT)
INFO:root:MFT for partition at offset 915814992 is fragmented. Trying to merge 2 parts...
DEBUG:root:Merging partition with MFT offset 915814976 into Partition (NTFS, 3.01 MB, 14 files, Recoverable, Offset: 915814992, Offset (b): 468897275904, Sec/Clus: 1, MFT offset: 915817050, MFT mirror offset: 915820154) (fragmented MFT)
INFO:root:MFT for partition at offset 915821184 is fragmented. Trying to merge 2 parts...
DEBUG:root:Merging partition with MFT offset 915821168 into Partition (NTFS, 3.01 MB, 14 files, Recoverable, Offset: 915821184, Offset (b): 468900446208, Sec/Clus: 1, MFT offset: 915823242, MFT mirror offset: 915826346) (fragmented MFT)
INFO:root:MFT for partition at offset 915827376 is fragmented. Trying to merge 2 parts...
DEBUG:root:Merging partition with MFT offset 915827360 into Partition (NTFS, 3.01 MB, 14 files, Recoverable, Offset: 915827376, Offset (b): 468903616512, Sec/Clus: 1, MFT offset: 915829434, MFT mirror offset: 915832538) (fragmented MFT)
INFO:root:MFT for partition at offset 915863272 is fragmented. Trying to merge 2 parts...
DEBUG:root:Merging partition with MFT offset 915863256 into Partition (NTFS, 3.01 MB, 14 files, Recoverable, Offset: 915863272, Offset (b): 468921995264, Sec/Clus: 1, MFT offset: 915865330, MFT mirror offset: 915868434) (fragmented MFT)
INFO:root:MFT for partition at offset 977543036 is fragmented. Trying to merge 2 parts...
DEBUG:root:Merging partition with MFT offset 977543020 into Partition (NTFS, 3.01 MB, 14 files, Recoverable, Offset: 977543036, Offset (b): 500502034432, Sec/Clus: 1, MFT offset: 977545094, MFT mirror offset: 977548198) (fragmented MFT)
INFO:root:MFT for partition at offset 1357136628 is fragmented. Trying to merge 2 parts...
DEBUG:root:Merging partition with MFT offset 1357136612 into Partition (NTFS, 3.01 MB, 14 files, Recoverable, Offset: 1357136628, Offset (b): 694853953536, Sec/Clus: 1, MFT offset: 1357138686, MFT mirror offset: 1357141790) (fragmented MFT)
INFO:root:124 partitions found.

Write command ("help" for details):
>

118 is an old Windows Install do not need to recover that ... Thank you for the tip with the CSV file, I ll try that.

Let me know if I can assist with python3 / tests.

Lazza commented 3 years ago

Thank you for your feedback! I have pushed another commit that hopefully will help with cPython as well.

If you could try that one as well, it would be great.

schulmenueplaner-de commented 3 years ago

It also works with python3 (tested: version 3.7.3) now. Thank you !