New Formats (wishlist) - Githubissues

kholia commented 11 years ago

[x] Improve DMG formats (sparsebundle, sparseimage)
[x] Fast and better support for fast hashes (sayantan, all)
[x] Improve 7z support (dhiru, all)
[x] MS OneNote (2003 to 2013) http://www.microsoft.com/en-in/download/details.aspx?id=14565 (OfficeFileFormatsProtocols.zip)
[x] MS Windows EFS https://bitbucket.org/jmichel/dpapick
[x] MS Windows BitLocker https://github.com/libyal/libbde (Elena has a PR #2427)
[x] MS Access >= 2007 (ACCDB)
[x] MS Access (2003 to 2007, Google "DocRecrypt")
[x] BestCrypt 8.0 (JBC) http://www.jetico.com/bcmac-help/AB_containerfile.html http://www.jetico.com/linux/bcrypt-help/b_cntff.htm

alienjax commented 11 years ago

Update dmg2john to work with .sparseimage and .sparsebundle

kholia commented 11 years ago

@gat3way do you have support for .sparseimage and .sparsebundle in your DMG plug-in?

gat3way commented 11 years ago

I think it's broken right now...the dmg plugin will accept the file, but cracking would likely give you false negatives.

On Wed, Aug 28, 2013 at 10:00 AM, Dhiru Kholia notifications@github.comwrote:

@gat3way https://github.com/gat3way do you have support for .sparseimage and .sparsebundle in your DMG plug-in?

— Reply to this email directly or view it on GitHubhttps://github.com/magnumripper/JohnTheRipper/issues/359#issuecomment-23395041 .

alienjax commented 11 years ago

dmg2john.py will accept a .sparseimage and output a hash, but john won't output the password.

A .sparsebundle disk image is basically a folder/bundle with a directory called "Bands" inside with the entire contents of the disk divided in ~8mb chunks, numbered by hex. (although some chunks maybe smaller or missing entirely)

There's a small file (122kb) inside the .sparsebundle called "token" which appears to be the header part, starting with "encrcdsa". (An unencrypted .sparsebundle will have a token file with 0kb) Using dmg2john on "token" will return an error due to the following:

   # read last chunk
    fd.seek(dataoffset, 0)
    chunk2 = fd.read(4096)
    if len(chunk2) != 4096:
        sys.stderr.write("%s is not a DMG file!\n" % filename)
        return

When I used cat to join "token" with /Bands/0 (or even all the bands) dmg2john produced a hash file, but again john didn't output the password.

kholia commented 11 years ago

@alienjizz are .sparseimage and .sparsebundle formats "popular" enough?

I will take a look (at some point) but @gat3way might be faster at it.

@magnumripper does this look like something you would be interested in?

gat3way commented 11 years ago

I think the only thing that needs to be done for that format are the heuristic checks in the dmg, the ones that we perform for dmg apparently do not work for sparseimage. Other than this, everything should be the same for both dmg and sparseimage. Some sample sparseimages would be needed to validate. I spent some time on that a while ago but then a higher priority stuff came so I left it unfinished. It should be pretty straightforward to deal with once you got enough samples I believe.

Regards, Milen.

On Thu, Aug 29, 2013 at 1:12 PM, Dhiru Kholia notifications@github.comwrote:

@alienjizz https://github.com/alienjizz are .sparseimage and .sparsebundle formats "popular" enough?

I will take a look (at some point) but @gat3wayhttps://github.com/gat3waymight be faster at it.

@magnumripper https://github.com/magnumripper does this look like something you would be interested in?

— Reply to this email directly or view it on GitHubhttps://github.com/magnumripper/JohnTheRipper/issues/359#issuecomment-23479234 .

magnumripper commented 11 years ago

On 29 Aug, 2013, at 12:12 , Dhiru Kholia notifications@github.com wrote:

@magnumripper does this look like something you would be interested in?

I don't think so. I had not even heard of it until now.

kholia commented 11 years ago

@magnumripper I now know why you are so busy and non-attentive these days!

magnum-selling-icecreams-small

Stop focussing on your ice-cream business and write code ;)

magnumripper commented 11 years ago

Oh, but the ice cream biz is so much cooler (lulz I kill me)

alienjax commented 11 years ago

@kholia It's hard to judge what's popular "enough", Sparse Bundle was introduced in OSX Leopard, it has the sparse capability (to grow as needed) and is divided in chunks. It's used by "Time Machine" - which can use encryption, and by FileVault (up until Lion). The internet is full of requests for help with .sparsebundle disk images by people using these features.

Disk Utility has a "sparse bundle disk image" option when creating disk images, it looks and behaves like a plain DMG disk image, and afaik uses the same encryption technique, as evident by the "token" file starting with "encrcdsa".

As a matter of fact I believe the samples you have up at https://github.com/kholia/VileFault/tree/master/tests are the same as a .sparsebundle created by Disk Utility. (the first four files)

magnumripper commented 11 years ago

Aha, so we have samples. Maybe I'll have a look at it.

So what is the difference between sparse image and sparse bundle?

magnumripper commented 11 years ago

OK, re-reading all the above I think I get it:

for sparse image, we need to update the heuristics in the dmg format.
for sparse bundle, we need to update dmg2john (for a starter).

kholia commented 11 years ago

I am committing new disk images to https://github.com/kholia/moar-apple-disk-images.

Let me know if you guys need something specific.

At some point, I will merge these images into VileFault repository.

alienjax commented 11 years ago

I fiddled around a bit more with the various file types, and have some files that I can upload to help with the process. If you tell me how/where to upload them I will gladly do so.

This is a bundle directory layout:
name.sparsebundle/               <- appears as single file
name.sparsebundle/token          <- encryption header kept in an individual file
name.sparsebundle/bands/<0...ff> (up to 10,000 files or more)

A re-joined unencrypted bundle (.sparsebundle/bands/<0...ff> topped off by .sparsebundle/token) is apparently identical to a .dmg file and is mounted by the system without a hitch. Encrypted, the system will authenticate the disk image correctly but gives a "Not Recognized" error. The following is from the log:

diskimages-helper[46907]: CEncryptedEncoding: need to repair aj.dmg
diskimages-helper[46907]: expected length: 122368, actual length: 15953408

Joining .sparsebundle/bands/... while leaving token in place didn't give any errors, it mounted normally.

The following are byte positions/length of various data chunks within the disk images. 

Encrypted disk image:
+--------------------+-------------------+------------------+------------------------+
|                    | DMG               | Sparse Image     | Sparse Bundle (joined) |
+--------------------+-------------------+------------------+------------------------+
| "encrcdsa" header  | 0...122368        | 0...122368       | 0...122368             |
| <encrypted data>   | 122368...<EOF>    | 122368...<EOF>   | 122368...<EOF>         |
| Ending block       | <encrypted data>  | <null>           | <encrypted data>       |
+--------------------+-------------------+------------------+------------------------+

Unencrypted disk image:
+--------------------+-------------------+------------------+------------------------+
|                    | DMG               | Sparse Image     | Sparse Bundle (joined) |
+--------------------+-------------------+------------------+------------------------+
| Beginning block    | <null>            | sprs             | <null>                 |
| HFS+ signature     | 1024              | 5120             | 1024                   |
| Ending block       | <null>            | <null>           | <null>                 |
+--------------------+-------------------+------------------+------------------------+

In many aspects a sparse seems more different than the other two.

magnumripper commented 11 years ago

If they are not too large they can be uploaded to http://openwall.info/wiki/john/sample-non-hashes#Apple-DMG-files

alienjax commented 11 years ago

@magnumripper Thanks! The information has been added.

Sample files are here https://github.com/alienjizz/UDIF

alienjax commented 11 years ago

Got it figured out! The bundle is missing the "datasize" int64 at decimal byte 56 in the token file, and can be easily extrapolated by "dataoffset". The data chunks needed for scrapping are located in .sparsebundle/bands/0, and can be located by:

datasize = os.path.getsize(".sparsebundle/bands/0") - dataoffset

afaik with a .sparsebundle it's always 8388608 for the file size and 122368 for offset. (variable names taken from dmg2john.py)

magnumripper commented 11 years ago

Excellent! If you produce any code I'll be happy to accept a pull request.

mledford commented 11 years ago

I have a sparseimage I am trying to work with but john won't recognize the output from dmg2john. Doing some detective work got me here. It seems that a sparseimage is different enough that it won't work. What is the status for this support? Is there anything I can do to lend a hand in getting it working?

kholia commented 11 years ago

@mledford if you can convince @magnumripper to stop killing me over the valid() stuff, I can try to add this support this coming weekend.

@mledford let me warn you that @magnumripper is hard to please.

magnumripper commented 11 years ago

LOL, I rather see you spend time on DMG before any hardening, any day of the week. The valid safety demands ultimately comes from Solar - I'm just trying to help! BTW I fixed some of my own formats now too, I'm not that much better than you (it's easier for me cause I wrote like five formats and you wrote more like 500).

mledford commented 11 years ago

Great!

In doing some more research I found the following: http://nah6.com/~itsme/cvs-xdadevtools/iphone/tools/dumpencrdsa.pl

I haven't had a chance to use it yet but it looks like it might fill in some holes in your header. For example my sparseimage has the value 0x02 for the keycount and keys field documented at the end of that linked file. Furthermore, when I run:

hdiutil isencrypted file.sparseimage -plist

for the sparseimage I'm working with it returns:

...
<plist version="1.0">
<dict>
    <key>blocksize</key>
    <integer>4096</integer>
    <key>encrypted</key>
    <true/>
    <key>max-key-count</key>
    <integer>2</integer>
    <key>passphrase-count</key>
    <integer>1</integer>
    <key>private-key-count</key>
    <integer>1</integer>
    <key>private-keys</key>
    <array>
        <dict>
            <key>public-key-hash</key>
            <data>
            [REDACTED]
            </data>
        </dict>
    </array>
    <key>uuid</key>
    <string>[REDACTED]</string>
    <key>version</key>
    <integer>2</integer>
</dict>
</plist>

Hopefully this is helpful.

Also, let me know if this is the proper place to continue talking about this or if it should be moved elsewhere.

mledford commented 11 years ago

I have solved part of the problem by modifying dmg2john to be able to handle multiple keys. In my particular sparseimage I have two keys—a password and a private key. Using the aforementioned perl script as my guide I have been able to modify part of dmg2john to search for and use the first password header that is found. For the sparseimage I am working with I am now at the same point I believe you were mentioning earlier in the thread. I'll clean up the code and provide a pull request if you are interested.

kholia commented 11 years ago

@mledford

That would be great. Please go ahead.

mledford commented 11 years ago

@kholia

Good news! I made some additional changes that should allow encrypted sparseimage files to be recognized. It works on some test sparseimage files I created and even better I was able to recover the password on the sparseimage file I've been working with! I'll clean up the code to, hopefully, make it more acceptable and send the pull request.

kholia commented 11 years ago

@mledford

Awesome. I am waiting for it :-)

mledford commented 11 years ago

@kholia

Pull request submitted. https://github.com/magnumripper/JohnTheRipper/pull/390

magnumripper commented 10 years ago

Edited OP to be a Task List

kholia commented 9 years ago

BitLocker stuff is in progress now (https://github.com/kholia/libbde).

stephanschielke commented 8 years ago

Voting for BestCrypt (.jbc) container.

kholia commented 8 years ago

https://www.jetico.com/linux/BestCrypt-2.0.10.tar.gz (has source code which is great).

Updated package is available at https://www.jetico.com/linux/BestCrypt-3.0.tar.gz.

https://www.jetico.com/bdk.zip (BestCrypt Development Kit) also seems useful.

Local copy of BestCrypt-3.0.0.tar.gz.