Decrypting to GPT (NTFS) taking a long time

[vanadu]

I have a decryption running using reallymine Release 2 on Ubuntu 19.04 HPZ400 Xeon X5670. After two hours, not quite 4 GB of the 2TB is decrypted to the img, which is writing to a 3TB drive formatted in Windows as GPT. In issue #47, @stynoo commented that "that dumping the data to an ntfs formatted drive will take a loooooong time". I calculate this dump will take literally weeks. Would it be possible and/or faster to write the img to an ext4 formatted drive? I read in one of the posts that a user had a transfer speed of several hundred GB in a few hours. How can I achieve that? Thanks for any help...

[themaddoctor]

Why don't you decrypt it into a raw disk, i.e., not as an image file, but as a clone?

[vanadu]

I didn't know that was a thing. I see 'decrypt disk outfile' in the reallymine help - if there's a clone command I'm missing it. My DEK is:

bridge type JMicron DEK: AED4637AEB75BEBC65FEE8F8A34C91EBB336E1D4C92B0575EF6C81F8718CFCE2 decryption steps: reverse decrypt reverse

@themaddoctor, I'm thinking maybe I should just try the decryption procedure in your PDF that @Adriana-decora detailed in #47.

[themaddoctor]

Isn't there a way to make the output file = full disk? @andlabs

[themaddoctor]

In linux, set the output file to be "/dev/sdX" wherfe X is the number of the target disk. Be carefull-- if you make a typo you could wipe out the source disk or the system disk.

[vanadu]

OK I'll try it tomorrow. I'm really not too worried about the source or system disk. The former is mostly stupid, legally downloaded German 'Tatort' shows that nobody including me cares much about and the latter is my backup machine and the system is backed up. So I'll be careful within reason.

[vanadu]

I tried this using a 3TB drive unallocated disk, and then formatted the disk in Ubuntu as GPT:

sudo ./reallymine decrypt /dev/sdb /dev/sdc

and got the same result both times: 'error running decrypt: open /dev/sdc: file exists.

That's the same error I got when I tried to output to an img with an already-existing filename. There are no partitions on the target disk because it's a clone target. Any ideas?

[themaddoctor]

You'll have to ask @andlabs . He should know how to get it to work.

The procedure in my PDF is more complicated, because you need to build a kernel module, but it can be used to write to a bare disk.

[vanadu]

OK... thanks for your help and I'll try the PDF approach if it's not possible to write output to anything but img. @andlabs, thanks for reallymine, it's awesome... any ideas for outputting decrypt straight to disk or any way to accelerate the output? I can't use my primary machine for the decrypt and it will take weeks with my older Xeon.

[stynoo]

In issue #47, @stynoo commented that "that dumping the data to an ntfs formatted drive will take a loooooong time". I calculate this dump will take literally weeks. Would it be possible and/or faster to write the img to an ext4 formatted drive?

Yes, iirc I blamed the ntfs driver and ended up creating a zfs pool as I needed a bit more storage anyways. Ext4 or any other native linux fs should speed things up..

[vanadu]

@stynoo you are the bomb! 500MB per minute writing to ext4 vs 1MB per eternity writing to GPT... @andlabs if you're still out there I'd highly recommend adding this info to the notes. Hopefully I'll have this done today... will keep you posted.

[MrDecay]

Excellent news.. Now I wonder if it would be the same in transferring the decrypted data to the gpt ntfs. Or is it a slowed down related to the gpt structure

On Sat, May 18, 2019, 11:19 AM vanadu notifications@github.com wrote:

@stynoo https://github.com/stynoo you are the bomb! 500MB per minute writing to ext4 vs 1MB per eternity writing to GPT... @andlabs https://github.com/andlabs if you're still out there I'd highly recommend adding this info to the notes. Hopefully I'll have this done today... will keep you posted.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/andlabs/reallymine/issues/89?email_source=notifications&email_token=AEATVRP5542JTSND3XE4N33PWAUBVA5CNFSM4HNXCOAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVWRQSA#issuecomment-493688904, or mute the thread https://github.com/notifications/unsubscribe-auth/AEATVRLSHODZKU6Z7KZSPA3PWAUBVANCNFSM4HNXCOAA .

[vanadu]

We shall see, hopefully by tomorrow. But even if it takes a month to copy the decrypted files, I can live with that. The hard part is not knowing whether the data are toast or not.

Edit: Even at this rate it will take a couple days at least to build the img file...but that's still much better than several weeks. I'll let you know how it comes out when it's done.

[MrDecay]

What you can do check to see if your decrypting is working. is run photorec on the image part that you already decrypted and if it exports files...usually jpg or graphics..then you can verify quickly if the process is working properly

Photorec and testdisk is usually bundled or easily installable.

On Sat, May 18, 2019, 11:40 AM vanadu notifications@github.com wrote:

We shall see, hopefully by tomorrow. But even if it takes a month to copy the decrypted files, I can live with that. The hard part is not knowing whether the data are toast or not.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/andlabs/reallymine/issues/89?email_source=notifications&email_token=AEATVRN2CVJ76JG4BSS6JNTPWAWPFA5CNFSM4HNXCOAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVWR3FY#issuecomment-493690263, or mute the thread https://github.com/notifications/unsubscribe-auth/AEATVRKFF33UTNMFT57GTHTPWAWPFANCNFSM4HNXCOAA .

[vanadu]

Thanks @MrDecay -- photorec was able to access readable data in all formats in the 20 GB already output to img. Filenames were not intact, but I think it was just accessing deleted data, and I know from experience with Photorec that it can't restore filenames of deleted files. I'll just hope that when it gets to the undeleted existing folder structure the filenames will be intact and all that (probably quite useless) data will be whole again. But it's the principle! Even if I can live without all those old TV shows, WD had no right to rob me of them. Screw WD! BTW so far it looks quite promising so thanks again @themaddoctor for your help and @andlabs for this awesome tool!

[MrDecay]

Just to clarify...photorec is a data carver..imagine ..it scans from sector zero to the end sector that is available..in your case I think you said 20 gigs...looking for header and footer info. (Begining of a file to the end of the file) then copies it out to a file..bypassing names and folder structure pertaining to the file system..once its done decrypting you could either mount it using a tool of choice...or extract the files to yet another. Destination...testdisk can parse the file system and export all the files with structure intact

On Sat, May 18, 2019, 12:10 PM vanadu notifications@github.com wrote:

Thanks @MrDecay https://github.com/MrDecay -- photorec was able to access readable data in all formats in the 20 GB already output to img. Filenames were not intact, but I think it was just accessing deleted data, and I know from experience with Photorec that it can't restore filenames of deleted files. I'll just hope that when it gets to the undeleted existing folder structure the filenames will be intact and all that (probably quite useless) data will be whole again. But it's the principle! Even if I can live without all those old TV shows, WD had no right to rob me of them. Screw WD!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/andlabs/reallymine/issues/89?email_source=notifications&email_token=AEATVRO6AY72AHGON5CEUELPWA2A3A5CNFSM4HNXCOAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVWSKZQ#issuecomment-493692262, or mute the thread https://github.com/notifications/unsubscribe-auth/AEATVRNBQJPXSLOE3A3WP4TPWA2A3ANCNFSM4HNXCOAA .

[vanadu]

OK, thanks. When it's done, I'll probably try to mount the image and just copy the data. Right now, I'm just happy to take a break while it runs. This whole thing has given me a headache!

[themaddoctor]

sounds like good news. congrats

[vanadu]

OK, all done, decrypted files are copying to backup. First -- thanks to @andlabs from me and any other unsuspecting purchasers of these WD drives from hell, and to @themaddoctor, @stynoo and @MrDecay for their help. Summary: reallymine does really work, but you need an extra drive with equal space as the encrypted WD drive that you can reformat as ext4 and a lot of time to babysit the decryption run, which can take days - I had to abort the first try after 24 hours due to low voltage issue on the UPS. In retrospect, since I have a lot of command line experience I wish I'd just taken the time to do the process @themaddoctor detailed here:

(https://github.com/themaddoctor/linux-mybook-tools/blob/master/Mounting%20encrypted%20WD%20disks%20in%20linux.pdf)

If you don't have any Linux or command line experience, reallymine is definitely your better option. @themaddoctor's process would have actually saved me a lot of time because the result is a decrypted source drive, i.e. no need to run a day-long process to build a decrypted image on a separate ext4 drive.

Runtime for this process was about 100 hours on a HP Z400 Xeon X5670 running Ubuntu 19.4 Live USB with the WD drive connected to boot drive port and target drive connected to DVD/ROM port. If you're thinking about doing this with reallymine, you should have a spare machine to boot to Linux and to run the decryption process on, and you should connect both the source and target drive to SATA ports internally. The target drive should be clean formatted as ext4. If you try to write the decrypted image to an NTFS/GPT drive it's at least 20X slower and can take literally weeks. The decrypted image opened right up for me in Ubuntu Nautilus (Right-click on img, select Disk Mounter). I'm copying the data from the decrypted image to an NTFS drive, and it's copying at regular speed, so that's not an issue.

BTW -- once the decrypted image was opened on the target drive and I'd confirmed that the data was intact, I just reformatted the original WD 2 TB drive as NTFS/GPT and am now copying all the data from the img back to it. Ideally, I'd have yet another 2 TB drive to copy the img data to, and keep the original encrypted WD drive as backup, but... and this is why @themaddoctor's process has advantages because it makes all that copying around unnecessary. You just need to clone the source disk once for backup and then decrypt it - done.

So thanks again, and to anyone who has a WD MyBook - do this now because that encryption chip card WILL fail eventually and they you'll be back here all stressed out and sans data.

[vanadu]

So thanks again, and to anyone who has a WD MyBook - do this now because that encryption chip card WILL fail eventually and they you'll be back here all stressed out and sans data.

What I meant to say was, back your data up and toss that WD encryption chip in the garbage OR ELSE you'll be back here all stressed out and sans data :-)

[Adriana-decora]

Congratulations, @vanadu! Another battle won againgst the confiscation of data by WD! In my case, owning a WD My Book with JMicron JMS538S chip and no experience in Linux, the decryption option described in "Mounting encrypted WD disks in linux.pdf" and the patient help of @themaddoctor has been definitely the best and fastest option: I recovered all my files (3TB)

[hasplal]

Thanks @vanadu @stynoo I have A 2TB to decrypt and got A speed of 350Kb/s on NTFS (should take about 2 month +)

Now on an EXT4 i'm getting about x30 times faster 10Mb/s

Thanks