Add ATA Secure Erase feature for HDDs & SSDs

[PartialVolume]

Add ATA secure erase after the re-worked drive selection window (#357)

[Firminator]

Have you figured out the reported problem/issue with standby? It seems that this was a roadblock until further investigation. If you need us to test or research something let us know. Probably should go into a new milestone for v0.34.

[Firminator]

Comments: https://github.com/martijnvanbrummelen/nwipe/issues/101 https://github.com/martijnvanbrummelen/nwipe/issues/282

[PartialVolume]

Have you figured out the reported problem/issue with standby?

On ShredOS, no, although there is nothing stopping me implementing ATA secure erase in nwipe first then working out why ShredOS doesn't wake from sleep.

[Firminator]

LOL, yes, that's a good mindset. I was just curious/excited thinking that you figured already out the Standby issue somehow. But yes SecureErase would be a huge feature addition making it an even more viable option for companies that have to adhere to certain guidelines/policies. [offtopic: Might want to setup a Sponsor page for your profile ( https://docs.github.com/en/sponsors/receiving-sponsorships-through-github-sponsors ) so it will be more prominent on this platform here instead of going through your buymeacoffee site.]

[Firminator]

There is an open-source tool that builds on top of PartedMagic's implementation of 'SecureErase' and verifies that the drive actually was wiped (since there is no easy way to verify that the manufacturer's FW is wiping correctly). It's open-source @ https://www.hamishmb.com/html/Docs/diskverifier.tar.gz (from https://www.hamishmb.com/html/diskverifier.php )

Description is:

Disk Verifier is an add-on for Parted Magic's Secure Erase GUI that is used for NIST 800-88 compliant disk erase verification.

We don't have the current license for PartedMagic so I can't verify or check what this looks like, but I would think that's a step in the right direction for anyone using 'SecureErase'. PartialVolume you might want to check this out and see if that's something that could seemlessly be added to whatever you envision for NWIPE's SecureErase functionality.

[PartialVolume]

I've not read all of it or looked at the source code but I'd imagine it just does a ones or maybe zeros verification ?if so, that's what nwipe can already do with nwipes two methods 'verify zeros' or 'verify ones' depending on how the SSD resets it's sectors?

Personally I don't think that type of verification is good enough. If you are paranoid about security, because with compromised firmware the drive firmware may realise it's being tested and simply return the values, i.e. ones, it thinks the verification expected. I still think verification on a PRNG stream takes verification to the next step. With PRNG unless the drive can reproduce any given sector in the stream, and it won't be able to do that unless it stored every single sector somewhere.

Remember the recent VW scandal where the cars firmware was able to recognise that it was being tested and then make adjustments to give better and misleading real world results.

Anything less than a verified PRNG stream ( also done offline, to prevent the compromised firmware saving the stream for later verification ) At least if somebody with malicious intent was writing firmware they would be looking at tricks like this to make the user think they had verified a drive. The key component is to get everybody to rely on a simple command such as 'secure erase' and simple verification. I know this is all very unlikely but ...

Good grief, I'm sounding paranoid. :-) what I think I'm getting at is that PRNG is generally a mathematical sound method. 'secure erase' for me relies too much on trust. Hoping the firmware isn't buggy, hoping the firmware isn't compromised.

What I propose with nwipe's two new wipe methods.

Secure erase #1 - security level medium Issue ATA secure erase command. Verify with zeros or ones depending upon how the drive erases itself. Nwipe will determine the appropriate method.

Secure erase #2 security level high Issue ATA secure erase command. Verify with zeros/ones as appropriate. Write PRNG stream Verify PRNG stream

The second method would allow the optional use of blanking with either a traditional zero or ones wipe or a secure erase.

[PartialVolume]

The new drive select display will also show whether the drive is frozen and nwipe will issue the sleep command on hitting 'S' to start the wipe. Before any of that though, it will first check the hardware to determine whether the hardware supports a suspend to RAM, not all motherboards can. If all good and it comes back from the sleep after a few seconds, the ATA secure erase is issued. Because you can't tell the progress of a secure wipe as far as I'm aware, nwipe will display the 'knight rider' style moving icon and no percentage wipe info, except when it's doing a traditional write or verification.

[Firminator]

That a thoroughly thought-through plan. Hats off.

[Firminator]

I think there is a SMART value that is populated by the manufacturer/vendor with how long their 'SecureWipe' is going to take. That way the user has an idea what to expect. I've seen it in Parted Magic ranging from several seconds (expected) to more than 30 minutes. Which makes you wonder why changing a the underlying cryptographic key is taking 30 minutes???!? This just shows that there is not telling what exactly the vendor's 'SecureErase' is actually executing. BlackBox.

[PartialVolume]

@Firminator As the security erase estimated duration is in the ball park but not accurate to the second or even minute (I see a ten minute discrepancy on one of my drives) I propose that when nwipe does a secure erase wipe it shows the wipe progress not as a percentage but as a T- value, i.e. If the drive smart value reports a 80 minute wipe the progress value initially reads [T-80:00] at the start of the wipe then [T-79:59] as the wipe progresses counting down to [T-00:00]. The wipe may end before T-00:00 in which case it ends when it ends irrespective of what the progress says or sometimes may take longer so then becomes [T-00:01], then [T-00:00], then [T+00:01], then [T+00:02] etc. The logs will show the actual wipe duration.

I guess this gives the user an approximate idea of how long the wipe is going to take and how much time is left but also differentiates a ATA secure wipe from a standard wipe.

[PartialVolume]

After the ATA secure erase has been completed and assuming a verification has been selected, nwipe will examine a few of the blocks to determine whether a standard erase (all zeros) or enhanced wipe (random) has been completed. On a old drive I'm testing the end result is the same, so even though I select enhanced erase it still writes zeros. So nwipe can't just assume because a enhanced erase was performed that randoms get written and same for standard erase and zeros. It then chooses the appropriate verification based on what it finds has been written.

With randoms this is harder to verify because nwipe can't reproduce the random data the drive produced as we don't know the method or seed however we can verify the data is random by analysis. Maybe using the dieharder software #327

[PartialVolume]

@Firminator Next on my list is releasing Nwipe 0.33 so Martijn can upload into Debian, I'm hoping to get that done this week, then as I promised I'll put some notes together about building ShredOS from scratch. Is there a particular Linux Distribution you want to build on? Once the notes are completed I'll probably add them to the end of README.md. After that I'm going to start adding secure erase to nwipe.

[Firminator]

I think instructions for Debian would benefit most since Debian is the foundation for the Distros that most people use (at least accd. to Distrowatch... if that is any kind of good measure).

[ggruber]

for the duration of this October I could offer a Dell R720XD running bookworm with a mix of SSDs: Samsung, Crucial, Sandisk, Zeus/STEC/HGST (and on request also Intel) and I could also plug NVME SATA and PCIe Cards in. If this would help anybody to perform development/tests I'd even offer ssh logins.

[ggruber]

disks for testing are still available and will be, I will keep this box to support nwipe development as long as {needed|usefull|requested}. could think about changing the sector size for one of the STECs back to 520.

[PartialVolume]

Just wondering if sg_scan -i sees those discs ?

Interesting blog about 520 byte drives

[PartialVolume]

https://eki.moe/posts/using-520-byte-sector-disks/

[ggruber]

@PartialVolume : you're invited to check out with the disks in the testbox , as well as the 520 bytes stuff ;-)

[PartialVolume]

Won't be able to check it out this evening or tomorrow but may have some time on Friday. Are you planning on leaving the 520 byte discs in there a while?

[ggruber]

sure I do. The 720 has become kind of part of the heating concept of our flat, way of ;-)

[PartialVolume]

sure I do. The 720 has become kind of part of the heating concept of our flat, way of ;-)

You need a couple of 300W solar panels to power that beast. 😎

[ggruber]

a challenge on a listed apartment building

[PartialVolume]

Seems it would be a challenge in the UK too. But possible, requires approval and maybe even planning permission. Probably impossible unless you own the entire building. https://www.mypoweruk.com/news/can-i-install-solar-panels-on-a-listed-building/#:~:text=If%20you%20own%20a%20listed,solar%20panels%20on%20the%20roof. Makes adding 6 x 300w panels to my pitched garage roof look a lot easier as no permission or approval required.

[ggruber]

will implement the detection of smart erase capabilities into lsdiskowipe now, following https://www.ibm.com/docs/en/linux-on-systems?topic=linuxonibm/liaau/t10pi.htm https://www.ibm.com/docs/en/linux-on-systems?topic=devices-secure-data-deletion-sas-drives https://www.ibm.com/docs/en/linuxonibm/liaau/secure_sata.html and https://www.ibm.com/docs/en/linuxonibm/liaau/secure_nvme.html

just erasing 12 SAS SSDs with sg_sanitize which I suggest to call from nwipe, too. But the complete read to verify all zeroes on disks could/should come from nwipe natively, shouldn't it?

[PartialVolume]

@ggruber

But the complete read to verify all zeroes on disks could/should come from nwipe natively, shouldn't it?

Yes, agreed. However a new verify feature may need to be implemented. A sort of smart verify. The reason I say this is that I've come across at least one SSD that doesn't end up with zeros or ones but a repeating pattern. In every block there was a line of 3F3F....

I don't know if this was some weird fault in the SSD or this was the pattern it used to wipe the drive. Something to think about, that sometimes a wipe by firmware might not end with all zeros or ones.

What are you typically seeing on your SSDs after a wipe by the drives firmware?

[ggruber]

root@thelia:~# hexdump -C /dev/sdz
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
1bf1fc56000
root@thelia:~#

did this only for one disk so far, it will run now over night for all the 12 SSDs now

Before sanitizing I had seen remains of the former content of the disks.

[ggruber]

btw, it was the same for all 12 of them

[PartialVolume]

Just to clarify, so everything is looking good and as you would expect?

[ggruber]

everything went as expected (back in November) the disks have left my site weeks ago. I'm currently struggeling with user 3,5" 4TB SAS disks with enabled encryption.

[PartialVolume]

everything went as expected (back in November) the disks have left my site weeks ago.

Good to hear 👍

I'm currently struggling with user 3,5" 4TB SAS disks with enabled encryption.

What's the problem with those?

[ggruber]

they had encryption enabled but were not accessible in the sense of hexdump /dev/sdc only resulted in error. smartctl worked flawlessly, but hdparm -I didn't. Just error. Meanwhile I run SeaTools to make the disks usable, what will take a week for the 4TB disks.

[tnu646]

I've not read all of it or looked at the source code but I'd imagine it just does a ones or maybe zeros verification ?if so, that's what nwipe can already do with nwipes two methods 'verify zeros' or 'verify ones' depending on how the SSD resets it's sectors?

Personally I don't think that type of verification is good enough. If you are paranoid about security, because with compromised firmware the drive firmware may realise it's being tested and simply return the values, i.e. ones, it thinks the verification expected. I still think verification on a PRNG stream takes verification to the next step. With PRNG unless the drive can reproduce any given sector in the stream, and it won't be able to do that unless it stored every single sector somewhere.

Remember the recent VW scandal where the cars firmware was able to recognise that it was being tested and then make adjustments to give better and misleading real world results.

Anything less than a verified PRNG stream ( also done offline, to prevent the compromised firmware saving the stream for later verification ) At least if somebody with malicious intent was writing firmware they would be looking at tricks like this to make the user think they had verified a drive. The key component is to get everybody to rely on a simple command such as 'secure erase' and simple verification. I know this is all very unlikely but ...

Good grief, I'm sounding paranoid. :-) what I think I'm getting at is that PRNG is generally a mathematical sound method. 'secure erase' for me relies too much on trust. Hoping the firmware isn't buggy, hoping the firmware isn't compromised.

What I propose with nwipe's two new wipe methods.

Secure erase #1 - security level medium Issue ATA secure erase command. Verify with zeros or ones depending upon how the drive erases itself. Nwipe will determine the appropriate method.

Secure erase #2 security level high Issue ATA secure erase command. Verify with zeros/ones as appropriate. Write PRNG stream Verify PRNG stream

The second method would allow the optional use of blanking with either a traditional zero or ones wipe or a secure erase.

I've not read all of it or looked at the source code but I'd imagine it just does a ones or maybe zeros verification ?if so, that's what nwipe can already do with nwipes two methods 'verify zeros' or 'verify ones' depending on how the SSD resets it's sectors?

Personally I don't think that type of verification is good enough. If you are paranoid about security, because with compromised firmware the drive firmware may realise it's being tested and simply return the values, i.e. ones, it thinks the verification expected. I still think verification on a PRNG stream takes verification to the next step. With PRNG unless the drive can reproduce any given sector in the stream, and it won't be able to do that unless it stored every single sector somewhere.

Remember the recent VW scandal where the cars firmware was able to recognise that it was being tested and then make adjustments to give better and misleading real world results.

Anything less than a verified PRNG stream ( also done offline, to prevent the compromised firmware saving the stream for later verification ) At least if somebody with malicious intent was writing firmware they would be looking at tricks like this to make the user think they had verified a drive. The key component is to get everybody to rely on a simple command such as 'secure erase' and simple verification. I know this is all very unlikely but ...

Good grief, I'm sounding paranoid. :-) what I think I'm getting at is that PRNG is generally a mathematical sound method. 'secure erase' for me relies too much on trust. Hoping the firmware isn't buggy, hoping the firmware isn't compromised.

What I propose with nwipe's two new wipe methods.

Secure erase #1 - security level medium Issue ATA secure erase command. Verify with zeros or ones depending upon how the drive erases itself. Nwipe will determine the appropriate method.

Secure erase #2 security level high Issue ATA secure erase command. Verify with zeros/ones as appropriate. Write PRNG stream Verify PRNG stream

The second method would allow the optional use of blanking with either a traditional zero or ones wipe or a secure erase.

I've not read all of it or looked at the source code but I'd imagine it just does a ones or maybe zeros verification ?if so, that's what nwipe can already do with nwipes two methods 'verify zeros' or 'verify ones' depending on how the SSD resets it's sectors?

Personally I don't think that type of verification is good enough. If you are paranoid about security, because with compromised firmware the drive firmware may realise it's being tested and simply return the values, i.e. ones, it thinks the verification expected. I still think verification on a PRNG stream takes verification to the next step. With PRNG unless the drive can reproduce any given sector in the stream, and it won't be able to do that unless it stored every single sector somewhere.

Remember the recent VW scandal where the cars firmware was able to recognise that it was being tested and then make adjustments to give better and misleading real world results.

Anything less than a verified PRNG stream ( also done offline, to prevent the compromised firmware saving the stream for later verification ) At least if somebody with malicious intent was writing firmware they would be looking at tricks like this to make the user think they had verified a drive. The key component is to get everybody to rely on a simple command such as 'secure erase' and simple verification. I know this is all very unlikely but ...

Good grief, I'm sounding paranoid. :-) what I think I'm getting at is that PRNG is generally a mathematical sound method. 'secure erase' for me relies too much on trust. Hoping the firmware isn't buggy, hoping the firmware isn't compromised.

What I propose with nwipe's two new wipe methods.

Secure erase #1 - security level medium Issue ATA secure erase command. Verify with zeros or ones depending upon how the drive erases itself. Nwipe will determine the appropriate method.

Secure erase #2 security level high Issue ATA secure erase command. Verify with zeros/ones as appropriate. Write PRNG stream Verify PRNG stream

The second method would allow the optional use of blanking with either a traditional zero or ones wipe or a secure erase.

"To be paranoid does not mean you are not followed"

As for matter, every efforts should be weighted with data importance and possibility to aquire that data by other means.

As you mentioned, there could be malicious intent (National security...,) bug in firmware or algorithm, possible new forensic approaches in future.

The only way be sure is physical destruction. NIST states also degaussing, but have you in-home environment sufficient equipment ?

Dismantling must be absolute. It means hammer or drilling is not sufficient for sensitive data. You could not be sure you destroyed all disks plates beyond possible bit recovery. Of course, now we talking in terms of high sensitive data.