SSD erase

[PartialVolume]

Not an issue but an enhancement. A new feature could be added to Nwipe to identify SSD drives and default to the internal firmware erase method followed by an optional verification pass for these type of drives.

For a company wiping many systems that contain a mix of SSDs and mechanical drives this would improve wipe speed and avoid any unnecessary wiping of an SSD using the traditional method. The SSD internal wipe method could be optionally overridden if required.

[mdcato]

I'll generalize this and say the internal firmware erase can used on SSD and spinning media (if new enough to support it). I've been doing the firmware erase on spinning media after nwipe as an additional protection for drives that are new/large enough to be potentially re-used. (I've also improved nwipe's total throughput calculation, but that'll be a different post.)

[PartialVolume]

I didn't realise newer spinning discs had a firmware erase capability, in which case yes that option should also be available.

What procedure/software are you currently using for firmware erase for SSDs and HDDs ? I know next to nothing about erasing SSDs so any pointers and links to accessing the hardware is appreciated. The only thing I've read so far is about using hdparm.

[mdcato]

I currently use a bash script where the heart of the work is done by HDPARM: hdparm --user-master u --security-erase[-enhanced] /dev/xxxx The down side is that the device must NOT be frozen as is done by default by the BIOS/UEFI, so you have to remove and re-apply power to the device; hardly an automatic operation to be done on more than a small number of devices. I've also seen that putting the computer to sleep and waking it back up will unfreeze the drives, but again it's not an automatic procedure.

[PartialVolume]

Yes, the frozen drive issue is a bit messy. Do you use

echo -n mem > /sys/power/state

to remove power? Feels like there's got to be a slicker way of doing this.

[mdcato]

I physically pull the power cable -- very crude. Yes, there should be a slicker way for wiping purposes, but I don't think there is a standard for it, and it would be opposed for security reasons on running systems (IMHO). (Imagine malware that unfreezes, and then erases.)

[PartialVolume]

I've just been reading about the freeze issue. It's nice to see windows users also have the same problem and it's not just a Linux issue.. 😉

I just tried using sleep on my laptop and it did indeed unfreeze the drive allowing it to be wiped. It shouldn't be too hard to add some calls to hdparm from within nwipe, either that or dig into hdparm to find the code that's needed, strip it out and use it directly or maybe a bit of both. Regarding altering the ncurses interface to accommodate a firmware or traditional wipe. One way would be, with the drive highlighted using up/down arrow keys you press the space bar as is currently the case but if you then press the space bar multiple times it cycles through X (for traditional wipe), F (for firmware wipe) and B (for traditional wipe followed by firmware wipe). I'm not sure I'm using the correct terminology here. Firmware wipe sounds like I'm trying to wipe the drives firmware rather than erase the drive using the drives inbuilt firmware.

[louib]

I'm not sure I'm using the correct terminology here. Firmware wipe sounds like I'm trying to wipe the drives firmware rather than erase the drive using the drives inbuilt firmware.

maybe manufacturer wipe would be appropriate?

I think for now we could at least document (in manpage for example) that doing a "native" wipe is possible with some storage devices, and link to some appropriate tooling to do so (hdparm documentation for example). That way users know about this option even if it's not (yet) implemented in nwipe.

[gaston-senac]

Is nwipe currently able to erase a SSD drive? I'm a bit confused about the ability to erase SSDs and why it's different than a normal HDD from nwipe point of view. Also I thought that nwipe was already using the firmware wipe if available. Thanks.

[PartialVolume]

Nwipe can erase an SSD using the traditional writing to every block method. In fact I did that on a 240GB SSD drive a few days ago. Whether that's advisable or not is a matter for debate. However nwipe does not currently support activating the drives internal erase command. For that you would need to use hdparm. It's something that may be added to nwipe in the future.

No, nwipe does not use the firmware erase if available.

[anarcat]

Nwipe can erase an SSD using the traditional writing to every block method. In fact I did that on a 240GB SSD drive a few days ago. Whether that's advisable or not is a matter for debate.

All the literature I have found on how SSD work tell me this is not "a matter for debate", it is a fact that blanking an SSD will not remove the data reliably from the drive. SSDs have spare capacity and do not actually delete or even overwrite data when you write over a file or a sector: it allocates a new one and marks the older one as available (and that's only if "TRIM" commands are sent).

some references:

http://kcall.co.uk/ssd/ https://security.stackexchange.com/questions/62253/what-is-the-difference-between-ata-secure-erase-and-security-erase-how-can-i-en https://arstechnica.com/information-technology/2011/03/ask-ars-how-can-i-safely-erase-the-data-from-my-ssd-drive/

[mdcato]

@anarcat, This is similar to an earlier discussion on SSDs. Some of the articles you mentioned have better explanations than earlier references. To summarize the earlier discussion, SSD controllers recognize data patterns, such as all zeros, and map all blocks containing zeros to a single block that really contains zeros. Earlier data from those re-mapped blocks still exists and technically could be forensically read (but not with normal reads). One security method is to write random data to the entire device (low likelihood of repeated data in a block which would allow shared mapping), which nwipe currently supports. Then give the SSD controller the Security Erase Enhanced command, which nwipe does NOT currently support, but can be done with HDPARM in the interim. Or perhaps swap the order to do Security Erase Enhanced first then wipe with random data.

From: anarcat notifications@github.com Sent: Wednesday, February 12, 2020 09:10 To: martijnvanbrummelen/nwipe nwipe@noreply.github.com Cc: Mike Cato / Hays Technical Services mike@haystech.com; Comment comment@noreply.github.com Subject: Re: [martijnvanbrummelen/nwipe] SSD erase (#101)

Nwipe can erase an SSD using the traditional writing to every block method. In fact I did that on a 240GB SSD drive a few days ago. Whether that's advisable or not is a matter for debate.

All the literature I have found on how SSD work tell me this is not "a matter for debate", it is a fact that blanking an SSD will not remove the data reliably from the drive. SSDs have spare capacity and do not actually delete or even overwrite data when you write over a file or a sector: it allocates a new one and marks the older one as available (and that's only if "TRIM" commands are sent).

some references:

http://kcall.co.uk/ssd/ https://security.stackexchange.com/questions/62253/what-is-the-difference-between-ata-secure-erase-and-security-erase-how-can-i-en https://arstechnica.com/information-technology/2011/03/ask-ars-how-can-i-safely-erase-the-data-from-my-ssd-drive/

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/martijnvanbrummelen/nwipe/issues/101?email_source=notifications&email_token=ANGK2PWWC52PIVLJ5PGOQOLRCQGMBA5CNFSM4IVNIMC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELRDTJA#issuecomment-585251236, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANGK2PUAU4QVD2YXBINCFSDRCQGMBANCNFSM4IVNIMCQ.

[anarcat]

To summarize the earlier discussion, SSD controllers recognize data patterns, such as all zeros, and map all blocks containing zeros to a single block that really contains zeros.

That's news to me: I wasn't aware of SSDs doing data replication.

Earlier data from those re-mapped blocks still exists and technically could be forensically read (but not with normal reads). One security method is to write random data to the entire device (low likelihood of repeated data in a block which would allow shared mapping), which nwipe currently supports.

The problem with that approach is that you don't have access to the entire "device" or, to be more specific, the storage backend. The SSD controller firmware does. There's extra capacity on there that you will never have access to from nwipe.

Then give the SSD controller the Security Erase Enhanced command, which nwipe does NOT currently support, but can be done with HDPARM in the interim. Or perhaps swap the order to do Security Erase Enhanced first then wipe with random data.

My understanding is that wiping an SSD brings close to zero security benefit if you are worried about reliably deleting data, period. You can try to do a "Secure Erased" command, but that's an entirely different mechanism, where you delegate that work to the SSD controller and hope that it actually does delete that data. You'll notice that those commands return immediately and the controller (hopefully!) does that work in the background. But there's zero guarantee, from the OS's perspective, that it's doing anything at all.

I would be extremely careful at giving the impression that nwipe can be used on SSD drives. ANy such claim, in my opinion, is flawed and would only give a false sense of security.

The only reliable way to delete data from an SSD is to encrypt it before it ever hits the controller in the first place, and when you actually want to delete the data, you drop the encryption key. (And it's in fact how certain SSD drives implement secure erase internally, for what that's worth.)

But once you get to the nwipe stage, it's too late for that obviously... :)

[mdcato]

The mapping was described (if memory serves) as how one (unnamed) SSD controller conserves writes to optimize it’s lifetime. Don’t know how widespread it is. But that even one uses that method indicates writing a static pattern is ineffective. It must be random to overcome this optimization.

Yes, recognizing that there is not direct access to the SSD’s cells is what caused the previous discussion, and thus how nwipe should evolve.

And yes, delegating to the SSD controller via the Security Erase Enhanced command, while seemingly unverifiable, is supposed to change the SSD’s internal encryption key. However, that doesn’t answer whether the cells’ data was actually changed, and thus could you get the data forensically by directly accessing the NAND.

From: anarcat notifications@github.com Sent: Wednesday, February 12, 2020 12:56 To: martijnvanbrummelen/nwipe nwipe@noreply.github.com Cc: Mike Cato / Hays Technical Services mike@haystech.com; Comment comment@noreply.github.com Subject: Re: [martijnvanbrummelen/nwipe] SSD erase (#101)

To summarize the earlier discussion, SSD controllers recognize data patterns, such as all zeros, and map all blocks containing zeros to a single block that really contains zeros.

That's news to me: I wasn't aware of SSDs doing data replication.

Earlier data from those re-mapped blocks still exists and technically could be forensically read (but not with normal reads). One security method is to write random data to the entire device (low likelihood of repeated data in a block which would allow shared mapping), which nwipe currently supports.

The problem with that approach is that you don't have access to the entire "device" or, to be more specific, the storage backend. The SSD controller firmware does. There's extra capacity on there that you will never have access to from nwipe.

Then give the SSD controller the Security Erase Enhanced command, which nwipe does NOT currently support, but can be done with HDPARM in the interim. Or perhaps swap the order to do Security Erase Enhanced first then wipe with random data.

My understanding is that wiping an SSD brings close to zero security benefit if you are worried about reliably deleting data, period. You can try to do a "Secure Erased" command, but that's an entirely different mechanism, where you delegate that work to the SSD controller and hope that it actually does delete that data. You'll notice that those commands return immediately and the controller (hopefully!) does that work in the background. But there's zero guarantee, from the OS's perspective, that it's doing anything at all.

I would be extremely careful at giving the impression that nwipe can be used on SSD drives. ANy such claim, in my opinion, is flawed and would only give a false sense of security.

The only reliable way to delete data from an SSD is to encrypt it before it ever hits the controller in the first place, and when you actually want to delete the data, you drop the encryption key. (And it's in fact how certain SSD drives implement secure erase internally, for what that's worth.)

But once you get to the nwipe stage, it's too late for that obviously... :)

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/martijnvanbrummelen/nwipe/issues/101?email_source=notifications&email_token=ANGK2PRU3NNL5SYRA2BCLZ3RCRA4PA5CNFSM4IVNIMC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELR6ZBQ#issuecomment-585362566, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANGK2PR3CUCVEGRV65XYRLTRCRA4PANCNFSM4IVNIMCQ.

[IDerr]

I share this interesting doc : http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.187.3062&rep=rep1&type=pdf cited by the blancco patent for ssd erase https://patents.google.com/patent/US9286231B2/en

[PartialVolume]

@IDerr I haven't read the whole paper yet but it would appear From just reading the first couple of paragraphs to support @mdcato that wiping the ENTIRE drive using random data & using the drives internal erase command is the best option. However when it comes to much smaller blocks of data such as files, traditional methods are not suitable. Luckily for nwipe we wipe the entire drive. Would you trust a drives internal command to wipe a drive when you can't actually verify it ? I wouldn't.

[PartialVolume]

@IDerr BTW, very interesting paper, thanks for posting.

[PartialVolume]

WOW! I'd love to know what SSD they tested that resulted in the following when they used the internal firmware erase... A definite reason to wipe with nwipes traditional method using random data.

Drive B’s behavior is the most disturbing: it reported that sanitization was successful, but all the data remained intact. In fact, the filesystem was still mountable.

[PartialVolume]

Re blancco patent, I read through and couldn't find anything they 'actually' invented. Typical of so many nonsense filings with the US patent office. If anybody else can see anything that hasn't been done before then please point it out.

[PartialVolume]

Interesting.. https://www.prnewswire.com/news-releases/itrenew-prevails-in-patent-non-infringement-lawsuit-against-blancco-300421125.html

So they are claiming that they invented the method of using random data to wipe SSDs.

I'm pretty sure people have had the capability to wipe drives using random data since SSDs were invented. It's hardly an invention.

[mdcato]

Thanks @PartialVolume for finding and sharing. Things that I noticed in this (highlighting mine): "It is extremely unusual in such cases that a countersuit asserting actual infringement is not filed, which speaks clearly to the irresponsible allegations made by Blancco," said Aidin Aghamiri, vice president of corporate strategy at ITRenew. "By filing the voluntary dismissal, Blancco effectively retracts its unfounded allegations, but that's not what the company will lead the public to believe." And: "ITRenew does not subscribe to the hype of telling customers an uncompressible data overwrite will eliminate the risk of data breach, as it fails to do so for flash-based storage," said Mickelson, chief architect of Teraware's SSD sanitization scheme. "After careful review of the Blancco patent in question, it was immediately understood that the patented process does in fact allow for a degree of compression. It furthermore executes a firmware erasure command, which would not be necessary if the technique of overwriting uncompressed data were sufficient and absolute." From: PartialVolume notifications@github.com Sent: Saturday, February 15, 2020 13:19 To: martijnvanbrummelen/nwipe nwipe@noreply.github.com Cc: Mike Cato / Hays Technical Services mike@haystech.com; Mention mention@noreply.github.com Subject: Re: [martijnvanbrummelen/nwipe] SSD erase (#101)

Interesting.. https://www.prnewswire.com/news-releases/itrenew-prevails-in-patent-non-infringement-lawsuit-against-blancco-300421125.html

So they are claiming that they invented the method of using random data to wipe SSDs.

I'm pretty sure people have had the capability to wipe drives using random data since SSDs were invented. It's hardly an invention.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/martijnvanbrummelen/nwipe/issues/101?email_source=notifications&email_token=ANGK2PU6YCJHEZ7KA2N7Y7DRDA52TA5CNFSM4IVNIMC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL3U3JY#issuecomment-586632615, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANGK2PWYW6Q52QUDYVJG5YTRDA52TANCNFSM4IVNIMCQ.

[mdcato]

@iderr, finally catching up to read (most of) this article. Thanks for sharing.

From: iderr notifications@github.com Sent: Friday, February 14, 2020 16:15 To: martijnvanbrummelen/nwipe nwipe@noreply.github.com Cc: Mike Cato / Hays Technical Services mike@haystech.com; Comment comment@noreply.github.com Subject: Re: [martijnvanbrummelen/nwipe] SSD erase (#101)

I share this doc : http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.187.3062&rep=rep1&type=pdf cited by the blancco patent for ssd erase https://patents.google.com/patent/US9286231B2/en

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/martijnvanbrummelen/nwipe/issues/101?email_source=notifications&email_token=ANGK2PRZLE2K3H5H7JDCZEDRC4JVZA5CNFSM4IVNIMC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL2TBVI#issuecomment-586494165, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANGK2PU7NPMTQZPVXVZ23DTRC4JVZANCNFSM4IVNIMCQ.

[mdcato]

I concur on not putting sole trust in internal wipe command, even if you can trust some mfrs implementation, but not all. I perceive nwipe being used by those who ethically want to erase data, despite it taking longer than executing the internal wipe command. You know, the type of people that want to avoid this scenario (admittedly small & now old, yet I keep hearing people having recurrences of it when buying used computers & drives): https://arstechnica.com/information-technology/2015/06/how-best-buys-computer-wiping-error-turned-me-into-an-amateur-blackhat/

From: PartialVolume notifications@github.com Sent: Friday, February 14, 2020 17:47 To: martijnvanbrummelen/nwipe nwipe@noreply.github.com Cc: Mike Cato / Hays Technical Services mike@haystech.com; Mention mention@noreply.github.com Subject: Re: [martijnvanbrummelen/nwipe] SSD erase (#101)

@IDerrhttps://github.com/IDerr I haven't read the whole paper yet but it would appear From just reading the first couple of paragraphs to support @mdcatohttps://github.com/mdcato that wiping the ENTIRE drive using random data & using the drives internal erase command is the best option. However when it comes to much smaller blocks of data such as files, traditional methods are not suitable. Luckily for nwipe we wipe the entire drive. Would you trust a drives internal command to wipe a drive when you can't actually verify it ? I wouldn't.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/martijnvanbrummelen/nwipe/issues/101?email_source=notifications&email_token=ANGK2PTH5F64YRGP42SY5YDRC4UQXA5CNFSM4IVNIMC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL2Z5WQ#issuecomment-586522330, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANGK2PWEH2ABFFAT2GKDHQDRC4UQXANCNFSM4IVNIMCQ.

[IDerr]

I use both techniques together(secure erase + nwipe), not sure that it is the best thing to do but I don't want to put my trust only on the ssd controller...

[PartialVolume]

Just adding this for reference: Of specific importance is the date 17 December 2010 found in the PDF properties. Can this date be verified from a secondary source ?. The fact this date is prior to 2013 is what makes it important. Contains a description of wiping a whole drive with random data.

Specifically section 3.2.2 Overwrite techniques. Of particular note is the sentence "This suggests that for maximum effectiveness, SSD overwrite procedures should use random data.".

From a prior art point of view doesn't this cause serious problems for a certain companies patent claim regarding using random data to wipe SSDs. This paper was published 3 years prior to the US patent. Do we have dates for EU and British patents ?

And the actual document: Wei SSD's and erase methods.pdf

Amended 24Feb20: Secondary source:. Ref. conference paper 2011. secondary source

You tube video by Michael Wei, co author of the above paper, reference erase whole drive with random data 12:30 dated 2011

[hameau]

Late to this party, but a couple of points:

It's risky, imo, to rely on a ten year old research paper to inform operation of current SSDs. I'm sure that there has been a lot of development in controllers since that paper was written. Not that the research should be ignored, but don't presume that it necessarily applies exactly to modern SSDs.

A further effect of the ATA Secure Erase on SSDs is that cells throughout the whole usable space are reinitialised, so it is also useful for restoring as-new write performance. This is important for redeploying a used SSD, or if a combination of operating system, SSD detection and SSD controller has resulting in inadequate TRIM during normal use. (Example, to my knowledge, Windows 10 + Crucial BX500 + drive power saving enabled.) Reference, e.g., https://www.crucial.com/support/articles-faq-ssd/ssd-used-to-be-faster-but-has-slowed-down.