Update NWipe's Wiping Methods

So, the documentation on your site says NWipe uses the following wiping techniques to wipe HDDs. However, all of them are from the 1990s, before there was SSD & Flash Memory. So I think it's time to update and use more advanced and optimized shredding techniques.

NWipe uses the following methods:

Quick erase - Fills the device with zeros, one round only.
RCMP TSSIT OPS-II - Royal Candian Mounted Police Technical Security Standard, OPS-II
DoD Short - The American Department of Defense 5220.22-M short 3 pass wipe. 1,2,& 7.
DoD 5220.22M - The American Department of Defense 5220.22-M full 7 pass wipe. 1-7
Gutmann Wipe - Peter Gutmann's method. (Secure Deletion of Data from Magnetic and Solid-State Memory)
PRNG Stream - Fills the device with a stream from the PRNG.
Verify only - This method only reads the device and checks that it is all zero.
HMG IS5 enhanced - Secure Sanitisation of Protectively Marked Information or Sensitive Information

Unfortunately, most of these are outdated and one (the Gutmann method) is a flat out waste of time.

The RCMP level 3 wipe has been superseded by CSEC ITSG-06.

What Does CSEC ITSG-06 Do? All data sanitization methods are similar, but what sets them apart from each other are the small details. For example, Write Zero is a method that only uses one pass of zeros. Gutmann overwrites the storage device with random characters, possibly up to dozens of times. However, the CSEC ITSG-06 data sanitization method is a little different in that it uses a combination of zeros and random characters, plus ones. It's usually implemented in the following way:

Pass 1: Writes a one or zero Pass 2: Writes the complement of the previously written character (e.g. one if Pass 1 was zero) Pass 3: Writes a random character and verifies the write

So, it's just a 3 wipe algorithm, but it's much more advanced. Here is what I believe is the white paper on it: IT Media Sanitization (ITSP.40.006)

Next is DoD 5220.22M ECE (7 wipes) and DoD 5220.22M aka DoD Short (3 wipes). These also have been superseded by standards from NIST.

The DoD 5220.22M was developed during a time when we were still using tape backup drives, optical memory, EEPROM, cathode ray tubes and the like. You can read an interesting history of it on White Canyon's DoD 5220.22-M Relevancy & The Evolution Of Wipe Standards. Apparently, the DoD is working on a 5200.48 standard, which they leaked the white paper on here: DoD Instruction 5200.48 "Controlled Unclassified Information CUI (PDF). However, it may be in alpha/beta stages and not fully ready for prime time. That's why they let NIST, a very capable agency (they picked AES aka Rijndael as the standard encryption scheme, as opposed to a host of others, and it has proven to be a very fast and versatile & fast encryption standard. Advanced Encryption Standard Process & Competition

NIST Special Publication 800-88 Revision 1 Guidelines for Media Sanitization: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-88r1.pdf

These are called NIST 800-88 Clear and NIST 800-88 Purge. Both of these are described in the above document.

Finally, we have the Gutmann Wipe, which it's inventor, Peter Gutmann, has said it is outdated and unnecessary.

Most of the patterns in the Gutmann method were designed for older MFM/RLL encoded disks. Gutmann himself has noted that more modern drives no longer use these older encoding techniques, making parts of the method irrelevant. He said "In the time since this paper was published, some people have treated the 35-pass overwrite technique described in it more as a kind of voodoo incantation to banish evil spirits than the result of a technical analysis of drive encoding techniques"

Source

I searched and searched for an updated version of HMG IS5 enhanced aka "HMG Infosec Standard 5", which is what the British (apparently still even currently) use to scrub sensitive data. Either that, or they are keeping their secure wipe methods very close to their chest, which is what intelligence agencies usually do anyway. There is apparently a HMG IS6 but it's still either in development or classified by the UK government. In light of that, IS5 should be fine, as I found many currenct documentats saying something along the lines of the "The British Agent will use Infosec Standard 5 to wipe the file"

I searched and searched for an updated version of HMG IS5 enhanced aka "HMG Infosec Standard 5", which is what the British (apparently still even currently) use to scrub sensitive data. Either that, or they are keeping their secure wipe methods very close to their chest, which is what intelligence agencies usually do anyway. There is apparently a HMG IS6, and I was approached by two gentlemen in suites and manhandled a little bit about finding this document. But, they eventually gave up and left. The documents read like they're only at the green or blue paper stage, and not full white paper stage.

Use of HMG Cryptography Policy left. The documents read like they're only at the green or blue paper stage, and not full white paper stage.

Use of HMG Cryptography Policy

Quick Erase would be good for SSD drives, as would PRNG Stream (although it doesn't specify the levels it does that at) they have a finite number of reads/writes they can do before they are completely destroyed. It's quit a lot, but if you're wiping your SSD HDD with the Gutmann method multiple times, running CCleaner with different level wipes, wiping free space, defragging often, you're more likely to encounter SSD failure (especially on how cheap or expensive the brand of the drive is).

As far as the two PSRNGs, it appears Mersenne Twister is not out of date, as much as there are better PSRNGs available, not to mention they were created in the 90s.

Letting Go Of The Mersenne Twister?

The author recommends instead, using CRNGs or Crypto Random Number Generators like [url=https://docs.microsoft.com/en-us/windows/win32/api/wincrypt/nf-wincrypt-cryptgenrandom]CryptGenRandom[/url] which was developed by Microsoft, but is now deprecated by Windows other CRNGs built into the OS, you couldn't use an API call from Windows to generate the random numbers in a bootable CD/USB environment.

CryptGenRandom - Wikipedia - Not to mention, the source code to wincrypt.h is available on the Wine Project's GitHub - WinCrypt.H. As well as one of MIcrosoft's alternatives, BCrypt.H, another CRNG. Bcrypt was developed using Blowfish, Blowfish is a cryptographic algorithm developed by the famous Bruce Schneier, who also went on to make it's successor, Twofish.

Blowfish General Designers Bruce Schneier First published 1993 Successors Twofish Cipher detail Key sizes 32–448 bits Block sizes 64 bits Structure Feistel network Rounds 16 Best public cryptanalysis Four rounds of Blowfish are susceptible to a second-order differential attack (Rijmen, 1997);[1] for a class of weak keys, 14 rounds of Blowfish can be distinguished from a pseudorandom permutation (Vaudenay, 1996).

So, Bcrypt might not be the best option, but thee are two alternative Crypto Random Number Generators w/publicly available source codes. Lets see what else we can find...

There's the AES CRT DRBG (Deterministic Random Bit Generator) Then there's the [Linear_Feedback_Shift_Register](https://en.wikipedia.org/wiki/Linear-feedback shift register) made with NIST's test suites and has some actual studies posted on it. Linear-feedback shift register source code . We have NIST's SP 800-90A Rev. 1, which has a bit of a negative history because one of its PRNGs had a backdoor installed into it by the NSA. There was the _Dual_ECDRBG which was based on an elliptical curve model that had some sort of backdoor from the NSA or RSA. But, NIST patched it out and opened it back up for public comment. "NIST Released Special Publication (SP) 800-90A Revision 1: Recommendation for Random Number Generation Using Deterministic Random Bit Generators" (PDF). So, you would be safe to use this one now, although I don't know why they didn't rename it Revision 2 or change any of the code numbers for the project, so it was so tainted by then. So, for individuals who still remember that, it might be a problem.

Finally, there is a very promising one called ChaCha20 that is used by NetBSD, OpenBSD, FreeBSD, and even Google when they were using their SPDY protocol for faster web traffic.

ChaCha family of ciphers, which aim to increase the diffusion per round while achieving the same or slightly better performance. The Aumasson et al. paper also attacks ChaCha, achieving one round fewer: for 256 bits ChaCha6 with complexity 2139 and ChaCha7 with complexity 2248. 128 bits ChaCha6 within 2107, but claims that the attack fails to break 128 bits ChaCha7. ChaCha's initial state is similar to the initial state of Salsa20, however there are some differences. ChaCha's initial state includes a 128-bit constant, a 256-bit key, a 32-bit counter, and a 96-bit nonce, arranged as a 4×4 matrix of 32-bit words. ChaCha also re-arranges some of the words in the initial state.

ChaCha is based off another algorithm called Salsa20, which uses add–rotate–XOR ARX to generate PSRNs. The main thing is that it's fast, which is why Google choose it to help speed up web browsing.

Both ciphers are built on a pseudorandom function based on add-rotate-XOR (ARX) operations — 32-bit addition, bitwise addition (XOR) and rotation operations. The core function maps a 256-bit key, a 64-bit nonce, and a 64-bit counter to a 512-bit block of the key stream (a Salsa version with a 128-bit key also exists). This gives Salsa20 and ChaCha the unusual advantage that the user can efficiently seek to any position in the key stream in constant time. Salsa20 offers speeds of around 4–14 cycles per byte in software on modern x86 processors, and reasonable hardware performance.

And since there are literally hundreds of thousands of PRNGs out there, I'm going to stop there and take a look at ISAAC. Which looks like your standard run of the mill RNGs. Although, it is also a bit old and uses RC4, which has shown to be insecure. But that's really only if you were going to use it as algorithm to connect to a VPN or something, not really pretty much instantly generating random numbers - although security researchers have even abandoned it as a RNG and switched to other ones.

RC4-based random number generators

Several attacks on RC4 are able to distinguish its output from a random sequence. Source

Ain't that a bitch?

So, I feel I've posted enough suggestions in this thread. I didn't want to make a new thread for a suggestable replacement for each algorithm (unless you really want me to) because I felt it would fill up the Issues forum with TONs of suggestions.

Also, to the author(s) of this project, I think it's good and a good idea since DBAN stopped development in 2015 and has become commercial software, we need a HDD wiping tool that's good, free and open source. But, as I read through the documentation I saw the same algorithms for wiping that I've seen everywhere in the last ~15 years. So, I thought there has GOT to be some newer secure wiping algorithms out there that are drop in replacements for these old ones. And there are several good ones! NIST 800-88 Clear and NIST 800-88 Purge are much more secure than the current DoD 5220.22 one, and it works on SSDs, which DoD 5220.22 was never intended to, as it was invented before the idea of SSD and Flash memory was just a teardrop in it's father's eye. CSEC ITSG-06 also sounds very promising. I also wanted to mention you did an excellent job of porting this project to multiple operating systems and architectures.

The thing is you don't need a dozen different secure wiping algorithms for the software to work good. You only need one to three good ones. People can pick for themselves. As far as RNGs, I think I would use ChaCha20 for one. Btw, I forgot to link the source code for it, which is here:

Simple C99 ChaCha20 Implementation

Then, I might give the user a second option to use a [Cryptographically-secure pseudorandom number generator](Cryptographically-secure pseudorandom number generator), of which there are many, which will slow down the overall process, but you know you'll be getting more secure entropy at the parts being wiped. Since I believe this project is being written in Python, the Python library has it's on CSRNG called Secrets although it might not be ideal for what you're using. If I'm correct that you're writing this in Python, I can dig around and try to find Python implementations of those new cryptographic algorithms to use.

And just searching Google, I found a TON of people making their own Crypto RNGs just using the calls in Python's OS.

Anyway, that was a long write up for me, but if you want me to break them into individual posts for each algorithm I can do so if you are actually interested in trying them.

I hope that might give you a few ideas and it wasn't too boring or tedious to read. Thanks and keep up the good work! pogue

So, after I finished this, I went through Google and searched for the source codes of the algorithms I posted, specifically the Canadian one and the one by NIST. I can't find them ANYWHERE. I see companies claiming they have tools that use these algorithms, but I don't know if they had to license the code or are just lying or the source code is so well hidden you have to go down a deep maze of FTP servers or a Rabbit hole of hidden source code directories to find them. But, I will keep at it!

I finally ended up emailing NIST and the Canadian CyberSecurity Centre (ugh, should be Center) and asking them what the deal was. I'll post back when I get a response and keep looking. Luckily the RNG generators have a lot more readily available source codes for them in multiple languages.

Thanks for the effort you have spent on this issue, however, you don't need to waste your time looking for source code for wipe algorithms, the required method or procedure is all that's needed. Every implementation of a wipe method will have different source code anyway. The wipe methods are generally so simple anybody that is proficient in what ever computer language they are using should be able to code the method without needing an example in code. Having said that, it's different for PRNG's hence why you can find lots of code examples of prng's, so that if you code the algorithm slightly differently you can compare your implementation results against the results produced by the author's code.

I hope to add more wipe methods and PRNGs in time and would certainly review pull requests for both. HMG IS5 was the most recent PR that was submitted as a PR to nwipe.

Having said that, I totally agree with all you comments especially your comment about the Gutmann method being a waste of time. I would probably go further and say that there is a opinion held by quite a few people that anything more than a zero fill or prng fill followed by a zero wipe is all you need on a modern spinning platter disc. There is a paper from 2009 https://web.archive.org/web/20131208184307/http://www.h-online.com/newsticker/news/item/Secure-deletion-a-single-overwrite-will-do-it-739699.html that says much the same.

When you think about it, if it was possible to completely recover a zero wiped disc, i.e. effectively we're talking about that disc being able to store twice as much data. If I was the manufacturer I'd be leveraging that feature in order to double the capacity of my drives and beat the competition. So personally for me on a spinning platter disc a prng stream followed by a verification followed by a zero plus verification is all I would do, and that's probably over kill. SSD's are a completely different matter and if wiping using nwipe rather than the drives internal ATA secure erase command, I would always do a random pass plus verification. In this way almost every sector, if not all sectors would be filled with a unique set of data different to every other sector on the drive. There is plenty of comments on this subject if you look through the issues section on nwipe. That's not to say the wipe methods you mentioned won't be implemented but it all boils down to how much time I have to spend on this, hence why I'm happy to review pull requests for new implementations. This should probably be over on the nwipe page as that's where the changes will be made.

Here's a few nwipe issues related to wipe methods and prngs.

https://github.com/martijnvanbrummelen/nwipe/issues/160

https://github.com/martijnvanbrummelen/nwipe/issues/289

https://github.com/martijnvanbrummelen/nwipe/issues/282

https://github.com/martijnvanbrummelen/nwipe/issues/176

https://github.com/martijnvanbrummelen/nwipe/issues/331

Thanks for the effort you have spent on this issue, however, you don't need to waste your time looking for source code for wipe algorithms, the required method or procedure is all that's needed. Every implementation of a wipe method will have different source code anyway. The wipe methods are generally so simple anybody that is proficient in what ever computer language they are using should be able to code the method without needing an example in code. Having said that, it's different for PRNG's hence why you can find lots of code examples of prng's, so that if you code the algorithm slightly differently you can compare your implementation results against the results produced by the author's code.

Bro, I'm a complete idiot. I was reading all about Dban and all these other "secure wiping" softwares, but they're total frauds. There is NO SUCH ALGORITHM AS CSEC ITSG-06 OR NIST 800-88. I got so confused, because I kept seeing this software saying they had these algorithms as deletion methods, so I thought, surely there's a source code out there. I even emailed NIST and the Canadian Cybersecurity Center asking for the source code. So after like hours of hunting for them, I went on Reddit and they explained it to me.

These are just standardization guides, e.g. manuals to tell HDD manufacturers that include onboard encryption with their HDD the proper steps to do a secure deletion, and usually the only secure deletion they do is delete the main key, while leaving all the data on the HDD.

Check out my Reddit thread: What's the story behind these phantom secure wiping algorithms - NIST 800-88 Clear, NIST 800-88 Purge & CSEC ITSG-06?

The pros under /r/NISTControls had to explain it to me.

One thing I'm still confused is about wiping SSD drives. DBAN's commercial version claims it can securely wipe SSDs, but according to some studies they linked me, they aren't really sure. There might be some residual data left over on the SSD after you wipe it or not. The only way you're going to get rid of an SSD is with a hammer. All these secure wipe methods only work for ATA platter HDDs.

Having said that, I totally agree with all you comments especially your comment about the Gutmann method being a waste of time. I would probably go further and say that there is a opinion held by quite a few people that anything more than a zero fill or prng fill followed by a zero wipe is all you need on a modern spinning platter disc. There is a paper from 2009 https://web.archive.org/web/20131208184307/http://www.h-online.com/newsticker/news/item/Secure-deletion-a-single-overwrite-will-do-it-739699.html that says much the same.

It really depends what kind of forensic software law enforcement might be working with to try and recover data off a HDD. LIke if you're dealing with the local police, the FBI, the CIA, or Mossad. Of course, they could always use the $5 hammer trick.

Encryption vs Reality

I've seen conflicting data. But, one other thing to look at besides that study you looked at is Steve Gibson's SpinRite software (PM, and I'll get you a link). It does the opposite of wiping, which is data recovery, and he has very advanced algorithms that will easily pull data off bad sectors of platter HDDs and move them to another sector, but sometimes it can take 12 hours, 24, a week, etc. It's all written in assembly language, and he claims it works for SSD, but I don't see how. But, unless the boot record is totally corrupt (Hey, maybe there's an idea - go back and look at some of the old school viruses that actually attacked files or the HDD and see how they worked to destroy a HDD. I remember reading about a virus that would overwrite your MBR so thoroughly or with all these weird characters over and over that it would make the HDD unreadable). You could also do something revolutionary, like take some code from VeraCrypt and encrypt different parts of the HDD and leave some readable, then wipe the readable parts. Almost like Ransomwear.

So, I think there is some validity in doing more than a zero sector wipe, but more than 7 and you're really just wasting your time. But even Steve Gibson's GRC isn't going to be able to rescue a zero wiped drive. But Drive Savers might. They do the kind of stuff you would expect somebody like Mossad or some super intelligence agency would do. They take everything into a clean room and examine the platter with electron microscopes and shit.

I'll have to take a look at those Nwipe methods later. I wiped myself out spending basically all night trying to hunt down this information for what I thought was going to be an easy time of finding the algorithm or source codes and plugging them into Nwipe.

I do wonder if there are any open source/freeware projects out there that are working on any type of new and improved or fancy secure wiping methods or what to do about the SSD problem. Surely, there is someone out there that knows the answer to that.

Anyway man, check out that Reddit thread. I completely screwed up and didn't know what I was telling you was completely incorrect information (with the exception about the RNGs and Crypto RNGs).

Thanks for getting back to me. I heard of your software through a friend of yours on Dread (dunno if you know what that is and I forgot his username), so I thought I'd stop by to help, but ended up learning a big lesson myself about the gullibility of advertising.

pogue

This is a recurring topic and as I mentioned several years ago @ https://github.com/NHellFire/dban/issues/18#issuecomment-297483125 your best bet for wiping flash media is ATA Secure Erase. This is best implemented in my opinion with Parted Magic ( https://partedmagic.com/secure-erase ) where Patrick has created a workflow (unlock device by entering standby, then wipe) and GUI around what the instructions @ https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase say. It would be a really nice addition to ShredOS to get this implemented, but I feel PartialVolume has already enough on his plate being the only developer of ShredOS and semi-official maintainer of nwipe.

Secure erase is next on the list. :-)

I've been thinking of ways to be able to wipe or try the SSD drive. I've been recommending to people who need real opsec this device called USBKill. You stick it in a random USB port and it will fry the circuitry on the whole computer. But I was reading the documentation and saw the following:

When testing computer hardware, is data affected? When tested on computers, the device is not designed or intended to erase data. However, depending on the hardware >configuration (SSD vs Platter HDD), the drive controllers may be damaged to the point that data retrieval is impractical.

But, BlackBlaze (an online cloud storage company) recommends encrypting the SSD with a random key and then formatting it.

Maybe you could borrow some code from VeryCrypt to encrypt the HDD, use a PSNG to generate the password to encrypt the drive, and then run a full format on it.

Nothing really new in what BackBlaze suggested 4 years ago in this article. Changing the encryption key (for SED SSDs) is what the manufacturer's SecureErase function is supposed to provide and this has been used in the industry for a number of years. Some BIOS even have the Wiping/SecureErase option build-in nowadays. Now how reliable SecureErase works is up for debate for a number of years. I've read at least one research paper where I can clearly remember it suggested that storage manufacter's don't have any Quality Control and their SecureErase is just a bunch of shoddy code that even leaves recoverable data on the device. But again this is IT and the industry is constantly evolving so code and practices get better over time (hopefully).

Secure Erase sounds good, is it proprietary software or is it open source? All I could find from a Google search were vendors of hardware discussing it's benefits. Is there anyway for us to test that Secure Erase has worked on an SSD drive with some Forensic Software or something? I'm sure I could find some through.... certain methods. However, I can't volunteer my SSD drive to test it on.

Also, in that Backblaze article, they mention something I had forgotten are those "integrated SSD drives"

Integrated SSDs Some computers—most notably recent model Macs from Apple—include factory-installed SSDs that are integrated directly onto the motherboard and may not be removable. Sure, you might be able to run the whole main logic board through a shredder, but that seems…well, excessive.

In those cases, physically destroying the SSD becomes a lot harder. Which makes it doubly important to have another way to make sure your data is safe, like encrypting the drive, for example.

I've seen a lot of PC vendors do this where they'll take a tiny SSD drive and they use it to store the bootloader or pagefile or something to give the PC a bit more speed when booting or using a lot of RAM. Is there anyway to access these and securely wipe them? Are they given a hidden partition or something?

I also came across this document authored by the Center for Memory and Recording Research from UC San Diego titled "Tutorial on Disk Drive Data Sanitization". I've uploaded it to my Mega drive if you want to take a look (it's a PDF). They have a free utility for Secure Erase on their website, so I guess it must be public domain.

However, they mention some interesting points I thought were worth sharing, for example some problems with the DoD 5220 algorithm:

It is difficult for external software to reliably sanitize user data stored on a hard disk drive. Many commercial software packages are available using variations of DoD 5220, making as many as 35 overwrite passes. But in today’s drives, multiple overwrites are no more effective than a single overwrite. Off-track overwrites could be effective in some drives, but there is no such drive external command for a software utility to move heads offtrack. And even three overwrites can take more than a day to erase a large capacity hard disk drive. In busy IT facilities, such time is often not available and IT personnel are likely to take short cuts.

DoD 5220 overwriting has other vulnerabilities, such as erasing only to a drive’s Maximum Address, which can be set lower than its native capacity; not erasing reallocated (error) blocks; or miss extra partitions. External overwrites cannot access the reallocated sectors on most drives, and any data once recorded is left on these sectors. These sectors could conceivably be recovered and decoded by exotic forensics. While enterprise-class drives and drive systems (SCSI/FC/SAS/iSCSI) allow software commands to test all the user blocks for write and read ability, mass market drives (PATA/SATA) cannot read, write, or detect reassigned blocks since they have no logical block address for a user to access.

They then go on to discuss Secure Erase:

The Secure Erase (SE) command was added to the open ANSI standards that control disk drives, at the request of CMRR at UCSD. The ANSI T13.org committee oversees the ATA interface specification (also called IDE) and the ANSI T10.org committee governs the SCSI interface specification. Secure erase is built into the hard disk drive itself and thus is far less susceptible to malicious software attack than external software utilities. The SE command is implemented in all ATA interface drives manufactured after 2001 (drives with capacities greater than 15 GB), according to testing by CMRR. A standardized internal secure erase command also exists for SCSI drives, but is optional and not currently implemented in SCSI drives tested. Secure erase is a positive easy-to-use data destroy command, amounting to “electronic data shredding.” Executing the command causes a drive to internally completely erase all possible user data record areas by overwriting, including g-list records that could contain readable data in reallocated disk sectors (sectors that the drive no longer uses because they have hard errors).

SE is a simple addition to the existing “format drive” command present in computer operating systems and storage system software, and adds no cost to hard disk drives. Because the Secure Erase command is carried out within hard disk drives, no additional software is required either. Secure erase does a single on-track erasure of the data on the disk drive. The U.S. National Security Agency published an Information Assurance Approval of single pass overwrite, after technical testing at CMRR showed that multiple on-track overwrite passes gave no additional erasure. Secure erase has been approved by the U.S. National Institute for Standards and Technology (NIST), Computer Security Resource Center 7 . NIST document 800-88 approves SE at a higher security level than external software block overwrite utilities like as Norton Government Wipe, and it meets the legal requirements of HIPAA, PIPEDA, GLBA, and Sarbanes-Oxley.

There's much more and I encourage you to give it a quick read. It's only about 12 pages long.

I came across another paper published by Department of Computer Science and Engineering, University of California, San Diego and the same Center for Magnetic Recording and Research. The abstract itself is very telling.

Abstract Reliably erasing data from storage media (sanitizing the media) is a critical component of secure data management. While sanitizing entire disks and individual files is well-understood for hard drives, flash-based solid state disks have a very different internal architecture, so it is unclear whether hard drive techniques will work for SSDs as well. We empirically evaluate the effectiveness of hard drive-oriented techniques and of the SSDs’ built-in sanitization commands by extracting raw data from the SSD’s flash chips after applying these techniques and commands. Our results lead to three conclusions: First, built-in commands are effective, but manufacturers sometimes implement them incorrectly. Second, overwriting the entire visible address space of an SSD twice is usually, but not always, sufficient to sanitize the drive. Third, none of the existing hard drive-oriented techniques for individual file sanitization are effective on SSDs. This third conclusion leads us to develop flash translation layer extensions that exploit the details of flash memory’s behavior to efficiently support file sanitization. Overall, we find that reliable SSD sanitization requires built-in, verifiable sanitize operations.

Flash-based solid-state drives (SSDs) differ from hard drives in both the technology they use to store data (flash chips vs. magnetic disks) and the algorithms they use to manage and access that data. SSDs maintain a layer of indirection between the logical block addresses that computer systems use to access data and the raw flash addresses that identify physical storage. The layer of indirection enhances SSD performance and reliability by hiding flash memory’s idiosyncratic interface and managing its limited lifetime, but it can also produce copies of the data that are invisible to the user but that a sophisticated attacker can recover.

I'll try to take some time out later to read the whole study and see what their solution to this problem is, but, this study is fairly long.

Reliably Erasing Data From Flash-Based Solid State Drives

Secure Erase sounds good, is it proprietary software or is it open source?

I think you misunderstand, when people talk about 'secure erase' they are talking about a method of erasure that relies upon the drives internal firmware to do the actual erase. Some proprietary or open source software may support secure erase but it simply issues a ATA command to the drive to tell it to erase itself. ShredOS/nwipe currently don't issue a secure erase command, they write to every available sector of the drive and verify what was written is correct. It's unfortunate that the term secure erase, i.e. having the drives internal firmware do the wipe, somehow implies that other more traditional methods are not secure. There have been reports that some drive firmware has failed to wipe the drive! I suppose it boils down to whether you trust the drives firmware to do the wipe or whether you want the reassurance of a unique random number generated stream being written to the drive and verified by ShredOS/nwipe. If the drive was being sold on or redeployed outside of the organisation and I was particularly concerned about that data, I would do a secure erase, then a PRNG stream with verification and no blanking.

So 'secure erase' should really be called 'drive erasure by drive's own firmware'. I guess maybe marketing thought that was just too precise or long a description. 😁

Supposedly 'secure erase' also removes data from places that a sector by sector erasure cannot do... or does it?. Just because they say it does, doesn't mean it does under all situations. How about a nice little backdoor in the firmware that when the drive is told to zero or one the drive it simply returns zeros for sectors it supposedly overwrote. I'm sure the guys over at GCHQ or NSA have already been working on that one. But then how would you know a drive had really been zeroed? By writing something over the entire disc that it has to have recorded otherwise it can't return the appropriate data for every single sector upon being verified by ShredOS/nwipe.. That's where a unique PRNG stream comes in...

Makes you wonder doesn't it?. Why would some organisations be saying 'secure erase' is the better option. Better for who exactly. Maybe it is in terms of speed and extending the life of the device but unless I was the engineer that wrote that firmware I would always want verification.

Of course, I've let my paranoia level rise to 10, I believe government departments where security is paramount they rely upon total destruction of the device as being the only 100% sure way. That's a paranoia level of 1000. An unfortunate waste of good hardware but I guess they have their reasons.

I also came across this document authored by the Center for Memory and Recording Research from UC San Diego titled "Tutorial on Disk Drive Data Sanitization". I've uploaded it to my Mega drive if you want to take a look (it's a PDF). They have a free utility for Secure Erase on their website, so I guess it must be public domain.

pogue, that utility wasn't available anymore the last time I checked (half a decade ago). Also see https://github.com/NHellFire/dban/issues/18

@PartialVolume Check what the study from the Center for Memory and Recording Research from UC San Diego says in my post above Secure Erase. I didn't dig much into it, but it only does a 1 pass wipe, would that be enough, even though it's performed by a command issued by the mfg of the HDD? Has there been any forensic studies on Secure Wiped drive to see if it's forensically wiped?

@pogue I read that study some time ago. I'm not sure what you mean by 'it only does a one pass'. Their study shows that no single method can necessarily be trusted on its own. However it did actually show that on some drives it took two traditional random wipes to remove any sign of the fingerprint which to be honest, I thought was impressive. Even the ATA secure wipe can't necessarily be trusted from their evidence.

So much depends on the drive/make/model/firmware rev. hence why I propose for nwipe a secure wipe followed by a verification, then a random stream, then random stream verification, followed by an optional ones write and verification for the highest security level. For medium security a ATA secure wipe then verification. Based on their findings maybe a single ATA secure erase and verification should be classed as low security. So much depends on the make/model of the drive and how they implement the erasure in firmware. That's why I use a wipe method that incorporates both a drive initiated wipe and a traditional random stream wipe that should hopefully cover all possibilities.

It also depends on how valuable your data is and what lengths others might go to recover it. From what I've seen when it comes to military grade wiping nothing less than total destruction of the drive, i.e physically shredded having never left site is the only option. That includes engineers entering a military operation with a brand new spare drive. No hard drives leave site and if you brought it in you won't be taking it out, whether you used it or not !

However unless you were actually disposing of the drive, I would just do a secure wipe & verification.

The issues with secure wipe, as is pointed out in the article, is how can you verify the disc has really been wiped, unless you take out all the chips and read them. I think one of the points they were also making was that the drive itself should have some verification function. However they didn't really go into whether the drives could be compromised by malicious firmware, which invalidates everything about 'secure erase' being secure.

https://www.kaspersky.com/blog/equation-hdd-malware/7623/

It's probably quite rare ... until it isn't.

Just one other comment.

Kaspersky Lab (/kæˈspɜːrski/; Russian: Лаборатория Касперского, tr. Laboratoriya Kasperskogo) is a Russian multinational cybersecurity and anti-virus provider headquartered in Moscow, Russia[1] and operated by a holding company in the United Kingdom.

Just saying... From the article, if criminals don't have the millions required to develop malicious drive firmware, I wonder who does? :-).

@pogue I read that study some time ago. I'm not sure what you mean by 'it only does a one pass'. Their study shows that no single method can necessarily be trusted on its own. However it did actually show that on some drives it took two traditional random wipes to remove any sign of the fingerprint which to be honest, I thought was impressive. Even the ATA secure wipe can't necessarily be trusted from their evidence.

I'm with on you the Secure Erase. It seems like a solid solution. However, I would like to get some Forensic Software off somewhere coughBittorrent*cough and test it and see if there is anything left on the drive afterwards.

So much depends on the drive/make/model/firmware rev. hence why I propose for nwipe a secure wipe followed by a verification, then a random stream, then random stream verification, followed by an optional ones write and verification for the highest security level. For medium security a ATA secure wipe then verification. Based on their findings maybe a single ATA secure erase and verification should be classed as low security. So much depends on the make/model of the drive and how they implement the erasure in firmware. That's why I use a wipe method that incorporates both a drive initiated wipe and a traditional random stream wipe that should hopefully cover all possibilities.

Sorry, what is a random stream and random stream verification?

It also depends on how valuable your data is and what lengths others might go to recover it. From what I've seen when it comes to military grade wiping nothing less than total destruction of the drive, i.e physically shredded having never left site is the only option. That includes engineers entering a military operation with a brand new spare drive. No hard drives leave site and if you brought it in you won't be taking it out, whether you used it or not !

Yeah, from what I've read about Secure Wipe, it all depends on if the HDD mfg implemented it properly to do the job it was meant to do. However, with SSDs, it is incredibly hard to physically destroy them because, as mentioned, they are often now soldered onto laptops and motherboards made by vendors and to remove them you basically have to destroy the computer to get them out. That's why I still prefer the method I listed below. I've seen it recommended MANY MANY times, more than I can count. Just encrypt the drive with a PSRNG for the password string, encrypt it, then just format it. That is good enough for SSDs.

ATAs we've known how to wipe for a long time now.

However unless you were actually disposing of the drive, I would just do a secure wipe & verification.

The issues with secure wipe, as is pointed out in the article, is how can you verify the disc has really been wiped, unless you take out all the chips and read them. I think one of the points they were also making was that the drive itself should have some verification function. However they didn't really go into whether the drives could be compromised by malicious firmware, which invalidates everything about 'secure erase' being secure.

You don't necessarily have to take out the chips to read them, they should be all located on the HDD itself. And some good forensic software would read it. I would be happy if someone volunteered their SSD to be wiped using this then they could mail me the drive and I could plug it in and read it with some high Quality Forensic software and go through it and see if there is anything but random data left.

Not sure what happened to my other post...

I still like this idea as an alternative. I've seen it recommended SO MANY times to encrypt an SSD with a random key then format it.

[Secure Wiping of SSD Drives using Encryption & Formatting](https://bitbin.it/aJOhwukR/)

Sorry I had to post this on bitbin, I keep getting the error from Github "You can't comment at this time."

Also, I finally did find NIST document 800-88 According to that secure wiping methods document "approves SE at a higher security level than external software block overwrite utilities like as Norton Government Wipe, and it meets the legal requirements of HIPAA, PIPEDA, GLBA, and Sarbanes-Oxley.

However, I can find almost NOTHING about this so called "Norton Government Wipe". I will email Norton and ask them about this and if it is available for the general public or if it's only sold to LE/gov agencies.

But, BlackBlaze (an online cloud storage company) recommends encrypting the SSD with a random key and then formatting it.

Maybe you could borrow some code from VeryCrypt to encrypt the HDD, use a PSNG to generate the password to encrypt the drive, and then run a full format on it.

Hello,

This I think this here is the future of dealing with SSD's, it just makes sense. I remember I found a similar guide here: https://its.fsu.edu/sites/g/files/upcbnu1011/files/ISPO%20Pages/Sanitize%20Drive.pdf which looks like it was put together by the IT department of Florida State University. They give instructions of encrypting a drive with Bitlocker in Windows, formatting, then encrypting again to encrypt the first encryption key and formatting again.

My question would be if even using something like Bitlocker made by the great Microsoft, does it get into all the nooks and crannies of SSD's where possible data may be hiding or inaccessible? You would hope so. But I guess if you really needed something you can securely wipe, you wouldn't use SSD's in the first place.

Anyways, I guess all that I'm saying is... having a 2-pass encryption wipe method in nwipe would be really cool, and I would be interested in seeing these methods tested forensically.

This is a bit of a late response, but what an interesting thread topic here. I have a hard time trying to understand with Governent security specialists, cyber forenstic specialists, and the near same hdd manufacturers for the last 30 years why we do not have a unified answer to wiping, erasing disks. I agree with a comment above, we need 1-3 methods that should wipe/erase a drive in a secure, as fast as possible way. I agree heavily on the topic of needing updated wipe methods. We have basically the same regurgatated info on the same methods from decades ago. Nearly every company does not know either, so they implement booting from USB most of the time or CD and running Killdisk. I'm fond of killdisk actually... I do IT Support and it is a massive dread to do Wipes/erase... create the USB image, boot from USB, wait for hours for it to complete, copy the certificate to a sharepoint site, etc... and then there are the PC's that do not boot or power on and we have to disassemble the pc/laptop, and now we are also faced with Tablets, iPADs, Mobile phones, etc.. and seemingly no answer from any vendor, hdd manfacturer, government entity on what the best and quickest options are. I won't do more than a 3 pass wipe and verify. If it needs more than that, it is cheaper and more cost effective to just pull and physically destroy the drive, although we try to keep as many working units as possible for best prices from recyclers, which is a whole other topic. Try finding a recycler for Zebra label printers, forklift PC's and other oddities in IT Equipment.

I do guess more PC manfufacturers will make use of UEFI, and Secure-Erase/ATA type utilities, but they need to offer proof and verification options. I have to use 3rd party utilities that have a print out of proof of wipe/erase. There are risk management, Legal, and HR related associated issues that requires a printer copy of proof drives have been securely wiped and all data attempted best effort to erase/wipe. On the topic of "Encrypt, format, encrypt format...I have read some other articles that say this would work. Also, it seems like this would be one of the fast methods possibly as well, but I have not seen an "Authoritative Answer" to this topic anywhere yet. Anywho, thanks for the interesting and relevant discussion. :)

You summarized pretty well the recent (decade) issues in Data Destruction. I share and agree on all the points you brought up. Also what history has shown us is that you can't even rely on UEFI or drive manufacturers to wipe correctly as their implementations are sometimes shortcuts and/or buggy, and their source code is not open-source and verifiable. For example if SecureErase is implemeneted in the UEFI of a current mainboard it might work on this specific UEFI/BIOS version, but another mainboard model from the same vendor might have a totally different implementation and it might be buggy. We just don't know as information is sparce and they don't share versioning numbers, changelog or code of their SecureErase implemenation.

Well low and behold Dell just released an update for one of their laptops in our fleet admitting that the UEFI wipe wasn't working properly confirming in the real world what we know only from academic research of the past.

- Fixed the issue where the Data Wipe function does not work on certain solid state drives. source: https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=vxgmt

PartialVolume / shredos.x86_64

Update NWipe's Wiping Methods #68