search

martijnvanbrummelen / nwipe

nwipe secure disk eraser
GNU General Public License v2.0
804 stars 85 forks source link

Final blanking/verification pass with ones #138

Open n3gwg opened 5 years ago

n3gwg commented 5 years ago

It seems that with all the good intentions in the world this software provides a mechanism to let you "zero out" a disk. Whilst this may seem fine, what is really needed is to convert this option to "one out" a disk. Yes, write all ones instead of all zeros.

Huh? Why? What difference does that make?!???

The answer to these questions is that the life of a mechanical disk is entirely indifferent to if it is "oned out" or "zeroed out". However, in the case of an SSD, to "one out" a drive extends the life of the SSD where to "zero out" the SSD reduces its life.

Hope this change can be made easily and quickly.

Stuart

PartialVolume commented 5 years ago

Having a one's out & verification as an option as opposed to a zero out and verification is certainly possible.

n3gwg commented 5 years ago

Developer(s),

With the onslaught of SSDs into the market now and their being rather ubiquitous, there is surely an incumbency upon software developers to now consider this. For example, VMware compacts its VMDK files based on "zeroing them out first". For what its worth I did post a request for them to change this philosophy as well as an idea in the VMware Community VMware Fusion forum as well.

Thank you for your attention to this important matter.

Stuart

PartialVolume commented 5 years ago

Not that I'm doubting what you say, but can you provide a link to an authoritative paper or manufacturer's information that confirms that writing ones to a SSD rather than zeros causes degradation. My knowledge of how SSD's work is certainly limited and the forums that I have read about whether you should write zeros or one's seems to have opposing views so any evidence based data would be really useful.

That said it would be straight forward to add an option to nwipe that would wipe with one's and verify one's.

mdcato commented 5 years ago

I concur. While I found some references to a preference for ones on OLD drives, there is also reports that writing any consistent data, ones or zeros, is optimized by modern controllers. That is, writing all ones would result in a single physical block containing the ones, and all logical all-ones blocks would point to the same physical block. Similarly, and I’m extrapolating, I could see a controller inverting the bits to its preference thus ones-wipe might not really make a difference. Security wise, someone could attach directly to the flash and read data, but the controller would hide this through the SATA interface (effectively no different than putting spinning hard drive platters in a clean room and performing forensics on them). Since NVMe is a “logical device interface”, I am assuming the same hiding occurs here as well; someone more versed in NVMe should chime in.

I’d say it would be a nice option for nwipe to have a wipe-with-ones option; I’d also say it’s a low priority for the sake of modern SSDs.

That said, and because I also gleaned that the preferred way to wipe SSDs is with the ATA Secure Erase [Enhanced] function, which essentially encrypts the entire drive with a new key. (An aside, I’m assuming from the near-equivalent speed of execution of Samsung’s proprietary SSD erase and HDPARM’s ATA Secure Erase, that Samsung is using ATA Secure Erase.) Note that it wouldn’t work for partition-level wipes.

What I have been using with HDPARM is to first determine of the drive supports Enhanced erase or not, then issue: hdparm –user-master u –security-set-pass --security-erase[-enhanced]

From: PartialVolume notifications@github.com Sent: Friday, November 29, 2019 09:14 To: martijnvanbrummelen/nwipe nwipe@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: Re: [martijnvanbrummelen/nwipe] add method=one (#138)

Not that I'm doubting what you say, but can you provide a link to an authoritative paper or manufacturer's information that confirms that writing ones to a SSD rather than zeros causes degradation. My knowledge of how SSD's work is certainly limited and the forums that I have read about whether you should write zeros or one's seems to have opposing views so any evidence based data would be really useful.

That said it would be straight forward to add an option to nwipe that would wipe with one's and verify one's.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/martijnvanbrummelen/nwipe/issues/138?email_source=notifications&email_token=ANGK2PR3XUMPTFEGQFYY5TDQWEWUXA5CNFSM4JSJWMVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFPC2NA#issuecomment-559820084, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANGK2PQTPH4D5UJVMKW7LJLQWEWUXANCNFSM4JSJWMVA.

IDerr commented 5 years ago

Found some infos on that here: https://serverfault.com/questions/282555/zeroing-ssd-drives

Hope it can help a bit :D

mdcato commented 5 years ago

Interesting thread, started 6/11, latest comment 9/18. BUT, the insight that Ext4 does a TRIM helps with nwipe’ing partitions. Did a little searching and in https://sites.google.com/site/lightrush/random-1/linuxfilesystemsformodernssdsandtrimsupport (from 2010!) it is stated TRIM is supported for Ext4 formatting from kernel 2.6.33 on, as long as partition is aligned to block size, and TRIM is enabled for the partition. (BTRFS is also mentioned and discarded.) The TRIM should invalidate all blocks for that partition prior to the format which would simply write Ext4 structures. Combined with the serverfault link in previous reply, it would seem from a high level that individual partitions could be wiped via an Ext4 format, and for whole drive, the gparted library could remove all partitions, create a single full-disk partition which would be formatted with TRIM enabled, and then the partition deleted with the gparted library. This would provide a fairly thorough wiping (inability to access former data) with minimally induced wear to the SSD and thus greater reusability. My position is that nwipe’s purpose is to safely allow re-use of the drive, otherwise they should be physically destroyed; safety without overly wearing the SSD is the goal.

Additionally for greater scope, this article from 2017 mentions additional filesystems besides Ext4 and pros/cons. But since Ext4 is built into the kernel, it would seem to have the widest availability of various Linux distributions. https://www.maketecheasier.com/best-linux-filesystem-for-ssd/

From: iderr notifications@github.com Sent: Friday, November 29, 2019 13:55 To: martijnvanbrummelen/nwipe nwipe@noreply.github.com Cc: Mike Cato / Hays Technical Services mike@haystech.com; Comment comment@noreply.github.com Subject: Re: [martijnvanbrummelen/nwipe] add method=one (#138)

Found some infos on that here: https://serverfault.com/questions/282555/zeroing-ssd-drives

Hope it can help a bit :D

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/martijnvanbrummelen/nwipe/issues/138?email_source=notifications&email_token=ANGK2PUODGVVU3NL2KRHDQDQWFXSBA5CNFSM4JSJWMVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFPPR3A#issuecomment-559872236, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANGK2PU5CXTRERFJB6OX4ELQWFXSBANCNFSM4JSJWMVA.

PartialVolume commented 5 years ago

Thanks for those links. I recently upgraded my laptop from a 160GB SSD to a SanDisk 240GB and wasn't aware of the requirement of aligning the partition to 1024 and manually enabling the trim option in fstab to benefit from that feature. Very useful to know. Thanks !

PartialVolume commented 5 years ago

updated

and manually enabling the trim option in fstab to benefit from that feature.

by adding discard in fstab, once you have done that then the partition will actually be trimmed. /dev/sda1 / ext4 discard,errors=remount-ro 0 1

$ systemctl status fstrim.timer ● fstrim.timer - Discard unused blocks once a week Loaded: loaded (/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled) Active: active (waiting) since Thu 2018-07-05 05:45:11 BST; 4h 42min ago Trigger: Mon 2018-07-09 00:00:00 BST; 3 days left Docs: man:fstrim`

To show the periodic trim timer is running but apparently it will do nothing unless you put discard into fstab.

On my last 160GB SSD (that was still going strong after 5 years) I never had trim enabled and never even realised !

mdcato commented 5 years ago

Nice that ubuntu does a weekly trim. My concern related to wiping, is making sure TRIM is enabled before an Ext4 format is done and not relying on it being on by default. (This is assuming that method is used for wiping SSDs.) I’m remembering prior discussions/issues of nwipe behaving differently on various Linux distributions, so don’t want to rely on default settings. On the other hand, since TRIM has been supported in Ext4 since kernel 2.6.33, it seems reasonable to assume Ext4 is present and TRIM supported across several distributions. But a check to make sure the kernel is that version or later is probably prudent.

From: PartialVolume notifications@github.com Sent: Saturday, November 30, 2019 10:45 To: martijnvanbrummelen/nwipe nwipe@noreply.github.com Cc: Mike Cato / Hays Technical Services mike@haystech.com; Comment comment@noreply.github.com Subject: Re: [martijnvanbrummelen/nwipe] add method=one (#138)

and manually enabling the trim option in fstab to benefit from that feature.

Not that manually enabling trim in fstab is necessary as ubuntu does a weekly trim by default.

$ systemctl status fstrim.timer ● fstrim.timer - Discard unused blocks once a week Loaded: loaded (/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled) Active: active (waiting) since Thu 2018-07-05 05:45:11 BST; 4h 42min ago Trigger: Mon 2018-07-09 00:00:00 BST; 3 days left Docs: man:fstrim`

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/martijnvanbrummelen/nwipe/issues/138?email_source=notifications&email_token=ANGK2PWKEEZY5SLUO3S4E7TQWKKA3A5CNFSM4JSJWMVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFQNHPA#issuecomment-559993788, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANGK2PQAOKOTSJRDRA6TZ73QWKKA3ANCNFSM4JSJWMVA.

IDerr commented 5 years ago

just for clarity, can we rename this issue to "wipe ssd" or something like this ?

PartialVolume commented 5 years ago

@IDerr I'm OK with renaming to SSD wipe, unless there is any objections.

louib commented 5 years ago

@PartialVolume there's already an issue for SSD erase https://github.com/martijnvanbrummelen/nwipe/issues/101

PartialVolume commented 5 years ago

How about Final blanking pass/verification with one's for SSD?

IDerr commented 5 years ago

yes, perfect.

mdcato commented 5 years ago

@iderr, @PartialVolumemailto:notifications@github.com I’ll note again that this is not necessarily SSD specific, and that there is some question whether this is valid/useful for current SSDs (although valid for some older ones), and that in modern SSDs the controller creates one physical block with ones and then points all subsequent logical blocks to this physical one, thus this method is not effective at actually wiping all blocks.

That said, I think the ability to specify a pattern is a good idea (zeros, ones, any other value), it just shouldn’t give the impression that it is SSD-specific.

Other SSD-specific work should be done (as mentioned previously re: TRIM/Ext4 partition wipe and Secure-Erase whole device wipe) to keep nwipe’s quality as high as possible.

From: iderr notifications@github.com Sent: Tuesday, December 3, 2019 03:35 To: martijnvanbrummelen/nwipe nwipe@noreply.github.com Cc: Mike Cato / Hays Technical Services mike@haystech.com; Comment comment@noreply.github.com Subject: Re: [martijnvanbrummelen/nwipe] add method=one (#138)

yes, perfect.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/martijnvanbrummelen/nwipe/issues/138?email_source=notifications&email_token=ANGK2PS4MWNU53XQ2XO2H6TQWYR53A5CNFSM4JSJWMVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFYWRAI#issuecomment-561080449, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANGK2PVMOOIFMFEPYLQIG33QWYR53ANCNFSM4JSJWMVA.

PartialVolume commented 5 years ago

@mdcato I would also find a ones final pass useful. In my case as I test the code I need to make sure the disc is really being written so it must start of with the disc containing something other than zeros, so being able to alternate between a final blanking of zeros and then the next time I wipe the disc it will be ones and so would be useful. Currently I achieve the effect I require by using the PRNG stream.

JiMi7 commented 4 years ago

Hello, I also find zeroing or oneing important, but not for the same reasons I have read here. Before, I was using DBAN and I have just switched to Nwipe :-) I am glad to see that development is still intense : thanks to all the contributors :-)

I sometimes use those programs to wipe disks; but in fact I really use them for another purpose ;-) I don't really know if I am right, but I find those programs useful in order to test HDDs before using them. It can concern old HDDs which I am not really sure of the reliability any more; but it can also concern new HDDs that can have some flaws; and stressing them can reveal mechanical vulnerabilities that could have been detected only 1-2 years after, in a normal environment; which would be a very bad surprise (out of warranty and/or loss of data)

The wiping process is also useful in order to analyse if what's read is identical to what has been written; because everybody wants a reliable disk ;-) I use the PRNG method, but is it really 100% sure that all the bits on the disk are changed from 0 to 1 and from 1 to 0; let's say after 5 to 8 passes ? I doubt it can be guaranteed. In this context, it would be more sure to make a zeroing and then a oneing (the contrary would also be ok), would it not ?

And finally, to make everyone happy, there could be those blanking options : -Zeroing -Oneing -Zeroing, then oneing -Oneing, then zeroing

Perhaps the fact that Nwipe is useful not only for wiping but also for HDDs integrity checking could be added on the readme file ?

I am looking forward to reading from you :-)

JiMi7 commented 4 years ago

Hello, I have just noted that there are 2 wipe methods called: -"Zero Fill" -"HMG IS5 Enhanced" which does the following: "This method fills the device with 0s, then with 1s, then with a PRNG stream, then reads the device to verify the PRNG stream was successfully written"

That could do the job concerning analysing the reliability of a HDD :-) One has just to choose Verify : "all passes"

The only problem I can see is that not everybody has a computer dedicated for wiping, mostly those who use this kind of software at home for personal use. Personally, I use Nwipe mostly at night. The problem is that for a 2 To HDD, it takes approximatively 12 hours to do 1 write and 1 verification. So, if one does a "Zero Fill" or a "HMG IS5 Enhanced" and stops the wipe 12 hours after just after the first pass of 0s; and then begins a new wipe the next night, it will again be 0s and not 1s.

PartialVolume commented 4 years ago

@JiMi7 I can see how that could be useful for people having to run windows during the day and then wiping discs at night using nwipe.

A 'Pause' feature could be added to nwipe. Either a scheduled pause or manual pause or probably both.

When you hit pause information about the current wipe is written to block 0 of each disc. This information would contain the method, verify type, current round, current pass, current block, date and time paused. You could then exit nwipe. Run windows during the day then when you've finished with windows, boot back into nwipe (if you're using nwipe within ShredOS). Nwipe will automatically read block 0 looking for it's signature and wipe information and then ask you if you want to resume the wipe or start from the beginning.

I wonder if there would be interest in such a feature ? It would certainly be quite cool to implement. Not sure I've ever come across a pauseable erasure program ? Could be a unique feature.

PartialVolume commented 4 years ago

@JiMi7

I don't really know if I am right, but I find those programs useful in order to test HDDs before using them. It can concern old HDDs which I am not really sure of the reliability any more; but it can also concern new HDDs that can have some flaws; and stressing them can reveal mechanical vulnerabilities that could have been detected only 1-2 years after, in a normal environment; which would be a very bad surprise (out of warranty and/or loss of data)

The wiping process is also useful in order to analyse if what's read is identical to what has been written; because everybody >wants a reliable disk ;-)

Yes, I use nwipe to also test the reliability of discs.

PartialVolume commented 4 years ago

@JiMi7

I use the PRNG method, but is it really 100% sure that all the bits on the disk are changed from 0 to 1 and from 1 to 0; let's say after 5 to 8 passes ? I doubt it can be guaranteed. In this context, it would be more sure to make a zeroing and then a oneing (the contrary would also be ok), would it not ?

And finally, to make everyone happy, there could be those blanking options : -Zeroing -Oneing -Zeroing, then oneing -Oneing, then zeroing

Perhaps the fact that Nwipe is useful not only for wiping but also for HDDs integrity checking could be added on the readme file >?

I don't have any issues with adding extra methods that follow the patterns you describe above. We just need to make sure we're not duplicating an existing method.

JiMi7 commented 4 years ago

@PartialVolume

A 'Pause' feature could be added to nwipe.

That would be so great :-))) I'm voting for it !

Either a scheduled pause or manual pause or probably both.

I find a manual pause would be great, as well for stopping the wipe than for resuming it.

When you speak of scheduled pause, do you think of a scheduled pause and/or a scheduled resume ? Perhaps a schedule could be a good feature. Here are some of my thinkings :

Concerning an automatic shutdown when the wipe is 100% done, it could be a good feature, but the automatic log file save feature should be completed before (I saw it was a possible future feature :-) ). That would be useful for example if the wipe finishes at 8 in the morning but the computer owner only comes back at home at the end of the day.

When you hit pause information about the current wipe is written to block 0 of each disc.

Just to better understand : block 0 is the "first" block on the surface that has to be wiped ? So at the end of the wipe; should this block be specially erased using the same method applied to the rest of the disk ? (for example, 7 rounds of PRNG with verification for all the passes and a final blanking) (paranoid mode : because information on how a wipe was done could bring information on how to read the original data !? ...)

Yetoo1 commented 3 years ago

updated

and manually enabling the trim option in fstab to benefit from that feature.

by adding discard in fstab, once you have done that then the partition will actually be trimmed. /dev/sda1 / ext4 discard,errors=remount-ro 0 1

$ systemctl status fstrim.timer ● fstrim.timer - Discard unused blocks once a week Loaded: loaded (/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled) Active: active (waiting) since Thu 2018-07-05 05:45:11 BST; 4h 42min ago Trigger: Mon 2018-07-09 00:00:00 BST; 3 days left Docs: man:fstrim`

To show the periodic trim timer is running but apparently it will do nothing unless you put discard into fstab.

On my last 160GB SSD (that was still going strong after 5 years) I never had trim enabled and never even realised !

I'm not sure where you got the info that trim only works if discard is set on a drive in fstab, but from my understanding setting discard in fstab puts the drive in continuous trim mode rather than the weekly trim which apparently puts a lot of wear on the drive. I know Archwiki isn't an authoritative source, but it has a nice explanation for the different settings: https://wiki.archlinux.org/title/Solid_state_drive

PartialVolume commented 3 years ago

@Yetoo1 I've no idea where I got that, although I was talking about a laptop with the160GB drive from quite a few years ago, early days of SSDs with a Ubuntu release where I don't think they had started properly supporting trim and the only info that was available was via the odd forum where sometimes information can be a bit suspect. Nowadays Ubuntu just gets on with it and I never have to worry about trim, at least I assume the OS is periodically trimming. I've not looked at the system logs in a while.

m1karii commented 3 years ago

A 'Pause' feature could be added to nwipe. Either a scheduled pause or manual pause or probably both.

When you hit pause information about the current wipe is written to block 0 of each disc. This information would contain the method, verify type, current round, current pass, current block, date and time paused. You could then exit nwipe. Run windows during the day then when you've finished with windows, boot back into nwipe (if you're using nwipe within ShredOS). Nwipe will automatically read block 0 looking for it's signature and wipe information and then ask you if you want to resume the wipe or start from the beginning.

Some people use nwipe before disposing of physically damaged drives for additional security (lets assume these people dont completely physically destroy the drive afterwards). For cases such as these there should be an optional log file that would store all information about the command and the progress of completion in case of failure / interruption of the erasing process or as in your example if the process was paused (presumed the option is available). With physically damaged drives writing such information onto the drive itself ("block 0") would also not be a good idea for obvious reasons.

PartialVolume commented 3 years ago

@m1karii

For cases such as these there should be an optional log file that would store all information about the command and the progress of completion in case of failure / interruption of the erasing process or as in your example if the process was paused (presumed the option is available).

For the nwipe v0.32 release I'm planning on adding a new optional feature related to your comment above. On completion of a successful or failed wipe a pdf certificate is produced detailing the drive information, wipe details, possible smart info etc. Each certificate will pertain to an individual drive. Along with a fancy colour pdf version complete with graphics and icon and text version will include the basic data and could be processed by a users own procedures if need be. I'm going to try to make this as customisable as possible so that layouts and pdf backdrops could be defined by the user. Basically if they don't like the default nwipe certificate, then they could design their own. This is something I've personally wanted for a long time now and I'm only now getting to the stage where I'm generally happy with nwipe and I can start coding it. Not to say there are plenty of other improvements waiting in the wings.

m1karii commented 3 years ago

I meant it (the log file) more as an ability to resume failed / paused wipes as in https://github.com/martijnvanbrummelen/nwipe/issues/329 (I wrote it here before realising it had its own issue)

PartialVolume commented 3 years ago

A file storing the pause information is certainly an option. However, I quite like the idea of storing encrypted pause info on the disc itself although both storage locations could be incorporated with a command line switch depending on your preference.