fthobe
commented
3 weeks ago @PartialVolume do you want me to add anything to the issue?
Open fthobe opened 3 weeks ago
fthobe
commented
3 weeks ago @PartialVolume do you want me to add anything to the issue?
PartialVolume
commented
3 weeks ago The current documentation does not outline the state of features regarding the process of sanitising SSDs (neither SAS / SCSI nor Sata). Information is scattered across multiple issues leaving a lot of unclarity.
Agreed, some clarification is required. Anybody wishing to submit a PR for review to improve the readme.md is always welcome.
For what it's worth, these are my comments to the questions raised: Some might feel I'm being a bit O.T.T but I'll try to justify my reasoning.
- Can SSDs currently be wiped with the same degree of security as rotating disks?
- If not, what are the information lacking?
Depends how you are doing it, generally I would say yes, there is more risk, especially if you are not doing it in the way I'm going to suggest. For me, the absolute best method is to use the manufacturers tool to perform a secure erase, or if unavailable hdparm or nvme, THEN follow up with SHREDOS/Nwipe with a single PRNG stream with verification, (NOT Zeros/ones or any of the other methods, this has to do with compression, i.e PRNG data is extremely hard if not impossible to compress THEN finish with a secure erase with the manufacturers tool or hdparm/nvme to reset the drive. For me that would probably be the gold standard, however if the drive was just being reused in the same organisation I probably wouldn't use all three methods. For a drive that is leaving the organisation to be sold on I would not use only ShredOS/nwipe at the current version without also using a secondary method such as the manufacturers tool or hdparm/nvme. While a PRNG stream may write over some of the over-provisioned area (used for garbage collection) it cannot be sure this was overwritten. You have to assume that the manufacturers tool or hdparm/nvme wipes these areas. Trouble is you can't verify that. And that's why I prefer a three step wipe, basically covering all your bases.
- Plenty of manufacturers offer Linux Tools for sanitisation of SSDs, are those currently utilized?
See answer to 1. & 2. If they are available they should be used in conjunction with a PRNG wipe & verification using ShredOS/nwipe.
- Should manufacturer tools be implemented if the are provided in a Linux version?
Yes, however don't assume the manufacturers tools or hdparm/nvme are flawless as they all assume the drive controller firmware is bug free.
- If manufacturer tools should not be implemented what is the reasoning behind it?
If manufacturer tools or third party tools hdparm/nvme are not used, i.e. only doing a ShredOS/nwipe traditional block erase then you can't be sure the entire overprovisioned area is wiped. A person with sufficient skills may be able to remove the memory from the drive and with the right equipment read the chips and find the over-provisioned area.
So risk has to be assessed, is the data valuable. For organisations such as the military, I would expect on-site physical destruction of SSD's, no question. For Finance I would also expect physical destruction. Not necessarily on site but maybe.
This is the current state of play with ShredOS/nwipe. Hopefully in the not to distant future ShredOS/nwipe will integrate secure erase into it's code for a more seamless method that includes a method that automatically does a secure erase followed by prng stream verification and a final secure erase. A second method that only does a prng stream plus verification followed by secure erase may also be an option.
PartialVolume
commented
3 weeks ago One further comment, both nvme and hdparm are included in ShredOS and available in most Linux distributions. Manufacturers tools that are for Linux but are proprietary would have to be sourced and might not work in ShredOS and therefore would have to be used with the Linux distribution they were built for.
mdcato
commented
3 weeks ago I concur with @PartialVolume and will add, that “Secure Erase” from the SSD mfrs seems to be a simplified process given the few seconds it takes to complete. My best guess is that they write a single block with the low-power bit (usually 1s) and then set all the block pointers to this single block, given how quick it completes. Whether the firmware then sets all the unused blocks to low-power resting state will also depend on whether the SSD remains powered, and if the firmware developers bothered to implement it as a background task. The point is, you cannot know how thorough the mfrs’ “Secure Erase” process is, thus @PartialVolume’s method should be followed to remove private data.
From: PartialVolume @.> Sent: Monday, August 19, 2024 07:47 To: martijnvanbrummelen/nwipe @.> Cc: Subscribed @.***> Subject: Re: [martijnvanbrummelen/nwipe] Documentation: Sata and SAS SSD support (Issue #587)
The current documentation does not outline the state of features regarding the process of sanitising SSDs (neither SAS / SCSI nor Sata). Information is scattered across multiple issues leaving a lot of unclarity.
Agreed, some clarification is required. Anybody wishing to submit a PR for review to improve the readme.md is always welcome.
For what it's worth, these are my comments to the questions raised: Some might feel I'm being a bit O.T.T but I'll try to justify my reasoning.
Depends how you are doing it, generally I would say yes, there is more risk, especially if you are not doing it in the way I'm going to suggest. For me, the absolute best method is to use the manufacturers tool to perform a secure erase, or if unavailable hdparm or nvme, THEN follow up with SHREDOS/Nwipe with a single PRNG stream with verification, (NOT Zeros/ones or any of the other methods, this has to do with compression, i.e PRNG data is extremely hard if not impossible to compress THEN finish with a secure erase with the manufacturers tool or hdparm/nvme to reset the drive. For me that would probably be the gold standard, however if the drive was just being reused in the same organisation I probably wouldn't use all three methods. For a drive that is leaving the organisation to be sold on I would not use only ShredOS/nwipe at the current version without also using a secondary method such as the manufacturers tool or hdparm/nvme. While a PRNG stream may write over some of the over-provisioned area (used for garbage collection) it cannot be sure this was overwritten. You have to assume that the manufacturers tool or hdparm/nvme wipes these areas. Trouble is you can't verify that. And that's why I prefer a three step wipe, basically covering all your bases.
See answer to 1. & 2. If they are available they should be used in conjunction with a PRNG wipe & verification using ShredOS/nwipe.
Yes, however don't assume the manufacturers tools or hdparm/nvme are flawless as they all assume the drive controller firmware is bug free.
If manufacturer tools or third party tools hdparm/nvme are not used, i.e. only doing a ShredOS/nwipe traditional block erase then you can't be sure the entire overprovisioned area is wiped. A person with sufficient skills may be able to remove the memory from the drive and with the right equipment read the chips and find the over-provisioned area.
So risk has to be assessed, is the data valuable. For organisations such as the military, I would expect on-site physical destruction of SSD's, no question. For Finance I would also expect physical destruction. Not necessarily on site but maybe.
This is the current state of play with ShredOS/nwipe. Hopefully in the not to distant future ShredOS/nwipe will integrate secure erase into it's code for a more seamless method that includes a method that automatically does a secure erase followed by prng stream verification and a final secure erase. A second method that only does a prng stream plus verification followed by secure erase may also be an option.
— Reply to this email directly, view it on GitHubhttps://github.com/martijnvanbrummelen/nwipe/issues/587#issuecomment-2296493823, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANGK2PTTNQPD2UYDVGWLTWLZSHSNLAVCNFSM6AAAAABMW2WOP6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJWGQ4TGOBSGM. You are receiving this because you are subscribed to this thread.Message ID: @.**@.>>
fthobe
commented
3 weeks ago Hey guys, Soooo... @PartialVolume OTT responses are always appreciated on documentation issues 😉 @mdcato I can reach out to manufacturers if that's of interest.
I can work on the docs, to recap what should be done in any case:
Regarding secure / enhanced sanitization: there's some low hanging fruit:
Problematic Cases:
I have significant contacts at Samsung and Western Digital and could start mapping out how can be wiped with ATA / SCSI standard commands and do some documentation legwork to establish a compatibility matrix standard sanitization commands and we could take it from there. Would that be interesting?
PartialVolume
commented
3 weeks ago Would that be interesting?
Yes, go for it. At some point we will be adding secure erase to ShredOS/nwipe but we can always adjust the documentation when that happens.
fthobe
commented
3 weeks ago Would that be interesting?
Yes, go for it. At some point we will be adding secure erase to ShredOS/nwipe but we can always adjust the documentation when that happens.
Do you have any idea how far that is down the road?
Basically we are talking about the integration of
If you want to go a little into detail I can check for sponsors.
PartialVolume
commented
3 weeks ago Do you have any idea how far that is down the road?
Unfortunately, I can't provide a schedule for this. Other things in my life are currently taking priority so if I write the code it will be when i can fit the work in.
Basically we are talking about the integration of
SATA Sanitize SAS Sanitize NVME Sanitize Integrated in the nwipe application, correct?
Yes, all will be integrated so you can select secure erase, secure erase enhanced. In addition to the secure erase, verification of a secure erased drive will require some code to first analyse the pattern used by the firmware. Sometimes the drives firmware will leave all zeros but then on other drives you might get a pattern of some sort, like 0x003F etc, so the verification code needs to be able to handle unknown repeating patterns. I've not come across any drives that leave prng data after a secure erase which is a good thing in terms of verification as it would be impossible to verify without knowing the seed the firmware used.
The code also needs to correctly handle frozen drives, i.e sending the system to sleep and waking after 5 seconds, then initiating the secure erase.
Also code needs to be written to provide progress based on the drives estimation. One of things I really don't like is that you can't interrogate the drive for a progress report after a secure erase has started, as far as I'm aware, so we have to estimate the completion time and then see if it's finished yet. This means the progress code will be different to the way the existing progress code works.
And finally the reports need to be updated with the new methods etc.
Sponsers are ok, but I don't have the time to put 8hrs a day, 5 days a week into this. I tend to write large amounts of code when I can find some free days or a free week. Hence why I can't provide a timeline.
Currently, top of my priority list https://github.com/users/PartialVolume/projects/1 is
fthobe
commented
2 weeks ago I walked through the several block command standards for SAS, NVME and SATA.
Yes, all will be integrated so you can select secure erase, secure erase enhanced.
Secure erase seems to be the older of the two commands and it seems like the certification bodies have made significant effort to standartize around sanitize commands (they were from the get go present on nvme and implemented into SAS / Sata in later revisions). Especially older drives to not seem to treat secure erase reliably in the same way.
Also code needs to be written to provide progress based on the drives estimation.
I absolutely believe in the value of progress bars, but I believe the time varies to largely in between drives. Maybe a continuous polling of the drive would be better.
Sponsers are ok, but I don't have the time to put 8hrs a day, 5 days a week into this. I tend to write large amounts of code when I can find some free days or a free week. Hence why I can't provide a timeline.
Do you feel there is somebody among past contributors that might help you on that?
Issue
The current documentation does not outline the state of features regarding the process of sanitising SSDs (neither SAS / SCSI nor Sata). Information is scattered across multiple issues leaving a lot of unclarity.
Therefor I pose following questions to later on produce a merge request against the current documentation and where necessary and (if desired) evaluate the sponsoring for the closing eventual gaps in functionality by our organisation.
Background
SSDs have been a dramatically growing segment within the storage technology space posing new challenges to the sanitisation of storage media when reaching end of life events. Particular challenges are caused by the fact that control over data allocation on the disk is left to intransparent controllers and / or firmwares. Some manufacturers do provide sanitisation tools for linux, while others do not.
By 2022 the situation was following:
The increase of usage of directly PCI attached storage technologies such as NVME and the growing market share of large size SSD s make lack of documentation and control over sanitisation an ever increasing vector of attack for data theft. Many companies rely therefor on physical destruction (eg shredding) to avoid missuse, but plenty are not sufficiently informed about the neccessaty to approach data sanitisation differently on solid state disks.
Notes
This issue has been crossposted to shredOS.