Help to reformat hard disk sector size 528 to 512

[yemencit]

I have problem with reformat this hard disk from 528 to 512 please help me note this is hard disk ibm and working flash ssd 0kb

[doug-gilbert]

Try adding the --long option to the sg_format command line. Also an extra verbose option (i.e. '-vv') would help looking at the failed output. It is possible that IBM have changed the firmware on that disk (made by Seagate) to stop them being reformatted to 512 byte sectors :-( Also try sg_readcap on that disk. It might have 4096 (plus a few) byte sectors.

[yemencit]

Thank you for your reply can I chnage frimware for this hard disk

[doug-gilbert]

I think that it is important that you run 'sg_readcap PD1' on that disk. It could be 4096+n byte sector size. I have been told that IBM allow 5xx byte sector size to be reformatted to another 5xx sector size (e.g. 520 to 512 byte). They also allow 4xxx bytes reformattings. But that don't allow 5xx to 4xxx and vice versa. Further I have been told that you cannot put Seagate firmware on that disk. If you google "Seagate STT004" you should find the Seagate product manual for that disk.

[yemencit]

Please see what I found when use rg_readcap

[yemencit]

And please Check this ![Uploading 0DBE80AB-34CB-4FA7-880B-208F2BF92C15.jpeg…]()

[doug-gilbert]

No good news after this long delay. The link above shows that the disk says there is an error starting at byte 13 (bit 7, so that is the whole byte, or in this case, the whole field). The command being sent is MODE SELECT(10) for the Read-write Recovery mode page (0x1). There is no --long option on the sg_format invocation so there is an 8 byte "General mode parameter block descriptor" which follows an 8 byte "Mode parameter header(10)" in the parameter list sent with the MODE SELECT(10) command. Byte 13 is the start of the "BLOCK LENGTH" field in the mode parameter block descriptor. And that field is exactly why that MODE SELECT is sent: to change the block length from 528 bytes to 512 bytes. Two different MODE SELECT(10) variants are tried: with the SP=1 then SP=0. Same results both times: the disk firmware is saying that it won't change the block length from 528 to 512 bytes. I have a HGST SAS SSD with an IBM sticker on it (luckily it is 512 byte formatted). Out of interest I asked an engineer in WDC if I could change the IBM firmware to standard HGST firmware and I was told NO! So I suspect there is some contractual arrangement between the folks that actually make these disks (Seagate, WD, Toshiba, Samsung, etc) and those who do "badge engineering (e.g. IBM, HP, DELL, etc) to stop exactly what you are trying to do :-(

[catalin7504]

I am in a similar situation, with an IBM 3.8TB SAS SSD drive. So, based on your last comment, changing the firmware with a standard HGST drive is not technically possible, or it's something that IBM does not want us to do? If it's not possible to change the firmware, is there any other option other than using them in an IBM storage unit? Thank you!

[doug-gilbert]

HGST is now owned by WD. Various companies like IBM, Dell and HP rebrand disks made by the likes of Seagate and WD. Obviously the contracts behind those "badge engineering" deals are confidential but it seems like disk manufacturers promise not to allow owners of those IBM/Dell/HP disks to bypass the specialized firmware they have on them. I wasn't told exactly that, but that seems to be the gist of some feedback from a technical person within WD when I asked about this case. I'm also not aware of how to tell Linux (for example) how to use the first 512 bytes of each logical block and ignore the rest but I can ask. Ah, I see from above that those disks report PROTECT=0 in the INQUIRY response, so PI is not an option.

[catalin7504]

Thank you for replying! I am not familiar with drive firmware, sas protocol etc., but I looked at the documentation provided by Seagate here. Chapter 6.4 Drive locking states that:

When the drive is shipped from the factory, the firmware download port is unlocked allowing the drive to accept any attempt to download new firmware. The drive owner must use the SID credential to lock the firmware download port before firmware updates will be rejected.

Also, chapter 6.7 Authenticated firmware download

In addition to providing a locking mechanism to prevent unwanted firmware download attempts, the drive also only accepts download files which have been cryptographically signed by the appropriate Seagate Design Center

Based on these notes, it seems that IBM has the option to lock the firmware port. I assume this is the port in front of the disk.

Also, looking online I found this page where the user ioan mentions that: you can try buying iprconfig-compatible sas controller (list of supported controllers you can find in source code) and then use iprconfig to convert them to jbod mode

Not sure how to do this, I understand I need a supported IBM sas card, but not sure if this needs to be done from the original storage system where these drives were running.

[doug-gilbert]

And this looks promising: https://github.com/bjking1/iprutils

[catalin7504]

Hello, I have some more details, and since I'm not familiar with Linux at all, maybe someone can figure out what I'm doing wrong. So, I purchased 2 controllers: an IBM ServerRaid M5015 and an IBM 57B3. I have a Dell workstation which I'm using and I installed Ubuntu. The controller M5015 is not useful, it sees the drive in the controller utility but cannot work with it at all. The second controller 57B3 is using 2 external sff-8088, so I connected the drive using cables. First thing, I don't know if this controller has any utility that can be accessed before OS loading (like an LSI utility), I could not find how. Second, after building the iprutils I can access the controller and the drive. The drive is shown as 2.2TB, instead of 3.8TB, but that's something maybe for later. Using the menus in iprutils, I can format the drive as RAID 0 and it will be visible in Ubuntu, I can create a partition and create a folder there. All the time the drive is reported as 512 sector size, but if I connect it to a normal controller it's 528. So, it seems that the RAID controller is hiding this info for the OS. There is an option to convert to JBOD, but whatever I tried I receive a message that "Initialization and format failed". I took me some time to figure out where the logs are, and I could find there messages suggesting something related to "Mode select" (sorry, I took a photo with my phone, but somehow it was not saved in the phone, will try to capture again).

At this point, I'm not sure if I really need an IBM Power machine, or if the controller I purchased is not the right version, or anything else....

[mr44er]

Out of interest I asked an engineer in WDC if I could change the IBM firmware to standard HGST firmware and I was told NO! So I suspect there is some contractual arrangement between the folks that actually make these disks (Seagate, WD, Toshiba, Samsung, etc) and those who do "badge engineering (e.g. IBM, HP, DELL, etc) to stop exactly what you are trying to do :-(

Yes, they don't want this. They wanna sell 'special firmware' for 3x normal price. But that is stupid after the chassis/filer/disk has gone EOL. Btw. I've sent you an E-Mail, tell me what you think. :+1:

[hreinecke]

It's really not worth it. HDDs don't cost a ton of money, and you get a lot of hassle if you repurpose storage array drive. Most of them have special firmware, requiring the use of T10-PI enabled commands (or even worse). And reflashing is also not a good idea, as you never know whether the firmware you are about to flash really matches your drive hardware-wise.

[hreinecke]

To reiterate: I don't think it's worth it. 528byte sector drives are spare parts for storage arrays, which are using these drives to store their internal metadata and checksums in the additional space. All of these drives have vendor-specific firmware, and there is no telling which bits of the stock firmware had been modified or disabled. If "sg_format" doesn't work I really wouldn't spend more time on that drive, but rather get a new one. Or leave it as it is; in the end, if the drive does work with 528byte sectors why would you change it? The hardware can act very funny if you start reformat it (tracks being in a different position, head adjustment going askew requiring constant repositioning etc...)

[hreinecke]

You can try to reformat them with 520 byte sectors. That might work, and would allow them to be used by linux.

[mr44er]

You might want to read this: https://hddoracle.com/viewtopic.php?f=59&t=3279 https://hddoracle.com/viewtopic.php?f=59&t=3280

These disks have software locks, it's all in the firmware on the disk. I tried more or less some brutal methods, changing firmware-parts, but killed some disks on the way.

You don't need special hardware, a regular SAS2/SAS3 HBA with IT-firmware that can talk directly to the disk is enough. I also have no solution, but narrowed it a bit down. If partitcular 3rd vendor firmware has locks (no other sector size than 520 (SSD should have 4096) or no write without verify), you must change the whole firmware to generic.

You need to send some magic unlock code(s?) to the disk, then a secure erase, quick format and whole firmware with mode14 (then a power on/off reset/camcontrol stop/start). In a way similar to the sbr change on a HBA ( https://marcan.st/2016/05/crossflashing-the-fujitsu-d2607/ ) I am just not sure about the sequence order and the magic codes. I found out the protection stuff for SCSI/SAS began somewhere in 2005 and upwards , it is documented at t10.org, but there are too many files.

More info is spread over config files in niagara, I just can't put it together. You need to combine the info found in the readable config files over different niagara versions, then it all narrows down to regular scsi/sas commands.

In the links hddoracle you'll find videos of baking firmware with expensive software, but that is only needed to crossflash with mode5 and mode7 directly.

The important things: quick format, secure erase and/or start/stop a special smart-test gives a small time window, where the disk accepts more or THE magic codes.

I just can't put it together.

[mr44er]

These discs slowly damaged my psychology over 4 years.

Yeah, totally understand that. :)

https://forum.hddguru.com/viewtopic.php?f=22&t=40915 <- There is this guy. He can do it for $$$, but doesn't share the secret.

From the models you can guess what goes where (firmware) and he does this trick https://www.youtube.com/watch?v=mAhS_sk3wKE

If done right, magic bytes sent to disk unlocks the firmware. Then the secure erase (or quickformat) saves the magic bytes. I just haven't found the correct steps in order. Regardless of the the disk models, the procedure has to be the same. The manufacture code part of the firmware has to be the first step and that is the requirement of all these SAS-disks.

I'd say asking him or trying with the right directions would be better than throwing them away right now.

The older the version is, the more you can read the different function files. More are scrambled in newer versions, to avoid peeking exactly that secure stuff. "Gold rush" aka "thego" variable in configs was wiped from the manuals with ??, that was the testing suite for changing and testing firmware. ftp://atxlab.ddns.net/ftp/hdd/hgst/Niagara/ https://files.hddguru.com/download/Software/HGST/ https://files.hddguru.com/download/Firmware%20updates/

sg_sanitize=secure erase

There is some config stuff to unlock the fw_dl port at the very end of some file, leading to secure stuff and the roles of manufacturer and maker(=3rd party vendor). The goal should be to switch off the protection in the disk (magic bytes? or secure erase with PSID), remove the maker/vendorroles/overlay config/running commands?, disk spin down/$maybesomeotherthing?, quickformat that removed stuff is saved and that it doesn't start the lock again, force whole firmware with mode14, repower the drive, force whole firmware again with mode5 and that should be it.

[catalin7504]

Thank you @mr44er for all the details! It makes me go back and try again with some of the IBM drives I got a while ago. I am not familiar with Disk Firmware and stuff, but it seems like something interesting to learn.

Regarding what @krkx mentioned in the previous post, I want to add that I got an IBM Power 7 (I think, need to double check) and one of the controllers I believe it's 57B3. After installing Linux and iprconfig, the drives are visible, but could not be converted to JBOD. I did not try AIX, but somehow I don't think it would be possible exclusively from software.

[mr44er]

No rule without exceptions, I guess. Anywhere he mentioned that if the firmware package is encrypted, nothing could be done, be it either encrypted and signed (from IBM/HGSTwhatever) directly on the disk or having it as binary package. Btw. I don't know him, I just know the thread. It's too bad that I haven't had success with any disk so far...maybe to see some more knobs for the steps etc...

To smooth down the monetary damage, you could sell it with exactly all details on some used-market, ebay etc. maybe someone needs exactly that disk and then you buy a disk with generic firmware.

I also found out that there are different versions of this secure stuff and I really know nothing useful about it. There is Opal, Opalite and Pyrite. https://www.truenas.com/docs/core/coretutorials/storage/sed/ On top important is some knowledge about SPC-3 and SPC-4 (SCSI primary commands, valid for SAS) and SBC-3 + SBC-4 (SCSI block commands, valid for SAS). When these drafts where born, the locking possibilities were discussed, starting ~2005 @ t10.org with tape loaders. There existed tapes with firmware updates and devices should have the option to reject wrong firmware.

[mr44er]

I have here this ready-baken crossflash firmware for HGST C10K600 generic into IBM D2F8 (rename to .7z). I don't have such a disk, but I'm a bit familiar with how these updates look in a hex editor.

C10K600.7z.zip

In the beginning you'll see an unusual loop where the possible disk models are listed twice. The first one should be fake "green card" to pass the check from the disk. The second programs magic bits and new modepages (maybe in mode4?, mode4 on HGST=temporary until otherwise saved and resets with repowering the disk), this is what I meant with the new sbr like on HBAs. At this point, the disk has still old firmware, but the allow/deny check for new firmware and then the rest is piece of cake, in theory.

Maybe someone here is good with a hexeditor and can disassemble things enough to understand things more how it's done in general...

[mr44er]

Oh and inside the segments the checksums have to be correct and I read that the file size is important. It must be dividable by 8, also the single segments.

[mr44er]

I remember that HGST disks use their serial number (8 characters/numbers) *4 resulting in 32 characters...as MSID...PSID?

[mr44er]

This gives ideas: Search in SA-modules of a generic disk for the serial Pull this SA-module from the locked OEM and compare findings Activate special mode (the one I found, hddoracle), overwrite only this SA-module with new one, maybe a simple ABCD1234 Restart disk Wipe-reset with MSID ABCD1234ABCD1234ABCD1234ABCD1234 ... try generic flash of firmware, mode5, mode7 and mode14 with restarting disk after every attempt, with small blocksize of 4096 (not to be confused with sector size)

[mr44er]

I don't like the communication via e-mail and having my mail public. Let's keep the discussion public. Either here or make an account at hddoracle.com with a new thread there (maybe better, because Seagate) or post in my thread: https://hddoracle.com/viewtopic.php?f=59&t=3279

If you really have "private" info, then pls PM me hddoracle.com

[alentoghostflame]

The person with IBM hardware apparently stopped just short of going by IBM documentation.

AIX way is to convert from pdisk (RAID candidate) to hdisk (JBOD disk). This directs controller firmware to do so. I don't know which "AIX Diag" dialog they got into, but interfacing controller requires a special package installed, documented in the same place. Linux iprconfig is also documented, similarly allowing to set disk as JBOD or RAID candidate (they use SCSI T10).

Then there's documentation for the controllers. In feature comparison it says in a footnote "JBOD is not supported on SSDs" if it is offered at all which it mostly isn't.

In older generation there are old 3 Gb/s parts offered probably for compatibility reasons. SSDs weren't a thing in 3Gb/s times. Just some controllers had them on-board and the footnote says "The integrated SSDs are allowed to run in JBOD mode."

A book-like PDF with spelled out procedures elaborates on this:

JBOD Workload Optimization Additional optimization parameters are available through the sissasraidmgr utility for the PCIe RAID and SSD SAS adapter 3 Gb x8 when using SSD module CCIN 58B2. The sissasraidmgr provides a command line function which may enhance I/O performance for some types of I/O workloads. See table 1 in AIX command line interface to see the complete command string. The command for workload optimization is used to prepare JBOD-only formatted SSDs (512 byte) for this performance tuning. The JBOD workload optimization setting can be returned to its default value by either executing an I/O per second workload optimization command, recycling the power on the PCIe RAID and SSD SAS adapter, or downloading firmware to the SSD. It is recommended that a script be established to ensure the desired JBOD workload optimization parameters are activated whenever one of these actions occurs.

So the setting is volatile and transient.

Oh hey, that's me with the IBM hardware! And I haven't seen that bit of documentation before, my search terms were probably crap. While I have the IBM hardware on hand, I don't have much experience actually using it or finding documentation for it.

The "AIX way" you linked I believe is one of the methods I've tried, though perhaps I should try again with the documentation on hand for that.

Is there any documentation for converting these SSD's in a DS8880 unit? I have a standalone unit on hand.

[catalin7504]

Is there any documentation for converting these SSD's in a DS8880 unit? I have a standalone unit on hand.

I found this pdf: https://www.redbooks.ibm.com/redbooks/pdfs/sg248323.pdf Page 108:

The DS8880 was designed to safely store and retrieve large amounts of data. Redundant Array of Independent Disks (RAID) is an industry-wide method to store data on multiple physical disks to enhance data redundancy. Many variants of RAID are in use today. The DS8880 supports RAID 6, RAID 10, and (on small drives by RPQ only) RAID 5. The DS8880 does not support a non-RAID configuration of disks, which is also known as just a bunch of disks (JBOD).

Based on this, I don't think it's possible using DS8880 unit. I tried with Linux on a Power 7 (I think) with RAID controller 57B3 indicated in the article https://www.ibm.com/docs/en/power8/8247-21L?topic=fcsrc-pcie-sas-raid-card-comparison-1 on Linux and iprconfig, but it failed. I connected the drive using a SAS to 4-SAS ports cable and power from separate power supply. I have no knowledge of AIX, never worked with one, but I'm willing to try if there is a way to download it.

[mr44er]

https://support.huawei.com/enterprise/en/doc/EDOC1100030818/33c5d16a/legacy-commands#EN-US_TOPIC_0230800402

Many VSCs with sg_raw examples, about fast restart and self destruction (i hope this means only locks). The WWN may also take a role with different firmwares.

https://www.ibm.com/docs/en/aix/7.1?topic=subsystem-scdisk-scsi-device-driver The reservations should mean these secure bands 0-15. What caught my eye: Attention: Exercise care when issuing the DKPRES_CLEAR operation. This operation leaves the device unreserved, which could allow a foreign initiator to access the device.

[mr44er]

Thx, that was a good idea.

I somehow suspected, but somehow didn't, that your model is so special in terms of hardware that it goes beyond writelocks and rejections in the event of incorrect microcode. Your model is also much newer than the disks I tried this with. In other words, the newer, the better the security.

If nobody wants them on HDD rescue forums for a good price, then you can be sure that they can't be reformatted/reflashed properly. Then you'll probably have no choice but to get rid of them on second-hand markets somehow, and the price shouldn't matter. But anything is better than throwing them away completely.

[mr44er]

For the ones that will come later and as a reminder for myself with the not so special/older disks.

What blocks the microcode/write-buffer is most likely a persistent reserve "team" (that's the 3rd party) entry with password.

I hope this could help to find a solution: https://www.t10.org/ftp/t10/document.08/08-025r2.pdf https://www.t10.org/ftp/t10/document.07/07-459r0.pdf https://docs.oracle.com/cd/E88353_01/html/E72487/sg-sanitize-8.html

sg_sanitize -ause --fail /dev/sgX ???

Value   Definition
0x00    Reserved
0x01 | exit-failure Exit Failure Mode
0x02 | start-block-erase    Start a Block Erase sanitize operation
0x03 | start-overwrite  Start an Overwrite sanitize operation
0x04 | start-crypto-erase   Start a Crypto Erase sanitize operation

Edit: And there is https://www.t10.org/ftp/t10/document.08/08-019r2.pdf ATA through SCSI download microcode to maybe override reservations?

https://www.t10.org/ftp/t10/document.05/05-003r2.pdf https://www.t10.org/ftp/t10/document.02/02-419r6.pdf