nachopants commented 1 year ago

Firmware upgrade failure, with segmented download variable it says 'successfully upgraded to SN03' (which is current firmware)

Block size failure as well.

Firmware upgrade from SN03 to SN04

Linux nachoServ 4.4.302+ #64570 SMP Thu Jul 20 00:07:27 CST 2023 x86_64 GNU/Linux synology_broadwell_3617xs

I have tried with various versions of SeaChest and SeaChest lite.

It also fails to change block size from 512 to 4096.

Consistent error across 8 x 16tb exos 18's

ash-4.4# ./openSeaChest_Firmware -d /dev/sg6 --downloadFW EvansBPExosX18SATA-STD-512E-SN04.LOD --downloadMode segmented

openSeaChest_Firmware - openSeaChest drive utilities - NVMe Enabled Copyright (c) 2014-2023 Seagate Technology LLC and/or its Affiliates, All Rights Reserved openSeaChest_Firmware Version: 3.4.0-4_1_1 X86_64 Build Date: Mar 27 2023 Today: Wed Sep 20 15:43:32 2023 User: root

/dev/sg6 - ST16000NM000J-2TW103 - ZR70SWR4 - SN03 - ATA ...... Firmware Download successful Firmware Download time (ms): 162.77 Average time/segment (ms): 1.64 Activate Time (ms): 90.97 New firmware version is SN03

https://pastebin.com/wSGV7krx

Swiss3003 commented 1 year ago

Nachopants, The first thing is to power cycle the drive. Sounds like the it did a differed download to the drive and the firmware will not update until the drive is power cycled. IF, I remember correctly "New firmware version is SN03" reports what the device reports after the download. If it is a differed download, well it will not update until it updates after the next power cycle.

after the power cycle check the firmware version by doing a -i on that device.

If it has not updated, then we need to check to see if the firmware is correct for your drive. Quick look it should be correct. But if the firmware does not match the drive will reject the firmware on the next power cycle.

For the format: run a --showSupportedFormats and Please send that output.

next run the --setSectorSize 4096 --confirmThis-may-erase-data -v3 and save all the output and send it to us. it will help with the debug.

Tim Gilmer Staff Engineer Field Diags Office: (720)-684-2624 Seagate Technology [cid:40a82c92-95f2-4789-901f-3bf0712affa5]

Seagate Internal

From: nachopants @.> Sent: Tuesday, September 19, 2023 11:47 PM To: Seagate/openSeaChest @.> Cc: Subscribed @.***> Subject: [Seagate/openSeaChest] Firmware update failing - Firmware Download Failed - Download Microcode returning: ABORTED (Issue #121)

This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.

Firmware upgrade failure, with segmented download variable it says 'successfully upgraded to SN03' (which is current firmware)

Block size failure as well.

Firmware upgrade from SN03 to SN04

Linux nachoServ 4.4.302+ #64570 SMP Thu Jul 20 00:07:27 CST 2023 x86_64 GNU/Linux synology_broadwell_3617xs

I have tried with various versions of SeaChest and SeaChest lite.

It also fails to change block size from 512 to 4096.

Consistent error across 8 x 16tb exos 18's

ash-4.4# ./openSeaChest_Firmware -d /dev/sg6 --downloadFW EvansBPExosX18SATA-STD-512E-SN04.LOD --downloadMode segmented openSeaChest_Firmware - openSeaChest drive utilities - NVMe Enabled Copyright (c) 2014-2023 Seagate Technology LLC and/or its Affiliates, All Rights Reserved openSeaChest_Firmware Version: 3.4.0-4_1_1 X86_64 Build Date: Mar 27 2023 Today: Wed Sep 20 15:43:32 2023 User: root

/dev/sg6 - ST16000NM000J-2TW103 - ZR70SWR4 - SN03 - ATA ...... Firmware Download successful Firmware Download time (ms): 162.77 Average time/segment (ms): 1.64 Activate Time (ms): 90.97 New firmware version is SN03

https://pastebin.com/wSGV7krx

— Reply to this email directly, view it on GitHubhttps://github.com/Seagate/openSeaChest/issues/121, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIA3L2HULBHWKLRMAZXS2GTX3J7NPANCNFSM6AAAAAA47KSSZU. You are receiving this because you are subscribed to this thread.Message ID: @.***>

nachopants commented 1 year ago

Hi Tim,

First of all thanks for your time in assisting with this. I must say all indications to a layman look as though the firmware is incorrect or being blocked despite the fact I've downloaded it from the Seagate website having tried a range of my SN03 -> SN04 serial numbers. I've tried with the bundled SeaChest binary, the latest SeaChest binary from Seagate and now the openSeaChest suite so far.

I have given up on the firmware for now, am hopeful you can point me in the right direction given the above.

Firstly, I'd like to make clear this is the current suite of drives

https://pastebin.com/ECertmi2

My focus for this exercise is /dev/sg2 and /dev/sg3

In attempting to upgrade the firmware once again, it was not successful, and yes have power cycled the drive etc.

https://pastebin.com/VXCDU06b

I decided to revisit the firmware later and for now push /dev/sg2 and /dev/sg3 to 4096 from 512.

/dev/sg2 appeared to be all good https://pastebin.com/KR9FY78G

/dev/sg3 does not... look good at all https://pastebin.com/iMFbCWEG

It now seemingly looks as though /dev/sg3 is unreadable entirely https://pastebin.com/4mMKbpm5

So now our mission is to recover /dev/sg3 so both sg2 & sg3 are stable at 4096.

Then re-visit why this firmware update isn't working...

nachopants commented 1 year ago

-rwx------+ 1 synology users 384 Oct 4 2022 ENBP-SN04.CFS -rwx------+ 1 synology users 2.6M Oct 3 2022 EvansBPExosX18SATA-STD-512E-SN04.LOD -rwx------+ 1 synology users 604K Mar 28 08:16 openSeaChest_Basics -rwx------+ 1 synology users 548K Mar 28 08:16 openSeaChest_Configure -rwx------+ 1 synology users 601K Mar 28 08:16 openSeaChest_Erase -rwx------+ 1 synology users 505K Mar 28 08:16 openSeaChest_Firmware -rwx------+ 1 synology users 578K Mar 28 08:16 openSeaChest_Format -rwx------+ 1 synology users 553K Mar 28 08:16 openSeaChest_GenericTests -rwx------+ 1 synology users 630K Mar 28 08:16 openSeaChest_Info -rwx------+ 1 synology users 531K Mar 28 08:16 openSeaChest_Logs -rwx------+ 1 synology users 556K Mar 28 08:16 openSeaChest_NVMe -rwx------+ 1 synology users 598K Mar 28 08:16 openSeaChest_PassthroughTest -rwx------+ 1 synology users 557K Mar 28 08:16 openSeaChest_PowerControl -rwx------+ 1 synology users 519K Mar 28 08:16 openSeaChest_Security -rwx------+ 1 synology users 780K Mar 28 08:16 openSeaChest_SMART -rwx------+ 1 synology users 489K Mar 28 08:16 openSeaChest_ZBD

nachopants commented 1 year ago

https://pastebin.com/bwGKgvie

Also this outlines the successfully modified drive and the failed drive.

Unsure on next steps!

DebabrataSTX commented 1 year ago

@nachopants I could not access to the link you have provided. It will be much better if you post it somewhere more accessible or attach it here. Also if you add "-v 4" to the command. This will provide us more info that will be helpful to debug the issue.

Swiss3003 commented 1 year ago

Nochopants, Let's start with a list of the serial numbers of the drives. I'll get someone to look / hunt for firmware to upgrade to.

Second, Looks like our firewall will not let me (IT issue) see any of the shares you did. Can you copy the text into the email. painful I know, but that way I know the information will come through and not get blocked by IT.

Also can you give me a breakdown on the system. OS, cards, RAID anything else that might be in the system that we should know about. Seems like the OS passthrough is working for you, because one drive did update. But I just would like to know what we are dealing with. Also are all the drive sata or are some of them sas?

I'll do some new builds on SeaChest to update your tools. Once I know the OS. I'll get you some new builds to make sure you are up to date. Erase, format, firmware. Do you need any other tools?

Format changes: here are a few things to look at when changing the sector sizes. Background tasks - If a drive has or needs to perform background tasks the change and the format can take a long time for the big drives. Let the tool finish the task. Any hard reset or power cycle can brick the drive during this process. MBR - sometimes I have found that a MBR or a partition on the drive will cause the drive to not change the sector size.

I found that erasing the drive, or writing 00's to the MBR ( LBA 0 -1000) and to MaxLBA (maxlba-1000) helps
```
  SeaChest_Basics -d /dev/sg<#> --displayLBA 1000
```
SeaChest_Basics -d /dev/sg<#> --overwrite 0 SeaChest_Basics -d /dev/sg<#> --overwrite 1000 --overwriteRange 2000

[image]

Tim Gilmer Staff Engineer Field Diags Office: (720)-684-2624 Seagate Technology [cid:fc606380-b1c3-46f4-9add-c72297284824]

Seagate Internal

From: nachopants @.> Sent: Wednesday, September 20, 2023 6:09 PM To: Seagate/openSeaChest @.> Cc: Tim Gilmer @.>; Comment @.> Subject: Re: [Seagate/openSeaChest] Firmware update failing - Firmware Download Failed - Download Microcode returning: ABORTED (Issue #121)

This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.

https://pastebin.com/bwGKgvie

Also this outlines the successfully modified drive and the failed drive.

Unsure on next steps!

— Reply to this email directly, view it on GitHubhttps://github.com/Seagate/openSeaChest/issues/121#issuecomment-1728578029, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIA3L2CTPA6KZU7NOIHE4P3X3OAS7ANCNFSM6AAAAAA47KSSZU. You are receiving this because you commented.Message ID: @.***>

nachopants commented 1 year ago

Hi team,

As a first step here are all of the debug links above in order...

https://seagate-debug.s3.ap-southeast-2.amazonaws.com/debug1.txt https://seagate-debug.s3.ap-southeast-2.amazonaws.com/debug2.txt https://seagate-debug.s3.ap-southeast-2.amazonaws.com/debug3.txt https://seagate-debug.s3.ap-southeast-2.amazonaws.com/debug4.txt https://seagate-debug.s3.ap-southeast-2.amazonaws.com/debug5.txt https://seagate-debug.s3.ap-southeast-2.amazonaws.com/debug6.txt https://seagate-debug.s3.ap-southeast-2.amazonaws.com/debug7.txt

I hope you can open these? These reveal all the information you're after, debug logs, serial numbers, etc

The device itself is a Synology DS3617xs

ash-4.4# uname -a

Linux nachoServ 4.4.302+ #64570 SMP Thu Jul 20 00:07:27 CST 2023 x86_64 GNU/Linux synology_broadwell_3617xs

Yes one drive did upgrade without issue, the other drive, I am guessing about 'halfway through' the sector size change all of a sudden said 'device not found' and started throwing errors. It has not recovered. It's simply blank on all columns in the scan as shown by the debug texts above

I am using the latest 'release' from the openSeaChest github repo for now.

The main issue is the failure of the drive at the bottom of https://seagate-debug.s3.ap-southeast-2.amazonaws.com/debug5.txt

nachopants commented 1 year ago

> ash-4.4# ./openSeaChest_Basics -s
> ==========================================================================================
>  openSeaChest_Basics - openSeaChest drive utilities - NVMe Enabled
>  Copyright (c) 2014-2023 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
>  openSeaChest_Basics Version: 3.5.2-4_1_1 X86_64
>  Build Date: Mar 27 2023
>  Today: Fri Sep 22 09:27:16 2023        User: root
> ==========================================================================================
> Vendor   Handle       Model Number            Serial Number          FwRev     
> ATA      /dev/sg0     ST3000VN007-2E4166      Z6A01TKZ               SC60      
> ATA      /dev/sg1     ST3000VN007-2AH16M      ZDH6DN24               SC60      
> ATA      /dev/sg2     ST3000VN007-2E4166      Z6A01CHP               SC60      
> ATA      /dev/sg3     ST3000VN007-2E4166      Z6A01RZJ               SC60      
> ATA      /dev/sg4     ST16000NM000J-2TW103    ZR60EBCQ               SN03      
> ATA      /dev/sg5     ST16000NM000J-2TW103    ZR60EBCH               SN03      
> ATA      /dev/sg6     ST16000NM000J-2TW103    ZR70SWR4               SN03      
>          /dev/sg7                                                              
> ATA      /dev/sg8     ST16000NM000J-2TW103    ZR60595N               SN03      
> ATA      /dev/sg9     ST16000NM000J-2TW103    ZRS0J0P8               SN03      
> ATA      /dev/sg10    ST16000NM000J-2TW103    ZR505EW1               SN03      
> ATA      /dev/sg11    ST16000NM000J-2TW103    ZR60595Q               SN03      
> MARVELL  /dev/sg12    Virtual Device          V Device               1.00      
> Synology /dev/sg13    DiskStation             615609077010           PMAP      
> 
> ash-4.4# ./openSeaChest_Basics -d /dev/sg7 --displayLBA 1000
> ==========================================================================================
>  openSeaChest_Basics - openSeaChest drive utilities - NVMe Enabled
>  Copyright (c) 2014-2023 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
>  openSeaChest_Basics Version: 3.5.2-4_1_1 X86_64
>  Build Date: Mar 27 2023
>  Today: Fri Sep 22 09:27:19 2023        User: root
> ==========================================================================================
> 
> /dev/sg7 -  -  -  - SCSI
> Could not allocate memory to read LBA.: No such device or address
> 
> ash-4.4# ./openSeaChest_Basics -d /dev/sg7 --overwrite 0 --confirm this-will-erase-data
> ==========================================================================================
>  openSeaChest_Basics - openSeaChest drive utilities - NVMe Enabled
>  Copyright (c) 2014-2023 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
>  openSeaChest_Basics Version: 3.5.2-4_1_1 X86_64
>  Build Date: Mar 27 2023
>  Today: Fri Sep 22 09:27:45 2023        User: root
> ==========================================================================================
> 
> /dev/sg7 -  -  -  - SCSI
> Floating point exception (core dumped)
> 
> ash-4.4# ./openSeaChest_Basics -d /dev/sg7 --overwrite 1000 --overwriteRange 2000 --confirm this-will-erase-data
> ==========================================================================================
>  openSeaChest_Basics - openSeaChest drive utilities - NVMe Enabled
>  Copyright (c) 2014-2023 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
>  openSeaChest_Basics Version: 3.5.2-4_1_1 X86_64
>  Build Date: Mar 27 2023
>  Today: Fri Sep 22 09:28:07 2023        User: root
> ==========================================================================================
> 
> /dev/sg7 -  -  -  - SCSI
> Floating point exception (core dumped)

Swiss3003 commented 1 year ago

So, I have verified that the *.LOD file that you have for updating the firmware is the correct one and that it will update on a similar drive. All 7 ST16000NM000J-2TW103 drives should be able to update to the SN04 code. I'll look into some more options to try in order to get the drive to upgrade to the new firmware.

Also, We are able to see the debug output now. We see that the devices are command aborting the set sector configuration ext command. Right now, it's hard to say what is going on. But it might be the drive stopping us from changing the sector size. Now we have to figure out why it might stop us. We will have to start looking at the identify information you sent us and see if something got disabled on the drive. This will take some time to figure out. I have a few people working on it now. I'll talk to two other guys on Monday when they come into the office.

Tim Gilmer Staff Engineer Field Diags Office: (720)-684-2624 Seagate Technology [cid:be4800a1-0b2c-4287-9476-821c544c610c]

Seagate Internal

From: nachopants @.> Sent: Thursday, September 21, 2023 5:28 PM To: Seagate/openSeaChest @.> Cc: Tim Gilmer @.>; Comment @.> Subject: Re: [Seagate/openSeaChest] Firmware update failing - Firmware Download Failed - Download Microcode returning: ABORTED (Issue #121)

This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.

ash-4.4# ./openSeaChest_Basics -s openSeaChest_Basics - openSeaChest drive utilities - NVMe Enabled Copyright (c) 2014-2023 Seagate Technology LLC and/or its Affiliates, All Rights Reserved openSeaChest_Basics Version: 3.5.2-4_1_1 X86_64 Build Date: Mar 27 2023 Today: Fri Sep 22 09:27:16 2023 User: root

Vendor Handle Model Number Serial Number FwRev ATA /dev/sg0 ST3000VN007-2E4166 Z6A01TKZ SC60 ATA /dev/sg1 ST3000VN007-2AH16M ZDH6DN24 SC60 ATA /dev/sg2 ST3000VN007-2E4166 Z6A01CHP SC60 ATA /dev/sg3 ST3000VN007-2E4166 Z6A01RZJ SC60 ATA /dev/sg4 ST16000NM000J-2TW103 ZR60EBCQ SN03 ATA /dev/sg5 ST16000NM000J-2TW103 ZR60EBCH SN03 ATA /dev/sg6 ST16000NM000J-2TW103 ZR70SWR4 SN03 /dev/sg7 ATA /dev/sg8 ST16000NM000J-2TW103 ZR60595N SN03 ATA /dev/sg9 ST16000NM000J-2TW103 ZRS0J0P8 SN03 ATA /dev/sg10 ST16000NM000J-2TW103 ZR505EW1 SN03 ATA /dev/sg11 ST16000NM000J-2TW103 ZR60595Q SN03 MARVELL /dev/sg12 Virtual Device V Device 1.00 Synology /dev/sg13 DiskStation 615609077010 PMAP

ash-4.4# ./openSeaChest_Basics -d /dev/sg7 --displayLBA 1000 openSeaChest_Basics - openSeaChest drive utilities - NVMe Enabled Copyright (c) 2014-2023 Seagate Technology LLC and/or its Affiliates, All Rights Reserved openSeaChest_Basics Version: 3.5.2-4_1_1 X86_64 Build Date: Mar 27 2023 Today: Fri Sep 22 09:27:19 2023 User: root

/dev/sg7 - - - - SCSI Could not allocate memory to read LBA.: No such device or address

ash-4.4# ./openSeaChest_Basics -d /dev/sg7 --overwrite 0 --confirm this-will-erase-data openSeaChest_Basics - openSeaChest drive utilities - NVMe Enabled Copyright (c) 2014-2023 Seagate Technology LLC and/or its Affiliates, All Rights Reserved openSeaChest_Basics Version: 3.5.2-4_1_1 X86_64 Build Date: Mar 27 2023 Today: Fri Sep 22 09:27:45 2023 User: root

/dev/sg7 - - - - SCSI Floating point exception (core dumped)

ash-4.4# ./openSeaChest_Basics -d /dev/sg7 --overwrite 1000 --overwriteRange 2000 --confirm this-will-erase-data openSeaChest_Basics - openSeaChest drive utilities - NVMe Enabled Copyright (c) 2014-2023 Seagate Technology LLC and/or its Affiliates, All Rights Reserved openSeaChest_Basics Version: 3.5.2-4_1_1 X86_64 Build Date: Mar 27 2023 Today: Fri Sep 22 09:28:07 2023 User: root

/dev/sg7 - - - - SCSI Floating point exception (core dumped)

— Reply to this email directly, view it on GitHubhttps://github.com/Seagate/openSeaChest/issues/121#issuecomment-1730485627, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIA3L2DLK3ZVDGVRW7RSILLX3TETNANCNFSM6AAAAAA47KSSZU. You are receiving this because you commented.Message ID: @.***>

Swiss3003 commented 1 year ago

Core dump failures - wow that's not good. First time I've seen that on Seachest basic. I'll get someone to look into that.

Tim Gilmer Staff Engineer Field Diags Office: (720)-684-2624 Seagate Technology [cid:dd1164a5-ee2a-4b61-ab0c-12b0a732daee]

Seagate Internal

From: Tim Gilmer @.> Sent: Friday, September 22, 2023 1:52 PM To: Seagate/openSeaChest @.>; Seagate/openSeaChest @.> Cc: Comment @.> Subject: Re: [Seagate/openSeaChest] Firmware update failing - Firmware Download Failed - Download Microcode returning: ABORTED (Issue #121)