Doesn't work with DSM 7.2.1-69057-3 due to additional checks for Synology SSDs in Storage Manager

[herbingk]

Seems like Synology implemented additional checks for it's SSDs in Storage Manager. The dedupe button is shown after running the script, but if you click it an error will be presented, that it is working with Synology SSDs only. Found the offending function in /var/packages/StorageManager/target/ui/storage_panel.js as follow:

validDedupeDisk(){let e=["SNV3400-400G","SNV3410-400G","SNV3500-400G","SNV3510-400G","SNV3400-800G","SNV3410-800G","SNV3500-800G","SNV3510-800G"].includes(this.model.trim());return!(!this.isSsd||!this.isSynoDrive||e)}};

This is queried later on in this code snippet

if(this.createPool)return this.needSelectRaidGroup?this.selectRaidGroups.every((e=>e.filter((e=>e.hasData)).every((e=>e.validDedupeDisk)))):!!this.selectDisks.length&&!!this.selectDisks.every((e=>e.validDedupeDisk));{let e=this.poolMap[this.$refs.choose_pool.poolId];if(!e)return!1;if(e.disks.filter((e=>this.diskMap[e])).map((e=>this.diskMap[e])).every((e=>e.validDedupeDisk)))return!0}return!1}

Thanks for looking into it

[apkehoe]

Also having this issue. Just updated to 7.2.1-69057-3 on a DS423+ and can enable dedupe on the HDD, not the M2 storage pool.

[007revad]

@herbingk Run https://github.com/007revad/Synology_HDD_db with the -n option. Then DSM shouldn't complain about your drives not being Synology drives.

I'm impressed that you found storage_panel.js and searched it's contents. I formatted a copy of it so I could more easily read it and it was over 40,000 lines! It took me days of trial and error to find what was preventing the Deduplication menu appearing for SATA SSDs.

@apkehoe Run https://github.com/007revad/Synology_HDD_db with the -n option but NOT the -f option. Support disk compatibility needs to enabled for deduplication to work.

Did you migrate those drives from a DS720+?

[herbingk]

@007revad

Update: Tried the modification proposed below regarding the validDedupeDisk() function. Can confirm it is not solving the problem. Tried to disable SSD cache as well, doesn't help also. The only thing that comes to my mind is, that I am using a HA cluster. May that have any impact? On the other hand apkehoe has the same problem with his M.2 SSD storage pool. Do you have any chance to test your script with the current DSM version?

Synology_HDD_db has been run with the -n option and did add my M2. and SATA SSDs to the database already (just confirmed it again) before opening the issue. Actually this was my firsts script to implement to get rid of DSM warning messages and for experimenting with M.2 storage pools. Synology_HDD_db is a life save and very much appreciated!

But as I wrote, the problem seems to be an additional check in Storage Manager's storage_panel.js, that explicitly checks for the Synology SSD models.

The functions is:

validDedupeDisk(){let e=["SNV3400-400G","SNV3410-400G","SNV3500-400G","SNV3510-400G","SNV3400-800G","SNV3410-800G","SNV3500-800G","SNV3510-800G"].includes(this.model.trim());return!(!this.isSsd||!this.isSynoDrive||e)}};

The reference in the checking code is

if(this.createPool)return` this.needSelectRaidGroup?this.selectRaidGroups.every((e=>e.filter((e=>e.hasData)).every((e=>e.validDedupeDisk)))):!!this.selectDisks.length&&!!this.selectDisks.every((e=>e.validDedupeDisk));{let e=this.poolMap[this.$refs.choose_pool.poolId];if(!e)return!1;if(e.disks.filter((e=>this.diskMap[e])).map((e=>this.diskMap[e])).every((e=>e.validDedupeDisk)))return!0}return!1}

I am not that firm with JavaScript but maybe rewriting the function as follows could solve the problem already:

validDedupeDisk(){return(this.isSynoDrive)}};

[apkehoe]

I did migrate the drives from a DS720+ unit.

I tried removing the -f option and it didn't make any difference. I also tried restoring the config and rerunning the script with -n -r, plus a reboot. Didn't stop the error dialog showing.

Task Scheduler has completed a triggered task.

Task: Enable Syno HDD DB Boot
Start time: Sat, 06 Jan 2024 12:15:50 GMT
Stop time: Sat, 06 Jan 2024 12:16:00 GMT
Current status: 0 (Normal)
Standard output/error:
Synology_HDD_db v3.3.73
DS423+ DSM 7.2.1-69057-3 
StorageManager 1.0.0-0017

Using options: -n -r
Running from: /volume1/Data/Synology_HDD_db/syno_hdd_db.sh

HDD/SSD models found: 1
WD181KFGX-68AFPN0,83.00A83

M.2 drive models found: 1
WD_BLACK SN770 2TB,731100WD

No M.2 PCIe cards found

No Expansion Units found

 [0;33mWD181KFGX-68AFPN0 [0m already exists in  [0;36mds423+_host_v7.db [0m
 [0;33mWD181KFGX-68AFPN0 [0m already exists in  [0;36mds720+_host_v7.db [0m
 [0;33mWD181KFGX-68AFPN0 [0m already exists in  [0;36mds720+_host_v7.db.new [0m
 [0;33mWD181KFGX-68AFPN0 [0m already exists in  [0;36mds423+_host_v7.db.new [0m
 [0;33mWD_BLACK SN770 2TB [0m already exists in  [0;36mds423+_host! _v7.db [0m
 [0;33mWD_BLACK SN770 2TB [0m already exists in  [0;36mds720+_host_v7.db [0m
 [0;33mWD_BLACK SN770 2TB [0m already exists in  [0;36mds720+_host_v7.db.new [0m
 [0;33mWD_BLACK SN770 2TB [0m already exists in  [0;36mds423+_host_v7.db.new [0m

Support disk compatibility already enabled.

Support memory compatibility already disabled.

Max memory already set to 18 GB.

NVMe support already enabled.

M.2 volume support already enabled.

Drive db auto updates already disabled.

DSM successfully checked disk compatibility.

You may need to  [0;36mreboot the Synology [0m to see the changes.

From DS3

[007revad]

@herbingk Which Synology NAS model do you have? Are you using RAID Groups?

I leave that section as it is and have no issue on my DS1821+ with DSM 7.2.1-69057 Update 3 running deduplication on no-Synology HDDs, SATA SSD and NVMe drives.

My javascript knowledge is only basic so I asked ChatGPT what that section of code does. ChatGPT is not very good at writing code but at least it does a good job of explaining what some simple code does.

That section returns true if the drive is one of the specified Synology drives AND is an SSD AND is a Synology drive.

The problem with replacing this:

get validDedupeDisk() {
    let e = ["SNV3400-400G", "SNV3410-400G", "SNV3500-400G", "SNV3510-400G", "SNV3400-800G", "SNV3410-800G", "SNV3500-800G", "SNV3510-800G"].includes(this.model.trim());
    return !(!this.isSsd || !this.isSynoDrive || e)
}

with this: get validDedupeDisk(){return(this.isSynoDrive)}};

is that Synology could change it in a future Storage Manager package update when they release new SSD drives, which means I'd constantly be updating the script to match whatever that array of drive modesl changes to.

The easiest solution might be to change:

return !(!this.isSsd || !this.isSynoDrive || e)

to this:

return (this.isSsd)

which would always return true if the drive is an SSD.

[007revad]

@herbingk @apkehoe

Do you want to try this to see if it fixes the issue: https://github.com/007revad/Synology_enable_Deduplication/blob/test/test.sh

It replaces this:

return!(!this.isSsd||!this.isSynoDrive||e)

with tihs:

return(this.isSsd)

[apkehoe]

I tried the script and it changed the js file, but still didn't work. I restarted nginx but I didn't reboot the NAS.

[herbingk]

@herbingk @apkehoe

Do you want to try this to see if it fixes the issue: https://github.com/007revad/Synology_enable_Deduplication/blob/test/test.sh

It replaces this:

return!(!this.isSsd||!this.isSynoDrive||e)

with tihs:

return(this.isSsd)

I executed syno_enable_dedupe.sh first and then the test.sh script. The error message still pops up. So the reason must be something else unfortunately.

[herbingk]

@herbingk Which Synology NAS model do you have? Are you using RAID Groups?

I use 2x DS1621+ in a Synology HA cluster. There is one volume in one storage pool with 6x SATA SSD RAID 5 and 2x M2 SSD cache RAID 1. I tried to disable the SSD cache and rebooted without changing the behavior.

[007revad]

I executed syno_enable_dedupe.sh first and then the test.sh script. The error message still pops up. So the reason must be something else unfortunately.

I have found that when I edit or restore /var/packages/StorageManager/target/ui/storage_panel.js it can take 20 or 30 minutes before the change shows in Storage Manager. And I also need to refresh the DSM window/tab to see the changes.

DSM seems to read storage_panel.js into memory on a schedule. But I haven't yet found a way to force DSM to read it right away (apart from rebooting).

[007revad]

I tried to disable the SSD cache and rebooted without changing the behavior.

So even after a reboot that change made no difference. That's disappointing.

[herbingk]

Interesting side note: The menu for deduplication turns up on my storage manager even without modifying storage_panel.js, the change to both synoinfo.conf files is sufficient for that to appear already.

[007revad]

The menu for deduplication turns up on my storage manager even without modifying storage_panel.js, the change to both synoinfo.conf files is sufficient for that to appear already.

I noticed that too. For me the default storage_panel.js only shows the deduplication menu for internal NVMe drives. I had to edit storage_panel.js to get the deduplication menu to appear for SATA SSDs... which coincidentally also made the deduplication menu appear for HDDs. I also had to edit storage_panel.js to get the deduplication menu to appear for NVMe drives in a PCIe adaptor card.

[herbingk]

The menu for deduplication turns up on my storage manager even without modifying storage_panel.js, the change to both synoinfo.conf files is sufficient for that to appear already.

I noticed that too. For me the default storage_panel.js only shows the deduplication menu for internal NVMe drives. I had to edit storage_panel.js to get the deduplication menu to appear for SATA SSDs... which coincidentally also made the deduplication menu appear for HDDs. I also had to edit storage_panel.js to get the deduplication menu to appear for NVMe drives in a PCIe adaptor card.

Strange: For me it shows up for my SATA SSDs immediately and without patching storage_panel.js

[007revad]

Strange: For me it shows up for my SATA SSDs immediately and without patching storage_panel.js

Now that you mention that I remember it did (does) for me too, but it wasn't for someone with a DS1618+ in issue #25

[chackl1990]

Also having this issue. DS923+ DSM7.2.1-69057 Update 3 with SATA SSDs Seeing the option, but getting an error that dedup on ssds is only for synology-ssds linke in https://github.com/007revad/Synology_enable_Deduplication/issues/43#issuecomment-1879576728

also executed the script https://github.com/007revad/Synology_enable_Deduplication/blob/test/test.sh and changed storage_panel.js but after waiting about 1 hour i am seeing the same problem.

[herbingk]

@007revad Solution found!

/usr/lib/libhwcontrol.so.1 still needs to be patched with your script, reboot done, error is gone and deduplication panel show up. I tried by commenting out your checks for the new Storage Manager and running the script again.

Maybe you have a patched libhwcontrol.so.1 on your system from prior script development activities?

Editing storage_panel.js wasn't needed for me and I skipped it for the test. I would suggest to adapt the script to always patch libhwcontrol.so.1 when it is present and passes your plausibility checks, no matter what DSM 7 version is running.

I would also suggest to make editing storage_panel.js optional with an extra param, as this is needed for using deduplication on HDDs only. I doubt that deduplication on HDDs is useful as it will defragment files on disk, thereby increases the seek rate and slows down file access on read and write.

[chackl1990]

@007revad Solution found!

/usr/lib/libhwcontrol.so.1 still needs to be patched with your script, reboot done, error is gone and deduplication panel show up. I tried by commenting out your checks for the new Storage Manager and running the script again.

Maybe you have a patched libhwcontrol.so.1 on your system from prior script development activities?

Editing storage_panel.js wasn't needed for me and I skipped it for the test. I would suggest to adapt the script to always patch libhwcontrol.so.1 when it is present and passes your plausibility checks, no matter what DSM 7 version is running.

I would also suggest to make editing storage_panel.js optional with an extra param, as this is needed for using deduplication on HDDs only. I doubt that deduplication on HDDs is useful as it will defragment files on disk, thereby increases the seek rate and slows down file access on read and write.

forcing the script to patch libhwcontrol.so.1 works for me and shows the dialog. Thanks! DS923+ DSM7.2.1-69057 Update 3 with SATA SSDs

[007revad]

Maybe you have a patched libhwcontrol.so.1 on your system from prior script development activities?

Yes, but I didn't realise it was still edited. Previous versions of syno_enable_dedupe edited libhwcontrol.so.1 and my syno_enable_m2_volume script also does the same edit in libhwcontrol.so.1

Editing storage_panel.js wasn't needed for me and I skipped it for the test. I would suggest to adapt the script to always patch libhwcontrol.so.1 when it is present and passes your plausibility checks, no matter what DSM 7 version is running.

Editing storage_panel.js was added to solve issue #25 and the edit enabling deduplication for HDDs was an unintentional side effect of that edit.

I would also suggest to make editing storage_panel.js optional with an extra param, as this is needed for using deduplication on HDDs only. I doubt that deduplication on HDDs is useful as it will defragment files on disk, thereby increases the seek rate and slows down file access on read and write.

Will do.

[007revad]

Update version available here: https://github.com/007revad/Synology_enable_Deduplication/releases/tag/v1.3.19

[herbingk]

Update version available here: https://github.com/007revad/Synology_enable_Deduplication/releases/tag/v1.3.19

Thanks very much! Works like a charm now. Thanks again for providing these useful scripts. Very much appreciated!