search

007revad / Synology_enable_Deduplication

Enable deduplication with non-Synology SSDs and unsupported NAS models
MIT License
70 stars 4 forks source link

Deduplication not available on DS3622xs+ #27

Closed jdpdata closed 8 months ago

jdpdata commented 9 months ago

I need help figuring out why data deduplication is not enable. I ran all the steps detailed in your instructions.

image

nas4

007revad commented 9 months ago

So your setup meets these 2 requirements:

  1. Deduplication only works on SSD volumes that are formatted in Btrfs.
  2. The SSD volume needs usage detail analysis enabled.

It could be a DSM 7.2 Update 3 issue, a DS3622xs+ issue or a combination of both. I have heard that my syno_enable_m2_volume script does not work for Xpenology users with DS3622+ using DSM 7.2 Update 3.

This script does the same edit to the same file in DSM as the syno_enable_m2_volume script does so this script would have same issue with DS3622+ using DSM 7.2 Update 3.

I know this script worked for a DS3622+ with DSM 7.2-64561 beta. But I've had no feedback on the DSM 7.2-64750 versions.

I've only ever tried it with RAID 1, but deduplcation works for SATA SSDs too so RAID 10 shouldn't be an issue.

jdpdata commented 9 months ago

I'm using M.2 nvme SSD. (https://www.amazon.com/dp/B07Z4J96C9?ref=ppx_yo2ov_dt_b_product_details&th=1) Are these nvme supported? Or do you mean SATA SSD only? I've verified Usage Details is enabled. I used syno_create_m2_volume.sh to create my nvme volume. Is that script different from syno_enable_me_volume.sh?  I'm on latest 7.2-64570 U3 using AuxArc loader. When I built Christian's loader, I selected some of his extension for DSM (nvmecahce, nvmestorage, drivedbpatch). I assumed these are your scripts that he has included in his loader. I'm willing to do beta testing for you. Just let me know what I need to do.

On Tuesday, September 12, 2023 at 04:25:48 AM PDT, Dave Russell ***@***.***> wrote:

So your setup meets these 2 requirements:

It could be a DSM 7.2 Update 3 issue, a DS3622xs+ issue or a combination of both. I have heard that my syno_enable_m2_volume script does not work for Xpenology users with DS3622+ using DSM 7.2 Update 3.

This script does the same edit to the same file in DSM as the syno_enable_m2_volume script does so this script would have same issue with DS3622+ using DSM 7.2 Update 3.

I know this script worked for a DS3622+ with DSM 7.2-64561 beta. But I've had no feedback on the DSM 7.2-64750 versions.

I've only ever tried it with RAID 1, but deduplcation works for SATA SSDs too so RAID 10 shouldn't be an issue.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

007revad commented 9 months ago

Deduplication works on SATA SSDs, M.2 SSDs (SATA and NVMe) as well as SAS SSDs.

Both nvmestorage and drivedbpatch are probably my syno_enable_me_volume and syno_hdd_db scripts. I haven't looked at AuXArc but I know tcrp-addons uses those 2 scripts of mine.

syno_enable_m2_volume and syno_enable_depupe are 99% the same script. The only differences are:

Syno_enable_M2_volume:

Syno_enable_dedupe:

Do the following commands both return "yes"

synogetkeyvalue /etc/synoinfo.conf support_btrfs_dedupe
synogetkeyvalue /etc.defaults/synoinfo.conf support_btrfs_dedupe
007revad commented 9 months ago

This did previously work with a DS3622xs+ using:

Nobody has reported if it works or not with 7.2-64570 Update 1 or Update 2.

Are you able to try:

My syno_dsm_restore script will let you install DSM 7.2-64570 (with Update 1) from the DSM webui. https://github.com/007revad/Synology_DSM_reinstall

You'll need to do all the steps to prevent DSM updating itself to Update 3 during the DSM installation.

jdpdata commented 9 months ago

Both commands returned nothing: I can try to roll-back to 64570 U1. Is that next logical step? That's not going to cause any problems with my current setup?

image

007revad commented 9 months ago

Run the following 2 commands:

sudo -i synosetkeyvalue /etc.defaults/synoinfo.conf support_btrfs_dedupe yes
sudo -i synosetkeyvalue /etc/synoinfo.conf support_btrfs_dedupe yes

Then check if the deduplication option is available in storage manager.

jdpdata commented 9 months ago

You're a genius! That worked!! I very happy to have data deduplication because I use this volume to store alot of database files that could be dupes and taking up valuable space.

On Tuesday, September 12, 2023 at 09:42:04 PM PDT, Dave Russell ***@***.***> wrote:

Run the following 2 commands: sudo -i synosetkeyvalue /etc.defaults/synoinfo.conf support_btrfs_dedupe yes sudo -i synosetkeyvalue /etc/synoinfo.conf support_btrfs_dedupe yes

Then check if the deduplication option is available in storage manager.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

007revad commented 9 months ago

I'm curious why the support_btrfs_dedupe wasn't already set to yes. The script only edits /etc.defaults/synoinfo.conf and DSM that normally copies that setting to /etc/synoinfo.conf Maybe with DSM 7.2 update 3 I need to set support_btrfs_dedupe to yes in both files.

I'm also curious if the setting survives a reboot in DSM 7.2 update 3.

jdpdata commented 9 months ago

I'll do a shutdown and restart. Will let you know if it survives.

jdpdata commented 9 months ago

It's survives after shutdown and a restart. Fantastic! Now I'm going to try this script on my other DS920+. I got a 2TB folder full of duplicated photos. You think it will work?

007revad commented 9 months ago

If the DS920+ has 16GB or more of memory it should work.

007revad commented 9 months ago

PS Synology says: "Do not run data deduplication if the selected volume already contains snapshots of shared folders, Thin Provisioned LUNs, or virtual machines (VMs)"

jdpdata commented 9 months ago

My DS920+ has 32GB. It didn't come up with Dedupe setting on Volume settings after a reboot. I ran both of those commands again with 'yes' at the end. Rebooting..to see if comes up

jdpdata commented 9 months ago

PS Synology says: "Do not run data deduplication if the selected volume already contains snapshots of shared folders, Thin Provisioned LUNs, or virtual machines (VMs)"

I do have snapshots of shared folders. Guess I'll have to turn off data deduplication.

DS920+ did not come up with dedupe settings even after multiple attempts. Oh well, I guess it's not a supported model. I do have a bad HDD on volume1. Maybe try it again when WD ships me replacement.

007revad commented 9 months ago

I'm a bit confused by Synology's "How do I configure Snapshot Replication while ensuring effective data deduplication?"

I think disabling snapshots is only a suggestion for those who are trying to save space.

007revad commented 9 months ago

Do you see your NVMe drives in storage manager like in this screenshot?

storage_mgr_pcie_m2_small2

jdpdata commented 9 months ago

nope I don't see that

On Wednesday, September 13, 2023 at 04:05:42 PM PDT, Dave Russell ***@***.***> wrote:

Do you see your NVMe drives in storage manager like in this screenshot?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

007revad commented 8 months ago

@jdpdata

I just noticed that this issue is still open. Did we resolve all your questions/issues?

jdpdata commented 8 months ago

Hi Dave, I don't think Deduplication is working on my Xpenology built DS3622+. I had full volume encryption at the time, which I've since removed due to another issue. I think full volume encryption and deduplication don't work well together.  I went with another route for deduping. GitHub - qarmin/czkawka: Multi functional app to find duplicates, empty folders, similar images etc.

|

GitHub - qarmin/czkawka: Multi functional app to find duplicates, empty ...

Multi functional app to find duplicates, empty folders, similar images etc. - GitHub - qarmin/czkawka: Multi fun... |

|

|

On Monday, October 23, 2023 at 09:42:39 PM PDT, Dave Russell ***@***.***> wrote:

@jdpdata

I just noticed that this issue is still open. Did we resolve all your questions/issues?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

007revad commented 8 months ago

GitHub - qarmin/czkawka

I've actually seen that before. I think someone on reddit mentioned it once.

007revad commented 6 months ago

@jdpdata

I've released a new version, v1.2.14, that works in DSM 7.2.1 on models with one of following CPU arch: V1000, R1000, Geminilake, Broadwellnkv2, Broadwellnk, Broadwell, Purley and Epyc7002. So it should work on a DS3622xs+ now.

https://github.com/007revad/Synology_enable_Deduplication#works-in-dsm-721-for-the-following-models