DawidPotgieter / Dynamics-CRM-Binary-Storage-Options

A Dynamics CRM plugin to transparently store attachment binaries outside CRM database. Azure Blob or File storage for instance.
GNU Affero General Public License v3.0
34 stars 15 forks source link

Only small storage gained - Dynamics 365 (Online) #22

Open klnv000 opened 4 years ago

klnv000 commented 4 years ago

After implementing the solution version 4. I had verified that the migration tool run completely by check that the "body" have been updated to "IA==". The storage size used does goes down referring to Dynamics Capacity Report but only by a small amount. We have only recovered about 17 GB out of 46 GB of storage.

I would have expected more storage space to be recovered seeing all attachments should be in Azure. Is this normal?

image

DawidPotgieter commented 4 years ago

Hi there,

In general, you should expect to save about 1.3 times the amount of data moved to your storage account, because the data stored in the CRM database is base64 encoded.

From the screenshot, it seems to imply that you have 26Gb of data in your blob storage account (but I can also see that there are other things going into your storage account). The graph on the right seems to be the data throughput, so I can't quite tell exactly how much data you have in your attachment and annotationbase containers.

Can you check (probably with the StorageExplorer) what the actual size is of only those two containers? I guessing Dynamics has some kind of optimization jobs that run automatically to compact databases, you might just want to check that too.

Let me know please, it's an interesting observation.

Cheers.

klnv000 commented 4 years ago

Hiya, Thanks for the reply.

There are around 3 GB of email attachment and 18 GB of annotation migrated to azure based on azure storage explorer. I verified that number in Azure Portal metric as well (refer below image). 2020-07-13_14-47-18

If you compared to the CRM Capacity Report image from my previous post, we can see that annotation was migrated nicely basically dropped from 20 GB to near 0 after migration, but attachment did not behave the same.

I would understand that not every CRM system has the same data. But i wonder if in your own environment:

  1. How much does storage size did it recovered / migrated for attachment?
  2. Has this (issue?) happened before(earlier/current version)?

Any comment as to why would be appreciated.

DawidPotgieter commented 4 years ago

So,

I see what you're saying. You would have expected the yellow graph to drop down to almost 0 at about the same time as the red one, which I agree with. Looking at what we have and what you should see, the attachments table should be tiny storage wise.

  1. As I mentioned before, you should save around 1.3 times the amount of UNCOMPRESSED/UNENCRYPTED bytes moved, because the content is base64 encoded in the CRM database. If you have looked at the body of all the attachments and they're all IA==, it means the space has been cleared (but it may still be allocated in the db).
  2. I've never seen or heard of this issue, however, I don't actually know how/when the Dynamics 365 database optimization tasks run. For our on-prem database, we shrunk it after migration to release the empty space, I just don't know how the online version's databases are managed.

From the graphs, it looks like it took about 10 days after migration for the annotations size to drop, is that correct?