Performance - How many Users do you have?

[jayrodmcneil]

Hey XRM Portals Community,

Just trying to gauge a basic understanding of the number of users of your portal supports. Our application is generally a 50/50 mix of read and write requests, nothing too fancy, mostly simple enough entity lists and forms.

We find that in our load balanced environment, once we get up around 100 users per portal server, the CPUs max out and performance drops miserably to about 30 seconds per page load.

The bottleneck for us doesn't appear to be CRM, rather the cpu on the portal servers, not sure the spec, but pretty beefy (all On-Premise).

I'm just curious if this is normal or if you guys get/expect far more concurrent users to be able to use the Portal.

Cheers, Jason

[jliberta]

Hi @jayrodmcneil

We have about 700-900 users per day at peak load and we have exactly the same issue, we performed a ton of optimizations on our code and reduced the web notifications to an absolute minimum and the performance is still relatively poor compared to ADX v7. To give you an idea, we were running on v7 with 2 load balanced VMs 4 core 14GB of ram, with XRM we had to increase that to 3 load balanced servers with 16 cores and 32 GB of ram and there were still 502 errors on the client side with high CPU usage/spikes.

After this we switched over to isolated environments in Azure Web Apps because this architecture makes use of the portal bus like they had with v7 when running load balanced web farms which allowed us to reduce web notifications even more. We started off with I2 tier and 6 instances of the web app, this is equivalent to 2 hyperthreaded CPUs and 7GB of memory on each instance and it was still not enough as we had more performance issues. We are now running I3 tier with 6 instances which is equivalent to 4 hyperthreaded CPUs and 14GB of memory on each instance and the performance is better than our VM configuration but still worse than ADX v7. We have run out of ways to optimize this system.

[jayrodmcneil]

Thanks @jiliberta out of curiosity, how many concurrent users does that setup support? Normally and at peak times? Also on the optimizations, do you mean specifically changes within the XRM Community Portal Asp.net web application? Anything you could share about what some of those optimizations were?

[jliberta]

@jayrodmcneil I am not sure how many concurrent users we have but I can say that during peak hours of the day we can see up to 900 users in one our across all 6 instances. regular load would be somewhere around 300. The data in our application insights is not 100% accurate so I will estimate about 60-70 concurrent users per instance.

Regarding the optimizations, we did it on our custom code as it is not very optimal to investigate the XRM source code. I have fixed and submitted a couple of bugs that I found with sharepoint integration but it was related to multilingual labels because it wasn't working and our website is required to be in English and French. We also optimized webnotifications, note that there is a difference between having multiple VMs each with its on webnotification URL and running the app in an Azure Web App. The web app allows you to use the portal bus cache handler like in v7 so only one web notification is required. Doing so allowed us to remove a significant amount of entities from web notifications. Also in the plugin registration tool there is an ADX Webnotification Plugin registered with all the steps for create/update/delete of each entity you have enabled, for the update step you can actually modify the filtering attributes to decide which attribute updates on an entity record will trigger a web notification. This has helped us a lot and the application runs a lot better but there are still performance issues with requests taking 35+ seconds just not as many as before. I hope this helps!

[godind]

      Hey XRM Portals Community,

Just trying to gauge a basic understanding of the number of users of your portal supports. Our application is generally a 50/50 mix of read and write requests, nothing too fancy, mostly simple enough entity lists and forms. We find that in our load balanced environment, once we get up around 100 users per portal server, the CPUs max out and performance drops miserably to about 30 seconds per page load. The bottleneck for us doesn't appear to be CRM, rather the cpu on the portal servers, not sure the spec, but pretty beefy (all On-Premise). I'm just curious if this is normal or if you guys get/expect far more concurrent users to be able to use the Portal. Cheers, Jason

@jayrodmcneil I think the question of using multilingual site setup could help pin point perf capacity, if it has any influence.

[jayrodmcneil]

@jayrodmcneil I am not sure how many concurrent users we have but I can say that during peak hours of the day we can see up to 900 users in one our across all 6 instances. regular load would be somewhere around 300. The data in our application insights is not 100% accurate so I will estimate about 60-70 concurrent users per instance.

Regarding the optimizations, we did it on our custom code as it is not very optimal to investigate the XRM source code. I have fixed and submitted a couple of bugs that I found with sharepoint integration but it was related to multilingual labels because it wasn't working and our website is required to be in English and French. We also optimized webnotifications, note that there is a difference between having multiple VMs each with its on webnotification URL and running the app in an Azure Web App. The web app allows you to use the portal bus cache handler like in v7 so only one web notification is required. Doing so allowed us to remove a significant amount of entities from web notifications. Also in the plugin registration tool there is an ADX Webnotification Plugin registered with all the steps for create/update/delete of each entity you have enabled, for the update step you can actually modify the filtering attributes to decide which attribute updates on an entity record will trigger a web notification. This has helped us a lot and the application runs a lot better but there are still performance issues with requests taking 35+ seconds just not as many as before. I hope this helps!

Thanks for all the input, we're considering siwtching over to Azure Web Apps rather than VMware on-prem servers. Is there any special config to setup for the different cache invalidation using the Service Bus? Cutting down the number of Web Notifications would be a huge benefit, any guidance or links would be super.