search

Orckestra / C1-CMS-Foundation

C1 CMS Foundation - .NET based, open source and a bundle of joy!
https://c1.orckestra.com/
Other
250 stars 109 forks source link

XmlDataProvider would crash the site when a critical amount of data records for a data type is reached. #224

Open napernik opened 8 years ago

napernik commented 8 years ago

When a site running XmlDataProvider has a data type with sufficient amount of records, the website will use all the available memory and use 100% CPU under load.

Where the critical amount is 10625 records on 64 bit running IIS process and 21250 on 32 bit processes. And defined by the fact that lists of pointers that occupy more than 85000 bytes are allocated in the Large Object Heap (LOH) which is not cleared under high load as of .net 4.5.1.

The data type in question is often IPage or Newsletter package statistics. We usually recommend upgrading to SqlDataProvider in this case, but it should be possible to address the high memory usage by forcing LOH garbage collection and/or by switching to data structures parts of would not reside in LOH.

burningice2866 commented 8 years ago

Using Newsletter package which such high amount of data on XML Storage doesn't only cause memory problems but is stressful for the harddrive due to high amount of data being constantly written.

burningice2866 commented 4 years ago

Is this on any roadmap or is the official recommendation not to use XML storage for high amount of data records?

napernik commented 4 years ago

For high amount of records, such as in Newsletter - SqlDataProvider is the recommended solution.

For easier maintenance it is possible to configure data providers to have only specific data types on SqlDataProvider (f.e. Newsletter and Extranet data). Though setting up process it is not done through code but with some manual operations.

Large object heap overgrowing can be fixed even in the existing XmlDataProvider, but high disk usage and inefficient queries may require a different approach.

In future, we may have an alternative way (either file or document based) for keeping pages and related data in the future, but it isn't on the roadmap at the moment.

burningice2866 commented 4 years ago

So maybe change the case label from bug to recommendation?