search

OrchardCMS / Orchard

Orchard is a free, open source, community-focused Content Management System built on the ASP.NET MVC platform.
https://orchardproject.net
BSD 3-Clause "New" or "Revised" License
2.38k stars 1.12k forks source link

Indexing clogging site with 15 indices #3777

Open orchardbot opened 11 years ago

orchardbot commented 11 years ago

@Piedone created: https://orchard.codeplex.com/workitem/19949

We have a site with 15 Lucene indices, where 13 of them are targeted at the same content types, thus containing the same 127 items (they shouldn't include the same items however, see: https://orchard.codeplex.com/workitem/19925).

When an item of the content types in question is published, indexing happens at the same time for all indices (that's OK), causing massive CPU and I/O spikes (to 100% CPU utilization) on the server that copes with the site with 3 indices, one being the problem-causing kind without showing any excessive resource usage (heavy issues described below seem to start to show at about 4 such indices).

Site startup time is lengthened by these indices too, but in a seemingly random manner (i.e. while without these indices the startup time is stable and OK, with indices it varies by the minutes and is generally much longer).

Also I experienced startup to get stuck with that many indices (what I wouldn't consider too many though) even on my machine where performance isn't an issue: startup runs at a stable CPU usage (of about 20% on my machine) but it never finishes (the site doesn't load).

Removing the problematic indices and leaving only one (with two additional indices present, containing different types of items) instantly resolves all issues described above.

Almost latest 1.x, log is empty.

With no profiling solution I'm aware of available for this Orchard version I can't tell anything more specific, but hopefully somebody can relate.

alberthajdu commented 7 years ago

More indexing bugs which lock the site.

  1. Index rebuild. With 5000 indexed content items the rebuild takes ~9s (on my machine), meanwhile the frontend is unreachable.
  2. Index update also locks the site in case of the index is freshly created and the automatic build isn't finished yet. This is the same issue as the above one because they are sharing the same logic.

More related bugs.

  1. You have 1 index with 500 indexed content items. Add 500 more content items. The index update bgtask runs in the background and you click on update. With this once I ended up with 1050 indexed items and once I ended up with 700 items. In case of 700 items the left 300 never updated. In case of 1050 the 50 is just there in the index redundantly. Of course if you hit a rebuild the result is 1000 indexed items but something definitely not OK with the current process.
  2. The same result can be achieved if you rebuild an another index during the first index background update.

The frontend lock issues maybe can be solved if we process the rebuild and the frontend triggered update in batches too. Or at least put a warning message somewhere near the rebuild and update links. The update mess maybe caused by the fact that the IndexingTaskRecord aren't per index or I don't know.

Piedone commented 7 years ago

@alberthajdu did you test with SQL Server or SQL CE? In the latter case could it be that the lockup happens due to the DB refusing to handle more load (have seen this occasionally)? I guess there is no log entry, or is there?

alberthajdu commented 7 years ago

I tested it with both. With SQL Server and 5000 content items the site is locked and rebuilding takes ~10s. Log is empty.