Open rivadblins opened 4 days ago
PVillarmarzo
commented
3 days ago To assess whether there is a performance issue, I need to know the type of database you are connecting to and its size. Additionally, the solution you suggested appears to be more of a feature request. It would be clearer to post it separately.
rivadblins
commented
2 days ago We are using MongoDB that contains 5.6 million documents, there are roughly 20k documents per day. I agree, I will create a feature request along side this, however I think it's still a bug that it takes 10 minutes to load as that makes this feature functionally unusable for us since we cannot wait 10 minutes to test out a value filter for every visualization we add.
MLopezIG
commented
2 days ago @rivadblins Can you tell us a bit more about about the field you're trying to filter? Is it a low or high cardinality column?
Thinking about possible workarounds here. You mention that this is needed to 'apply a field-value filter in the visualization editor'. Do you need this to be fixed too for the following?:
rivadblins
commented
2 days ago There are roughly 10-15 possible values. We have logs in a database with an eventSubType, we only want entries that have Create, Update, or Delete in the eventSubType field. When building the visualization I went to apply this filter and had to wait 10 minutes while it polled the database. Here is the discord conversation about the issue: https://discord.com/channels/1050800659911430254/1288944657954508880/1288970663662714922
Because of other performance issues we decided not to use the direct MongoDB Datasource type and wrote our own backend so this bug ticket was not created to resolve our issue.
brianlagunas
commented
2 days ago @rivadblins in your scenario, and due to the massive number of records you are dealing with, and the fact that you know exactly what you want from the logs, I would recommend using a custom query for your Create/Update/Delete logs data set. Then simply use that as your data source item.
Because of other performance issues we decided not to use the direct MongoDB Datasource type and wrote our own backend so this bug ticket was not created to resolve our issue.
Since you have decided not to use MongoDB, what have you moved to? I see you have submitted to issues regarding REST. Are you moving forward with REST? Have you had any perf issues with REST connecting to your MongoDB?
Looking at the core of this issue, it seems to be around the performance of the Mongo queries being generated by Reveal. We are using issue #288 to track that item. Would it be fair to say that this is essentially the same issue in just a different area of the code?
SDK Version
1.7
Client Framework
Other
Server Platform
ASP.NET
Operating System
Windows
Description
When attempting to apply a field-value filter in the visualization editor, the UI hangs for a large amount of time (upwards of 10 minutes) when the dataset is large. This is because it automatically fetches distinct values for a field from the database.
Instead of automatically fetching the distinct values from the database and choosing from the list of checkboxes, a simple text field to enter values and a button to fetch values from the database would be preferred.