search

extent-framework / klov

Report server for ExtentReports
http://klov.herokuapp.com
38 stars 32 forks source link

Performance issues by many reports in the database #77

Open filipwoz opened 3 years ago

filipwoz commented 3 years ago

Unfortunately the new KLOV has still the same problem as the old one: When database is growing the performance of loading the reports is continuously falling until it exceeds user acceptance boarder.

I observed it for example with the following number of records in the Mongo collections:

Table | Record Count project | 18 report | 340 test | 15.230 log | 198.085 media | 25.388 exception | 502

After deleting projects and getting below the log < 110.000, media < 15.000 it was acceptable again, but sorry it is quite unprofessional that here no strategy is offered to help that situation. Have you through an optimizing the queries?

This ways actually my main reason why I have not bought the KLOV professional licence for the old version and why I evaluated the new one. In such a shape I am also no willing to spend money on the new KLOV.

filipwoz commented 3 years ago

I have helped myself temporarily by writing a utility working on MongoDB level which is deleting reports older than X working days and X is configurable per KLOV project name. I am scheduling it every day, keeping my KLOV usable this way.

But at the end the performance is an issue here and you should work on it. I have an impression that you are not using the advantages of No-SQL DB properly. Why to use No-SQL when at the end you are keeping the test reports decomposed in different collections and you have to join it before rendering. With such data model a normal SQL DB would be quicker in my opinion. In a No-SQL DB like Mongo you could have been saving the complete test reports as JSON documents in one collection (report/test/log/exception all in one collection) without a need to join anything after that. Then the real benefit of No-SQL would be utilized. I am not sure about it because I am also new to No-SQL but this is how I understand it. No-SQL at the end should boost performance and not the other way round - it is its main benefit.

iamakshayshar commented 2 years ago

@filipwoz : if possible, can you please share the utility and its documentation with me. email : akshay.sharma979@gmail.com

Thanks in advance.

parkerqian commented 2 years ago

I have helped myself temporarily by writing a utility working on MongoDB level which is deleting reports older than X working days and X is configurable per KLOV project name. I am scheduling it every day, keeping my KLOV usable this way.

But at the end the performance is an issue here and you should work on it. I have an impression that you are not using the advantages of No-SQL DB properly. Why to use No-SQL when at the end you are keeping the test reports decomposed in different collections and you have to join it before rendering. With such data model a normal SQL DB would be quicker in my opinion. In a No-SQL DB like Mongo you could have been saving the complete test reports as JSON documents in one collection (report/test/log/exception all in one collection) without a need to join anything after that. Then the real benefit of No-SQL would be utilized. I am not sure about it because I am also new to No-SQL but this is how I understand it. No-SQL at the end should boost performance and not the other way round - it is its main benefit.

I have helped myself temporarily by writing a utility working on MongoDB level which is deleting reports older than X working days and X is configurable per KLOV project name. I am scheduling it every day, keeping my KLOV usable this way.

But at the end the performance is an issue here and you should work on it. I have an impression that you are not using the advantages of No-SQL DB properly. Why to use No-SQL when at the end you are keeping the test reports decomposed in different collections and you have to join it before rendering. With such data model a normal SQL DB would be quicker in my opinion. In a No-SQL DB like Mongo you could have been saving the complete test reports as JSON documents in one collection (report/test/log/exception all in one collection) without a need to join anything after that. Then the real benefit of No-SQL would be utilized. I am not sure about it because I am also new to No-SQL but this is how I understand it. No-SQL at the end should boost performance and not the other way round - it is its main benefit.

agree with you . My other problem is that my testng.xml includes so many case (perhaps more then 800) .Every case will output log info . At the end the total document is more then 16MB , mongodb can not hander that @iamakshayshar

iamakshayshar commented 2 years ago

I have helped myself temporarily by writing a utility working on MongoDB level which is deleting reports older than X working days and X is configurable per KLOV project name. I am scheduling it every day, keeping my KLOV usable this way.

But at the end the performance is an issue here and you should work on it. I have an impression that you are not using the advantages of No-SQL DB properly. Why to use No-SQL when at the end you are keeping the test reports decomposed in different collections and you have to join it before rendering. With such data model a normal SQL DB would be quicker in my opinion. In a No-SQL DB like Mongo you could have been saving the complete test reports as JSON documents in one collection (report/test/log/exception all in one collection) without a need to join anything after that. Then the real benefit of No-SQL would be utilized. I am not sure about it because I am also new to No-SQL but this is how I understand it. No-SQL at the end should boost performance and not the other way round - it is its main benefit.

I have helped myself temporarily by writing a utility working on MongoDB level which is deleting reports older than X working days and X is configurable per KLOV project name. I am scheduling it every day, keeping my KLOV usable this way.

But at the end the performance is an issue here and you should work on it. I have an impression that you are not using the advantages of No-SQL DB properly. Why to use No-SQL when at the end you are keeping the test reports decomposed in different collections and you have to join it before rendering. With such data model a normal SQL DB would be quicker in my opinion. In a No-SQL DB like Mongo you could have been saving the complete test reports as JSON documents in one collection (report/test/log/exception all in one collection) without a need to join anything after that. Then the real benefit of No-SQL would be utilized. I am not sure about it because I am also new to No-SQL but this is how I understand it. No-SQL at the end should boost performance and not the other way round - it is its main benefit.

agree with you . My other problem is that my testng.xml includes so many case (perhaps more then 800) .Every case will output log info . At the end the total document is more then 16MB , mongodb can not hander that @iamakshayshar

Yes, this is also the issue with Klov. One other problem is lack of support. 😟