search

tflobbe / solrmeter

Stress test tool for Apache Solr
70 stars 36 forks source link

Simple tool to extract queries from a Solr log #90

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
It would be nice to have a small tool inside SolrMeter to extract queries from 
a Solr log file and store them into a new file.
The generated file can then be used by SolrMeter to run a stress test thanks to 
Issue 55 (
http://code.google.com/p/solrmeter/issues/detail?id=55 ).

There is a screenshot attached to give a rough idea of how it should look like.

Original issue reported on code.google.com by e.to...@sourcesense.com on 7 Oct 2011 at 8:39

Attachments:

GoogleCodeExporter commented 9 years ago 
This is a first cut. I'm submitting it just to have your feedback.
At the moment you can select a solr log file and you have to type an output 
file name.
There isn't any error handling yet.

This is a list of issues that should be fixed:
- the output file is generated into /tmp directory
  solution: add an extra field to let a user specify the output directory. (Default directory will be the same of the Solr log file)
- at the moment the regex that extracts queries is inside the Java code. It 
would be nice to have it customizable. In that case I could introduce another 
UI field so a user can write his own regex depending on his log file.
- the regex now is not able to distinguish between select / update queries. 
Probably we would like just to have select queries and not ping or update 
queries.
- improve UI and add error handling

Any feedback is really appreciated

edo

Original comment by e.to...@sourcesense.com on 7 Oct 2011 at 8:56

Attachments:

GoogleCodeExporter commented 9 years ago 
This is great Edoardo! I added some internationalization. Added also a file 
selection for the output file. Default output file is called "output.txt" and 
is located under the same directory as the source file.
I also added a details panel, to show the location of the generated file or the 
message of the exception if there is an error. 
Still needs to be done:
- regex has to be customizable, probably by adding another input in the same 
dialog.
- regex still doesn't recognise updates / selectrs
- Still needs some more error handling.

Original comment by tflo...@gmail.com on 18 Oct 2011 at 1:20

GoogleCodeExporter commented 9 years ago 
Apparently I forgot to add some files. Here is the patch again.

Original comment by tflo...@gmail.com on 18 Oct 2011 at 1:41

Attachments:

GoogleCodeExporter commented 9 years ago 
Hi all,
today i had some spare time and i checked out your patch!
i also added a few more things:
- added an input textfield for the regular expression. The regex is now 
customizable from the UI and prefilled with the default regex
- fixed regex: now it gets queries executed only on the select query handler
- added some error handling

I also added some padding on the progress bar and on the textarea.

Finally i added another option called "remove duplicate queries". Basically 
with this option enabled the query file generated will not contain duplicated 
queries.

please give it a try and report possible bugs or improvements :)

edo

Original comment by edoardo.tosca on 7 Nov 2011 at 5:23

Attachments:

GoogleCodeExporter commented 9 years ago 
It works great. I'll commit this patch.

Original comment by tflo...@gmail.com on 9 Nov 2011 at 2:23

GoogleCodeExporter commented 9 years ago 
Committed in revision 279. Updates and improvements to this tool should be done 
in different issues, I'm closing this one.

Thanks for your work Edoardo.

Original comment by tflo...@gmail.com on 10 Nov 2011 at 12:09

GoogleCodeExporter commented 9 years ago

Original comment by tflo...@gmail.com on 9 Oct 2012 at 10:48