Content Search

[romruben]

• [x] Frameworks Definition
• [x] Experiment Design
• [x] Implementation
• [x] Execution & Analysis

[romruben]

In the first documentation round, i found the following info:

Content Search

• Full text search: The wikipedia entry for "full text search"

In this link we can found several points of study and a deficiencies list in the text search. Also the page contains some suggestions for avoid these deficiencies. On the other hand there is a list of free and open source frameworks as well as proprietary. (must-read)

• FullText search engines: This is a list of Fulltext Search Engines, and technologies that could potentially be used to build them.
• hOOt: Smallest full text search engine (lucene replacement) built from scratch using inverted WAH bitmap index, highly compact storage, operating in database and document modes
• Open Semantic Searh: Free software for your own search engine, data explorer and research tools based on Apache Lucene / Solr or Elastic Search open-source enterprise-search and open standards for Linked Data and Semantic Web.
• FullProof: A javascript-based fulltext search engine library.

[fpromero]

Ok, It is very important

1. to build an index from the bussiness documents provided by Docpath.
2. Define 3-5 examples queries
3. Perform the evaluation.

Eloy can help you in this tasks.

[romruben]

• Sphinx : A search engine designed for indexing database content. It natively supports MySQL, PostgreSQL, and XML pipe interfaces. It is written in C++ and has a GPL license.
• Xapian : Xapian is an Open Source Search Engine Library, released under the GPL. It's written in C++, with bindings to allow use from Perl, Python, PHP, Java, Tcl, C#, Ruby, Lua, Erlang and Node.js (so far!) Xapian is a highly adaptable toolkit which allows developers to easily add advanced indexing and search facilities to their own applications. It supports the Probabilistic Information Retrieval model and also supports a rich set of boolean query operators.
• Apache Lucene : Apache Lucene is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform.
• Apache Solr : Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling. Solr is highly scalable, providing distributed search and index replication, and it powers the search and navigation features of many of the world's largest internet sites.
• Elastic Search : ElasticSearch could be opted if, We want our search solution to be fast, we want a painless setup and a completely free search schema, we want to be able to index data simply using JSON over HTTP, we want our search server to be always available, we want to be able to start with one machine and scale to hundreds, we want real-time search, we want simple multi-tenancy, and we want a solution that is built for the cloud.
• Index Tank : IndexTank search engine powers search in Reddit, Social bookmarking site. IndexTank is acquired by LinkedIn and released the project as open source. It includes features like Variables boosts, Facets, Faceted search, Snippeting, Custom scoring functions, Suggest, and Autocomplete.
• Katta : Katta is a scalable, failure tolerant, distributed, data storage for real time access. Katta serves large, replicated, indices as shards to serve high loads and very large data sets. These indices can be of different type. Currently implementations are available for Lucene and Hadoop mapfiles.
• Bobo Search : Bobo Browse is an information retrieval technology that provides navigational browsing into a semi-structured dataset. Beyond the result set from queries and selections, Bobo Browse also provides the facets from this point of browsing. It provides support to sort documents on fields that have multiple values. It is stable and used by LinkedIn.
• Compass : Compass provides real time searchengine built on top of Lucene. It is distributed, transcational, supports Spring MVC, integrates with ORM. Compass provides google-style search, index updates as well as more advanced concepts such as caching and index sharding (sub indexes).
• Summa : Summa is a fast modular and scalable search engine written in Java. It can simultaneously access a number of different data and data sources and expose it in a unified interface. It supports distributed architecture and fault tolerant. It can be scaled up or down to handle any amount of data.
• Constellio: Constellio Open Source Enterprise Search is based on Apache Solr and using Google Search Appliances connectors architecture, it allows, with a single click, to find all relevant content in your organization (Web, email, ECM, CRM etc.).

[romruben]

Experiment Design

Content Search

• [ ] The execution time
• [ ] Memory consumption
• [ ] Tests with 5 queries the number of correct results obtained.
• [ ] Stress tests with JMETER or Gatling (http://gatling.io/)

i will use Docker for a fast deploy of the infrastructure as elasticsearch, solr, and so on. Each one in a docker container.

@fpromero

[fpromero]

OK

[romruben]

In this issue, the frameworks definition is needed.

[romruben]

• Sphinx : A search engine designed for indexing database content. It natively supports MySQL, PostgreSQL, and XML pipe interfaces. It is written in C++ and has a GPL license.
• Xapian : Xapian is an Open Source Search Engine Library, released under the GPL. It's written in C++, with bindings to allow use from Perl, Python, PHP, Java, Tcl, C#, Ruby, Lua, Erlang and Node.js (so far!) Xapian is a highly adaptable toolkit which allows developers to easily add advanced indexing and search facilities to their own applications. It supports the Probabilistic Information Retrieval model and also supports a rich set of boolean query operators.
• Apache Lucene : Apache Lucene is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform.
• Apache Solr : Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling. Solr is highly scalable, providing distributed search and index replication, and it powers the search and navigation features of many of the world's largest internet sites.
• Elastic Search : ElasticSearch could be opted if, We want our search solution to be fast, we want a painless setup and a completely free search schema, we want to be able to index data simply using JSON over HTTP, we want our search server to be always available, we want to be able to start with one machine and scale to hundreds, we want real-time search, we want simple multi-tenancy, and we want a solution that is built for the cloud.
• Index Tank : IndexTank search engine powers search in Reddit, Social bookmarking site. IndexTank is acquired by LinkedIn and released the project as open source. It includes features like Variables boosts, Facets, Faceted search, Snippeting, Custom scoring functions, Suggest, and Autocomplete.
• Katta : Katta is a scalable, failure tolerant, distributed, data storage for real time access. Katta serves large, replicated, indices as shards to serve high loads and very large data sets. These indices can be of different type. Currently implementations are available for Lucene and Hadoop mapfiles.
• Bobo Search : Bobo Browse is an information retrieval technology that provides navigational browsing into a semi-structured dataset. Beyond the result set from queries and selections, Bobo Browse also provides the facets from this point of browsing. It provides support to sort documents on fields that have multiple values. It is stable and used by LinkedIn.
• Compass : Compass provides real time searchengine built on top of Lucene. It is distributed, transcational, supports Spring MVC, integrates with ORM. Compass provides google-style search, index updates as well as more advanced concepts such as caching and index sharding (sub indexes).
• Summa : Summa is a fast modular and scalable search engine written in Java. It can simultaneously access a number of different data and data sources and expose it in a unified interface. It supports distributed architecture and fault tolerant. It can be scaled up or down to handle any amount of data.
• Constellio: Constellio Open Source Enterprise Search is based on Apache Solr and using Google Search Appliances connectors architecture, it allows, with a single click, to find all relevant content in your organization (Web, email, ECM, CRM etc.).

[romruben]

Frameworks Definition

Content Search

• Sphinx : A search engine designed for indexing database content. It natively supports MySQL, PostgreSQL, and XML pipe interfaces. It is written in C++ and has a GPL license.
• Apache Solr : Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling. Solr is highly scalable, providing distributed search and index replication, and it powers the search and navigation features of many of the world's largest internet sites.
• Elastic Search : ElasticSearch could be opted if, We want our search solution to be fast, we want a painless setup and a completely free search schema, we want to be able to index data simply using JSON over HTTP, we want our search server to be always available, we want to be able to start with one machine and scale to hundreds, we want real-time search, we want simple multi-tenancy, and we want a solution that is built for the cloud.

[fpromero]

Donde almacena los documentos SPHINX en una base de datos MySQL o Postgres?

[romruben]

ambas, normalmente mysql.

ya tengo instalado y configurado un elasticsearch y solr. Hoy no se si meterle indices random que encuentre por ahí o ponerme con el postagger.

[romruben]

Al final me he visto obligado a rehacer todo, lo de docker esta bien para un entorno en el que todo esta configurado y tal, pero me he topado con ciertos bugs de esta version y me ha dificultado el trabajo. Además la comunicación con el contenedor no iba muy bien.

Finalmente he optado por el camino tradicional, re-instalar y volver a configurar solr y elasticsearch en local. Una vez configurados se han indexado los documentos de docpath en plan prueba con solo dos campos de indice, el titulo del fichero y el contenido, tanto en solr como en elasticsearch.

Mañana empezaré a lanzar búsquedas y ver rendimientos, si es posible, empezaré a bombardear con peticiones haciendo uso de gatler.

[romruben]

Durante el día de hoy, he estado estudiando cómo realizar las búsquedas contra Solr y Elasticsearch. He buscado hacerlas lo más sencillo posible via URI. Una vez comprobado como funciona, lo he documentado y he pasado a evaluar la herramienta que hablamos, Gatling.

Esta herramienta es tondo un submundo por debajo, todo programado en scala, arquetipos de mvn propios, etc. Bueno finalmente he conseguido programar, compilar y lanzar un test sencillo contra el servidor local solr, consistente en 10.000 peticiones de búsqueda por parte de 100 usuarios concurrentes de la palabra "motorola".

Se pueden ver algunos resultados del reporte final obtenido:

Mañana hago el script en scala en el mismo proyecto que he subido al repositorio, esta vez para elasticsearch. Y me voy a poner con el #2 POSTagger, si te parece bien, porque en esta sección ya hay poco más que rascar hasta las pruebas con documentos mejores y una estrategia.

[romruben]

Ayer no hice comentarios puesto que me rechinaba algunos resultados que obtenía con gatling entre solr y elasticsearch y me he pasado un par de días analizándolos. Esta mañana he descubierto que estaba pasando y he conseguido hacer los test correctamente. Las conclusiones que se ha sacado es que, Solr está permitiéndonos realizar por lo general más peticiones de búsqueda por segundo (en torno al 15% más según mis tests).

De todas maneras esto, con las pruebas futuras y documentos reales, se podrá explorar mejor y ver qué configuración tiene cada uno mas detalladamente y hacer búsquedas más complejas. Puede que el rendimiento no este tanto en recibir y contestar peticiones sencillas, como el de responder a peticiones con alto numero de campos etc.