search

arquivo / pwa-technologies

Arquivo.pt main goal is the preservation and access of web contents that are no longer available online. During the developing of the PWA IR (information retrieval) system we faced limitations in searching speed, quality of results, scalability and usability. To cope with this, we modified the archive-access project (http://archive-access.sourceforge.net/) to support our web archive IR requirements. Nutchwax, Nutch and Wayback’s code were adapted to meet the requirements. Several optimizations were added, such as simplifications in the way document versions are searched and several bottlenecks were resolved. The PWA search engine is a public service at http://archive.pt and a research platform for web archiving. As it predecessor Nutch, it runs over Hadoop clusters for distributed computing following the map-reduce paradigm. Its major features include fast full-text search, URL search, phrase search, faceted search (date, format, site), and sorting by relevance and date. The PWA search engine is highly scalable and its architecture is flexible enough to enable the deployment of different configurations to respond to the different needs. Currently, it serves an archive collection searchable by full-text with 180 million documents ranging between 1996 and 2010.
http://www.arquivo.pt
GNU General Public License v3.0
38 stars 7 forks source link

Encoding Error in the display of Full-text Search results #236

Open Fernando-Melo opened 7 years ago

Fernando-Melo commented 7 years ago

If you try the following full-text search query the first results will present encoding issues:

http://arquivo.pt/search.jsp?l=pt&query=usou+filho+como+testa+de+ferro&btnSubmit=Pesquisar&dateStart=01%2F01%2F1996&dateEnd=31%2F12%2F2015

danielbicho commented 7 years ago

Another example:

http://arquivo.pt/search.jsp?query=s%C3%B3crates&dateStart=01/01/1996&dateEnd=31/12/2015&pag=14&start=130&hitsPerPage=10&hitsPerDup=2&dedupField=site&l=pt

Fernando-Melo commented 4 years ago

I'm pretty sure the problem is that those results were indexed with encoding problems, so there is nothing we can do in production to fix it.

danielbicho commented 4 years ago

Problem with the extracted text, there is nothing we can do in Production. Will re-tag the issue so we can test the new indexing against this collection.

https://arquivo.pt/textextracted?m=http%3A%2F%2Fsol.sapo.pt%2Finicio%2FSociedade%2FInterior.aspx%3Fcontent_id%3D34147%2F20111118160302

"snippet": "<em>Usou</em> <em>filho</em> <em>como</em> <em>testa</em>-<em>de</em>-<em>ferro</em> - Sociedade - Sol sexta-feira, 18 <em>de</em> Novembro <em>de</em> 2011, 16:03<span class=\"ellipsis\"> ... </span> <em>Usou</em> <em>filho</em> <em>como</em> <em>testa</em>-<em>de</em>-<em>ferro</em> 18&#65533;<em>de</em>&#65533;Novembro,&#65533;2011 por Fel&#65533;cia Cabrita Duarte Lima <em>usou</em> o <em>filho</em>, Pedro<span class=\"ellipsis\"> ... </span> recente contrata&#65533;&#65533;o do... &#65533; &#65533; Sociedade <em>Usou</em> <em>filho</em> <em>como</em> <em>testa</em>-<em>de</em>-<em>ferro</em> &#65533; &#65533; Tecnologia Smartphones<span class=\"ellipsis\"> ... </span>",

amourao commented 3 years ago

Related https://github.com/arquivo/pwa-technologies/issues/1059 Test if this special cases can be fixed in the full text information extraction process.

VascoRatoFCCN commented 11 months ago

reevaluate after solr implementation for textsearch