Arquivo.pt main goal is the preservation and access of web contents that are no longer available online. During the developing of the PWA IR (information retrieval) system we faced limitations in searching speed, quality of results, scalability and usability. To cope with this, we modified the archive-access project (http://archive-access.sourceforge.net/) to support our web archive IR requirements. Nutchwax, Nutch and Wayback’s code were adapted to meet the requirements. Several optimizations were added, such as simplifications in the way document versions are searched and several bottlenecks were resolved. The PWA search engine is a public service at http://archive.pt and a research platform for web archiving. As it predecessor Nutch, it runs over Hadoop clusters for distributed computing following the map-reduce paradigm. Its major features include fast full-text search, URL search, phrase search, faceted search (date, format, site), and sorting by relevance and date. The PWA search engine is highly scalable and its architecture is flexible enough to enable the deployment of different configurations to respond to the different needs. Currently, it serves an archive collection searchable by full-text with 180 million documents ranging between 1996 and 2010.
What happened?
The URLs of the resources that are removed from Arquivo.pt should be also removed from the image and page indexes.
The problem is that the Page Search component does not allow updates on the indexes and it is too old to be developed to support this feature (based on Lucene/NutchWAX).
What should have happened?
The new Page Search system to be developed based on Solr should support index updates.
Maybe we can remove the blocked URLs at the Component-SearchPage-SearchPageAPI component by adding a blacklist of URLs.
Procedimento técnico de remoção – ASA
5.1 Remoção imediata, no Apache
5.2 Remoção nos CDX, Lucene e Query server
5.3. Atenção ao noFrame, mobile e desktop.
5.4. Pesquisa de imagens:
Estes resultados podem ser bloqueados dos resultados de pesquisa utilizando como inspiração o script update_block.py"
dcgomes
commented
2 years ago
What is the URL that originated the issue? https://arquivo.pt/page/search?q=%22Ricardo+Jorge+Salvador+Lopes%22&l=pt&from=19910806&to=20220627&trackingId=2a18c5e9a9e75f395049_6501e338b6dc7ae6468b
What happened? The URLs of the resources that are removed from Arquivo.pt should be also removed from the image and page indexes.
The problem is that the Page Search component does not allow updates on the indexes and it is too old to be developed to support this feature (based on Lucene/NutchWAX).
What should have happened? The new Page Search system to be developed based on Solr should support index updates. Maybe we can remove the blocked URLs at the Component-SearchPage-SearchPageAPI component by adding a blacklist of URLs.
Screenshots