Too many versions for some URLs

arquivo / pwa-technologies

Arquivo.pt main goal is the preservation and access of web contents that are no longer available online. During the developing of the PWA IR (information retrieval) system we faced limitations in searching speed, quality of results, scalability and usability. To cope with this, we modified the archive-access project (http://archive-access.sourceforge.net/) to support our web archive IR requirements. Nutchwax, Nutch and Wayback’s code were adapted to meet the requirements. Several optimizations were added, such as simplifications in the way document versions are searched and several bottlenecks were resolved. The PWA search engine is a public service at http://archive.pt and a research platform for web archiving. As it predecessor Nutch, it runs over Hadoop clusters for distributed computing following the map-reduce paradigm. Its major features include fast full-text search, URL search, phrase search, faceted search (date, format, site), and sorting by relevance and date. The PWA search engine is highly scalable and its architecture is flexible enough to enable the deployment of different configurations to respond to the different needs. Currently, it serves an archive collection searchable by full-text with 180 million documents ranging between 1996 and 2010.

http://www.arquivo.pt

GNU General Public License v3.0

38 stars 7 forks source link

Too many versions for some URLs #1243

Closed PedroG1515 closed 1 year ago

PedroG1515 commented 2 years ago

What is the URL that originated the issue? https://arquivo.pt/url/search?q=https://www.portugal.gov.pt/pt/gc22/

What happened? For 2022 we already have 14105 versions for the same URL. We already check, and they are all different sizes (maybe due to the website dynamics)

What should have happened? We must have some kind of limitation or filter.

Screenshots

VascoRatoFCCN commented 2 years ago

Experimented adding a 4 item limit on every 6h time window. Available for testing on development:

https://dev.arquivo.pt/url/search?q=https://www.portugal.gov.pt/pt/gc22/

VascoRatoFCCN commented 2 years ago

Removed the 4 item limit per 6h, we decided to handle this as a crawling issue rather than front-end issue. We feared that the filtering might end up excluding content that the users would be interested in.

We'll treat this as an isolated case and revisit this if it happens again.

arquivo-awp commented 1 year ago

Recheck after deploy new CDXJ cleaned from warc/revisits

franciscoesteveira commented 1 year ago

Deploying cleaned up CDXJ files did not fix the issue. Postponing this issue to the next milestone.

dcgomes commented 1 year ago

The CDXJ entries for this URL are not warc/revisits and reference different contents.

VascoRatoFCCN commented 1 year ago

To tackle this issue we adopted the following procedure:

We identify URLs that have over 1000 entries on the same collection as being problematic URLs
If a collection has problematic URLs, then for that collection in particular we delete all CDX entries of that URL except for the earliest one.

After this was implemented, it now shows a much more reasonable number of results:

This also fixed #1308