Arquivo.pt main goal is the preservation and access of web contents that are no longer available online. During the developing of the PWA IR (information retrieval) system we faced limitations in searching speed, quality of results, scalability and usability. To cope with this, we modified the archive-access project (http://archive-access.sourceforge.net/) to support our web archive IR requirements. Nutchwax, Nutch and Wayback’s code were adapted to meet the requirements. Several optimizations were added, such as simplifications in the way document versions are searched and several bottlenecks were resolved. The PWA search engine is a public service at http://archive.pt and a research platform for web archiving. As it predecessor Nutch, it runs over Hadoop clusters for distributed computing following the map-reduce paradigm. Its major features include fast full-text search, URL search, phrase search, faceted search (date, format, site), and sorting by relevance and date. The PWA search engine is highly scalable and its architecture is flexible enough to enable the deployment of different configurations to respond to the different needs. Currently, it serves an archive collection searchable by full-text with 180 million documents ranging between 1996 and 2010.
Some times WARCs don't include the status code of the archived web page. This could be for a variety of reasons, but usually it's due to WARCS being constructed from donated server backups, rather than being collected from the live web.
This is a problem because some of our procedures filter results to show only entries with status 2xx or 3xx.
We need to make sure that these WARCs are being properly indexed and that they are accessible from the webapp.
VascoRatoFCCN
commented
1 year ago 
PedroG1515
commented
5 months ago
Some times WARCs don't include the status code of the archived web page. This could be for a variety of reasons, but usually it's due to WARCS being constructed from donated server backups, rather than being collected from the live web.
This is a problem because some of our procedures filter results to show only entries with status 2xx or 3xx.
We need to make sure that these WARCs are being properly indexed and that they are accessible from the webapp.
For example, the following archived webpage displays as having only 1 version: https://arquivo.pt/wayback/20091028084320/http://geocities.yahoo.com/v/alert.html#
Using the CDX API, we see that we actually have 3 different entries, but 2 of them have no status code: https://arquivo.pt/wayback/cdx?url=http://geocities.yahoo.com/v/alert.html#