Arquivo.pt main goal is the preservation and access of web contents that are no longer available online. During the developing of the PWA IR (information retrieval) system we faced limitations in searching speed, quality of results, scalability and usability. To cope with this, we modified the archive-access project (http://archive-access.sourceforge.net/) to support our web archive IR requirements. Nutchwax, Nutch and Wayback’s code were adapted to meet the requirements. Several optimizations were added, such as simplifications in the way document versions are searched and several bottlenecks were resolved. The PWA search engine is a public service at http://archive.pt and a research platform for web archiving. As it predecessor Nutch, it runs over Hadoop clusters for distributed computing following the map-reduce paradigm. Its major features include fast full-text search, URL search, phrase search, faceted search (date, format, site), and sorting by relevance and date. The PWA search engine is highly scalable and its architecture is flexible enough to enable the deployment of different configurations to respond to the different needs. Currently, it serves an archive collection searchable by full-text with 180 million documents ranging between 1996 and 2010.
As of the Eros deploy, our memento timemap replies link towards arquivo.pt/noFrame/replay rather than arquivo.pt/wayback.
A consequence of this is that arquivo404 links towards the unframed (fullscreen) version of archived websites.
Should arquivo404 redirect the user to the framed version of a page or the fullscreen version?
Showing the framed version would allow the user to see all the different versions of the page we have archived, while also promoting Arquivo.pt
On the other hand, returning the unframed version looks more natural and may avoid user confusion, while at the same time people might feel more inclined to use arquivo404 if there is no "Arquivo.pt" branding on it.
Should our memento timemap replies link towards the fullscreen version?
Linking the framed version is what other web archives do, like the Internet Archive. It also allows the user to explore different versions available on Arquivo.pt and have access to other services like "completePage" which increases the usefulness of the linked resource.
Latest Francisco release fixed this issue by hosting the framed replay directly on pywb, consequently the memento API does too, which is what arquivo404 uses to find an archived page.
As of the Eros deploy, our memento timemap replies link towards arquivo.pt/noFrame/replay rather than arquivo.pt/wayback. A consequence of this is that arquivo404 links towards the unframed (fullscreen) version of archived websites.
Should arquivo404 redirect the user to the framed version of a page or the fullscreen version?
Should our memento timemap replies link towards the fullscreen version?