Arquivo.pt main goal is the preservation and access of web contents that are no longer available online. During the developing of the PWA IR (information retrieval) system we faced limitations in searching speed, quality of results, scalability and usability. To cope with this, we modified the archive-access project (http://archive-access.sourceforge.net/) to support our web archive IR requirements. Nutchwax, Nutch and Wayback’s code were adapted to meet the requirements. Several optimizations were added, such as simplifications in the way document versions are searched and several bottlenecks were resolved. The PWA search engine is a public service at http://archive.pt and a research platform for web archiving. As it predecessor Nutch, it runs over Hadoop clusters for distributed computing following the map-reduce paradigm. Its major features include fast full-text search, URL search, phrase search, faceted search (date, format, site), and sorting by relevance and date. The PWA search engine is highly scalable and its architecture is flexible enough to enable the deployment of different configurations to respond to the different needs. Currently, it serves an archive collection searchable by full-text with 180 million documents ranging between 1996 and 2010.
Due to the latest changes on arquivo.pt webapp, wayback links with timestamps ending in im_ or with similar endings no longer link towards the original file, instead they link towards a framed replay page.
This caused problems with the webapp because the image search service relied on those URLs to display the image. This was never detected in development because the URL that the API returns is always pointing towards the production environment, even when it is hosted in a development environment. This meant that even in development, the images displayed in an image search were loaded from the prodution server. A quickfix was implemented to handle this issue.
However, a better solution for this would be the following:
Add a configuration on the image search API to make the URLs it returns reflect the production environment it is hosted on.
Add a new field to the image search API reply called imgLinkToNoFrame that links towards the image directly, and not replay page.
Alter the webapp to reflect these API changes (see image below):
This is no longer an issue, it had to do with the reimplementation of the framed replay using the webapp rather than pywb's template engine.
This reimplementation has been discarded because we have a lot of other services that depend on pywb that would need to be tweaked or reimplemented.
Since then this ceased to be an issue.
Due to the latest changes on arquivo.pt webapp, wayback links with timestamps ending in
im_
or with similar endings no longer link towards the original file, instead they link towards a framed replay page.This caused problems with the webapp because the image search service relied on those URLs to display the image. This was never detected in development because the URL that the API returns is always pointing towards the production environment, even when it is hosted in a development environment. This meant that even in development, the images displayed in an image search were loaded from the prodution server. A quickfix was implemented to handle this issue.
However, a better solution for this would be the following:
imgLinkToNoFrame
that links towards the image directly, and not replay page.imgLinkToArchive
pageLinkToArchive
imgLinkToNoFrame
Example of an image search API reply:
Should become: