search

arquivo / pwa-technologies

Arquivo.pt main goal is the preservation and access of web contents that are no longer available online. During the developing of the PWA IR (information retrieval) system we faced limitations in searching speed, quality of results, scalability and usability. To cope with this, we modified the archive-access project (http://archive-access.sourceforge.net/) to support our web archive IR requirements. Nutchwax, Nutch and Wayback’s code were adapted to meet the requirements. Several optimizations were added, such as simplifications in the way document versions are searched and several bottlenecks were resolved. The PWA search engine is a public service at http://archive.pt and a research platform for web archiving. As it predecessor Nutch, it runs over Hadoop clusters for distributed computing following the map-reduce paradigm. Its major features include fast full-text search, URL search, phrase search, faceted search (date, format, site), and sorting by relevance and date. The PWA search engine is highly scalable and its architecture is flexible enough to enable the deployment of different configurations to respond to the different needs. Currently, it serves an archive collection searchable by full-text with 180 million documents ranging between 1996 and 2010.
http://www.arquivo.pt
GNU General Public License v3.0
39 stars 7 forks source link

Add a link to save page now functionality on not archived page #924

Open igobranco opened 4 years ago

igobranco commented 4 years ago

Add a new link to the save page now functionality on the not archived page if the page still exists online it returns a 200.

If the user lands on: https://arquivo.pt/wayback/19961013145650/https://www.ft.com/content/67e1661b-f12b-4473-9bc2-aa2b5998ad73

And the original page still exists. https://www.ft.com/content/67e1661b-f12b-4473-9bc2-aa2b5998ad73

Then we need to show a link to the save page now functionality.

igobranco commented 4 years ago

Depends on #702

igobranco commented 4 years ago

To show the link only when if the page still exists online (return an HTTP status code 200), we need to create a new server side service that, for example, returns the current status code of any live web page. This service if implemented would require a QoS configuration to prevent abusive utilization. This new hypothetical service would require further development that wasn't planned. @dcgomes Could we just add the link to the service 'Save page now'?