arquivo / pwa-technologies

Arquivo.pt main goal is the preservation and access of web contents that are no longer available online. During the developing of the PWA IR (information retrieval) system we faced limitations in searching speed, quality of results, scalability and usability. To cope with this, we modified the archive-access project (http://archive-access.sourceforge.net/) to support our web archive IR requirements. Nutchwax, Nutch and Wayback’s code were adapted to meet the requirements. Several optimizations were added, such as simplifications in the way document versions are searched and several bottlenecks were resolved. The PWA search engine is a public service at http://archive.pt and a research platform for web archiving. As it predecessor Nutch, it runs over Hadoop clusters for distributed computing following the map-reduce paradigm. Its major features include fast full-text search, URL search, phrase search, faceted search (date, format, site), and sorting by relevance and date. The PWA search engine is highly scalable and its architecture is flexible enough to enable the deployment of different configurations to respond to the different needs. Currently, it serves an archive collection searchable by full-text with 180 million documents ranging between 1996 and 2010.
http://www.arquivo.pt
GNU General Public License v3.0
41 stars 7 forks source link

SavePageNow can't record websites whose CA certificate is slightly misconfigured #1284

Closed VascoRatoFCCN closed 2 years ago

VascoRatoFCCN commented 2 years ago

When trying to record some websites like www.essv.ipv.pt or stayawaycovid.pt we get the following response:

image

However, web browsers have no problem accessing these websites:

image

It happens because the CA certificate is missing some information that most web browsers can easily fill in. SavePageNow before starting to record a webpage, it tries to do a http request to the requested website. If anything goes wrong it doesn't start recording, in this case we get an error due to a misconfigured CA certificate.

VascoRatoFCCN commented 2 years ago

https://github.com/arquivo/arquivo-webapp-eros/commit/8e73a64e7ca460a2ae4601834093d2b675751a40 changes the checking behavior. Instead of blocking the request, it lets it go through and logs the error. e.g.: image

This is good because we are no longer blocking users from using our service.

However, we now are susceptible to getting some cryptic messages while trying to use SavePageNow: image

This SO answer has a solution to the CA certificate problem. We should analyze the logs to see if we should back to the previous implementation using this fix.

VascoRatoFCCN commented 2 years ago

A related problem, SavePageNow can be used to bypass security verification, it works on many of https://badssl.com/ examples of bad SSL. How the browser reacts: image How SavePageNow reacts: image

arquivo-awp commented 2 years ago
  1. Remove pre-validation of URL on front-end.
  2. Configure default error message on Pywb to avoid "ugly" error message on milestone Godhelpus https://github.com/arquivo/pwa-technologies/milestone/25
dcgomes commented 2 years ago

Fixed