search

arquivo / pwa-technologies

Arquivo.pt main goal is the preservation and access of web contents that are no longer available online. During the developing of the PWA IR (information retrieval) system we faced limitations in searching speed, quality of results, scalability and usability. To cope with this, we modified the archive-access project (http://archive-access.sourceforge.net/) to support our web archive IR requirements. Nutchwax, Nutch and Wayback’s code were adapted to meet the requirements. Several optimizations were added, such as simplifications in the way document versions are searched and several bottlenecks were resolved. The PWA search engine is a public service at http://archive.pt and a research platform for web archiving. As it predecessor Nutch, it runs over Hadoop clusters for distributed computing following the map-reduce paradigm. Its major features include fast full-text search, URL search, phrase search, faceted search (date, format, site), and sorting by relevance and date. The PWA search engine is highly scalable and its architecture is flexible enough to enable the deployment of different configurations to respond to the different needs. Currently, it serves an archive collection searchable by full-text with 180 million documents ranging between 1996 and 2010.
http://www.arquivo.pt
GNU General Public License v3.0
38 stars 7 forks source link

Some files can only be accessed via noFrame #1285

Closed VascoRatoFCCN closed 2 years ago

VascoRatoFCCN commented 2 years ago

The following file can only be replayed on noFrame:

https://arquivo.pt/textsearch?metadata=http%3A%2F%2Fw3.memoshoa.pt%2Findex.php%3Fview%3Darticle%26catid%3D48%3A27-janeiro-2011%26id%3D163%3Anao-e-brincadeira%26format%3Dpdf%2F20131109043826

It seems to be related with the mimetype, even though this is a PDF, the framed replay identifies this as being an HTML file:

https://websniffer.cc/?url=https%3A%2F%2Farquivo.pt%2Fwayback%2F20131109043826%2Fhttp%3A%2F%2Fw3.memoshoa.pt%2Findex.php%3Fview%3Darticle%26catid%3D48%3A27-janeiro-2011%26id%3D163%3Anao-e-brincadeira%26format%3Dpdf

noFrame replay correctly identifies it as being a PDF:

https://websniffer.cc/?url=https%3A%2F%2Farquivo.pt%2FnoFrame%2Freplay%2F20131109043826%2Fhttp%3A%2F%2Fw3.memoshoa.pt%2Findex.php%3Fview%3Darticle%26catid%3D48%3A27-janeiro-2011%26id%3D163%3Anao-e-brincadeira%26format%3Dpdf

VascoRatoFCCN commented 2 years ago

Other PDFs also have this mime type incoherence, but have no problem replaying with the frame:

https://arquivo.pt/wayback/20120123091809/http://www.cedefop.europa.eu/EN/Files/4064_en.pdf

VascoRatoFCCN commented 2 years ago

Another thing that might be the root of the problem is the fact that the URL has special characters like '&' and '?', which is also preventing the version list on the left side of the replay from loading. Related issue: https://github.com/arquivo/pwa-technologies/issues/1258

VascoRatoFCCN commented 2 years ago

Had to do with pywb replacing & in urls with &. This is a documented issue ( https://github.com/webrecorder/pywb/issues/696 ), the latest pywb-arquivo version ( https://github.com/arquivo/pywb-arquivo/commit/40bbcb8c933697e679bb3994dbcb59e390228c70 ) implemented a workaround to this problem.

The page is now displaying correctly in preprod: image