Arquivo.pt main goal is the preservation and access of web contents that are no longer available online. During the developing of the PWA IR (information retrieval) system we faced limitations in searching speed, quality of results, scalability and usability. To cope with this, we modified the archive-access project (http://archive-access.sourceforge.net/) to support our web archive IR requirements. Nutchwax, Nutch and Wayback’s code were adapted to meet the requirements. Several optimizations were added, such as simplifications in the way document versions are searched and several bottlenecks were resolved. The PWA search engine is a public service at http://archive.pt and a research platform for web archiving. As it predecessor Nutch, it runs over Hadoop clusters for distributed computing following the map-reduce paradigm. Its major features include fast full-text search, URL search, phrase search, faceted search (date, format, site), and sorting by relevance and date. The PWA search engine is highly scalable and its architecture is flexible enough to enable the deployment of different configurations to respond to the different needs. Currently, it serves an archive collection searchable by full-text with 180 million documents ranging between 1996 and 2010.
Periodically we have been crawling the citations contained in the open-access publications of the RCAAP network of OA repositories by parsing their exported sitemaps.
The sitemaps of the repositories hosted at FCCN (SARI service) are exported daily at 6h00.
For the other repositories we don't have control of their update frequency.
The RCAAP API v.2, to be released until August 2022, will enable automatically query all the Portuguese open-access repositories, even if they are hosted externally to FCCN (e.g. RepositoriUM).
RCAAP API v.2 will export links to PDFs and allow us to restrict the “published” date, so that we can select only the new records of articles published since the last crawl of the RCAAP citations.
The obtained URLs from RCAAP API should be ingested on the workflow of the new CitationSaver service(see also Issue #1147).
PedroG1515
commented
12 months ago
Periodically we have been crawling the citations contained in the open-access publications of the RCAAP network of OA repositories by parsing their exported sitemaps.
The sitemaps of the repositories hosted at FCCN (SARI service) are exported daily at 6h00. For the other repositories we don't have control of their update frequency.
The RCAAP API v.2, to be released until August 2022, will enable automatically query all the Portuguese open-access repositories, even if they are hosted externally to FCCN (e.g. RepositoriUM).
RCAAP API v.2 will export links to PDFs and allow us to restrict the “published” date, so that we can select only the new records of articles published since the last crawl of the RCAAP citations.
The obtained URLs from RCAAP API should be ingested on the workflow of the new CitationSaver service(see also Issue #1147).