search

CenterForOpenScience / scrapi

A data processing pipeline that schedules and runs content harvesters, normalizes their data, and outputs that normalized data to a variety of output streams. This is part of the SHARE project, and will be used to create a free and open dataset of research (meta)data. Data collected can be explored at https://osf.io/share/, and viewed at https://osf.io/api/v1/share/search/. Developer docs can be viewed at https://osf.io/wur56/wiki
Apache License 2.0
41 stars 45 forks source link

Feature/lastharvest #509

Open MerlinZhang opened 8 years ago

MerlinZhang commented 8 years ago

New Feature

Keeps track of when sources were most recently harvested. A new database model called LastHarvest keeps records of each source and the most recent date when it was harvested. The database is updated every time a new document from a more recent date is normalized. Code updates include a new view and url to view the database information for most recently updated sources.

jeffreyliu3230 commented 8 years ago

Can you add tests to this feature? We normally would require tests to be implemented for every new feature.