Improve the deduplication.

dice-group / Squirrel

Squirrel searches and collects Linked Data

Other

22 stars 19 forks source link

Improve the deduplication. #113

Closed sritejakv closed 4 years ago

sritejakv commented 5 years ago

Implementation for improving the deduplication.

Integrating deduplication into the updated crawler implementation.
Moving the endpoint to sparql.
Scrapper adds generated uris into metadata.
Calculate hash values for triples of generated uris fetched from metadata.
Compare/Update the duplicate triples found.

MichaelRoeder commented 5 years ago

We introduced an .editorconfig file (see https://github.com/dice-group/Squirrel/blob/develop/.editorconfig). Please copy it to your project and configure your IDE to use it. This should fix most of the changes which were introduced because of different formatting.