Add informative reference url and data modification time

InternetHealthReport / internet-yellow-pages

A knowledge graph for the Internet

https://iyp.iijlab.net

GNU General Public License v3.0

43 stars 18 forks source link

Add informative reference url and data modification time #125

Closed m-appel closed 8 months ago

m-appel commented 9 months ago

Description

As written in #121 we want to add an informative URL and the modification time for each data source if possible. This PR does the refactoring and adds these fields.

Motivation and Context

Resolves #121.

How Has This Been Tested?

Will be tested with a full create_db, once it's done.

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.

m-appel commented 8 months ago

This PR should be ready now, I tested it with a full create_db. There are some crawlers which we need to discuss at some point, but functionally should be fine.

One point: The reference_time_modification is None by default, i.e., the property will not exist if it is not set. The reference_url_info is an empty string, i.e., it is always there. We could also set this to None by default.