Due to a format change, the previous version was broken. We now use pandas to parse the CSV and be robust against future format changes. The current change also added a class field which is similar to a tag, which we now also import.
We do not import the country code field, since we do not know where it comes from and will probably just add redundant information.
Motivation and Context
Closes #133.
How Has This Been Tested?
Full run of the crawler.
Types of changes
[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
Checklist:
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
Description
Due to a format change, the previous version was broken. We now use pandas to parse the CSV and be robust against future format changes. The current change also added a class field which is similar to a tag, which we now also import. We do not import the country code field, since we do not know where it comes from and will probably just add redundant information.
Motivation and Context
Closes #133.
How Has This Been Tested?
Full run of the crawler.
Types of changes
Checklist: