Deduplicate transform improvements

CivicActions / edscrapers

US Department of Education Data Scraping Kit; see https://us-ed-scraping.ckan.io/dataset

GNU Affero General Public License v3.0

15 stars 9 forks source link

Deduplicate transform improvements #108

Closed osahon-okungbowa closed 4 years ago

osahon-okungbowa commented 4 years ago

TASK TACKLED

created a private helper function that allows querystring parameters to be flexibly stripped from url at the discretion of the developer

osahon-okungbowa commented 4 years ago

@nightsh For the meantime i included only 'referrer' to be removed/stripped from url during deduplication. But you can add more to the list as you investigate/discover