Open henrykironde opened 4 years ago
Almost all the datasets at Datahub collection already uses its own datapackage.json
to define the schema of the data.
https://datahub.io/docs/core-data
https://frictionlessdata.io/data-packages/
So can we not build a module that can parses datapackage.json
and restructure it to our format.
I can work on that if this seems a good idea..
@pathak-mayurdeep, thats great. We should add all the datapackages here https://github.com/weecology/retriever/blob/master/scripts/datapackages.yml and then create the module
@pathak-mayurdeep Here's our existing (unmerged) work from a couple of years ago on using existing Frictionless Data data packages: https://github.com/weecology/retriever/pull/980 Feel free to build off this if helpful.
Or maybe this got included in #1010? @henrykironde - #980 suggests that everything went into #1010, but I don't remember us getting all the way to loading external packages. Do you remember the status of that work as of #1010?
We had reached a point where we could ingest some of the data but then specifications changed. Currently we need to create a dictionary for the data types.
Thanks for all the info.. I'll start working on this.
Sorry about the late PR, I was offline for several days due to some personal health issues..
I have used datapackage-py
in this. Please let me know whatever changes is needed.
Collections
Bibliographic data note:
The raw dataset is XML, a good example of adding ingestion for xml formats see https://github.com/weecology/retriever/wiki/GSoC-2020-Project-Ideas#approach
List of datapackages created
Climate Change
List of datapackages created
Demographics (population)
List of datapackages created
Economic Data and Indicators
List of datapackages created
Education
List of datapackages created
Football
List of datapackages created
GeoJSON
List of datapackages created
Health Care Data
List of datapackages created
Inflation
List of datapackages created
Linked Open Data
List of datapackages created
Logistics
List of datapackages created
Machine Learning / Statistical
List of datapackages created
Movies and TV
List of datapackages created
Open Corporates
List of datapackages created
Property Prices
List of datapackages created
Reference Data
List of datapackages created
Stock Market Data
List of datapackages created
War and Peace
List of datapackages created
Wealth, Income and Inequality
List of datapackages created
World Bank
List of datapackages created
YAGO List of datapackages created