search

weecology / retriever

Quickly download, clean up, and install public datasets into a database management system
http://data-retriever.org
Other
306 stars 132 forks source link

List of Data sites to include #1218

Closed henrykironde closed 3 years ago

henrykironde commented 5 years ago

Feel free to edit and add more dataset links or sources.

  1. https://freegisdata.rtwilson.com/ (GIS Datasets)
  2. https://data.gov/
  3. https://data.worldbank.org/
  4. http://data.un.org/
  5. https://piktochart.com/blog/8-useful-databases-to-dig-for-data/(reference point)
prabhjyot28 commented 5 years ago

Data-One Another good site to find datasets.

DumbMachine commented 5 years ago

EU Open Data Portal Public Data by Google Unicef Data Resources Wikipedia Dump

henrykironde commented 5 years ago

https://edg.epa.gov/metadata/catalog/main/home.page

hrs2203 commented 4 years ago

https://www.kaggle.com/datasets https://www.figure-eight.com/data-for-everyone/ https://registry.opendata.aws/

ethanwhite commented 4 years ago

Kaggle required logging in to download data the last time I looked, which won't work for our current tooling. If they've implemented API key based access (something they were talking about the last time I talked with them) then we could talk about adding functionality to allow users to specify an API key for datasets that require it.

DumbMachine commented 4 years ago

Kaggle has a very solid, cli based api for downloading and uploading datasets. The api requires an access_token in the ~/.kaggle folder. User could allow retriever package to read that access_token and accordingly perform the required actions or we could take the key as a parameter.

ethanwhite commented 4 years ago

@DumbMachine - ah that's great! I'm :+1: for adding support for reading the api key and then adding the Kaggle datasets. When adding this support we'll need messaging to users that indicates that they need to get the api key if they try to download a kaggle dataset and don't have one. Would you be up for opening an issue describing this feature addition?

DumbMachine commented 4 years ago

@ethanwhite I would love to open the issue and describe it further. I'll do it soon.

henrykironde commented 3 years ago

Added to the latest data collection # 1575