Support more open datasets

utdemir / distributed-dataset

A distributed data processing framework in Haskell.

BSD 3-Clause "New" or "Revised" License

114 stars 5 forks source link

Open utdemir opened 4 years ago

utdemir commented 4 years ago

This is an umbrella issue to gather useful datasets which can be used freely.

Things to consider:

We should be able to access them freely & quickly. It helps if the data is provided in a partitioned fashion (a set of files).
It should be sufficiently large. There's no point using distributed-dataset if the data is smaller than ~10 gigabytes. It's even more impressive if the data is hundreds of gigabytes/terabytes (eg. gharchive).

Useful Collections:

utdemir commented 4 years ago

utdemir commented 4 years ago

petabytes (partitioned by date), mix of uncommon formats (WARC, WAT, WET...) , HTTP & S3

utdemir commented 4 years ago

utdemir commented 4 years ago

It might be an interesting experiment to implement a dataset of the Bitcoin transactions, if we have a way to process them in a partitioned way.

aycanirican commented 4 years ago

utdemir commented 4 years ago

https://www.data.gov/ has a lot of open datasets. They tend to be small on size, but there probably are exceptions.

utdemir commented 4 years ago

Tons of taxi trips, the partitioning seem ideal for distributed-dataset. Will have to investigate how performant the website is.

arobertn commented 3 years ago

Not mentioned (as far as I could see) in google's list:

Googling for "google ngram data" will turn up lots of scripts various people have developed for munging this data.