search

cc-archive / open-ledger

Prototype code and examples for work on the Creative Commons "CC Search" project
MIT License
48 stars 23 forks source link

Create loader for Flickr 100M dataset in AWS RDS database #35

Open lizadaly opened 7 years ago

lizadaly commented 7 years ago

Following whatever works best for #34, do the same, but for the much larger Flickr100M set:

s3://cc-openledger-sources/flickr100m/

lizadaly commented 7 years ago

Data now hanging out in that bucket:

$ aws s3 ls s3://cc-openledger-sources/flickr100m/
2016-10-12 17:00:37  717384403 yfcc100m_dataset-0.tsv
2016-10-12 17:08:53  717387993 yfcc100m_dataset-1.tsv
2016-10-12 17:17:39  717381670 yfcc100m_dataset-2.tsv
2016-10-13 09:28:39  717386715 yfcc100m_dataset-3.tsv
2016-10-13 09:36:32  717378750 yfcc100m_dataset-4.tsv
2016-10-13 09:44:33  717377568 yfcc100m_dataset-5.tsv
2016-10-13 09:52:30  718627725 yfcc100m_dataset-6.tsv
2016-10-13 10:00:42  718650892 yfcc100m_dataset-7.tsv
2016-10-13 10:08:48  718663933 yfcc100m_dataset-8.tsv
2016-10-13 10:16:54  718652455 yfcc100m_dataset-9.tsv
pa-w commented 6 years ago

Flickr 100M dataset integration