search

18F / open-data-maker

make it easy to turn a lot of potentially large csv files into easily accessible open data
Other
199 stars 135 forks source link

Importer won't find data.yaml if there are too many files in the S3 bucket #250

Closed yozlet closed 8 years ago

yozlet commented 8 years ago

When DataMagic::Config looks for data.yaml it grabs the list of files in the S3 bucket and searches it for the filename. However, @s3.list_objects() only returns 1000 objects at a time, and if there are over 1000 objects in the bucket (because of backups/snapshots) then data.yaml won't be in that list and won't be loaded.

I suspect the best solution here is to not rely on file listing, and instead just assume that data.yaml is there and try to read it, taking an alternate path if that fails.

dnesting commented 8 years ago

+1 to "just try to do the thing" and not "see if we think the thing will work, and then try it" as a general rule.