search

18F / open-data-maker

make it easy to turn a lot of potentially large csv files into easily accessible open data
Other
200 stars 135 forks source link

Support multiple endpoints #55

Open ange3 opened 9 years ago

ange3 commented 9 years ago

Currently there is only one api contained in the data.yaml file. However, in the future, there will probably be multiple endpoints. Support in config and the data.json endpoint (and potentially other files) is needed for this.

jmcarp commented 9 years ago

This would be a great addition. I'd be happy to implement if this is of interest.

One option would be to add more structure to data.yaml. For example:

files:
  cities:
    - cities1.csv
    - cities2.csv
  landmarks:
    - landmarks.csv

or

cities:
  files:
    - cities1.csv
    - cities2.csv
landmarks:
  files:
    - landmarks.csv

Alternatively, the files and metadata for each API could live in separate folders, S3 buckets, etc. Under this scenario, the cities endpoint might live at s3://cities, and the landmarks endpoint at /data/landmarks, each with its own data.yaml file and set of data files. Users could have the application fetch both directories by setting DATA_ENV to a list, like s3://cities, /data/landmarks.

For my use case, the second option would be a better fit. It also has the appeal of simplicity: the data.yaml schema doesn't change, and different indexes aren't constrained to pull from the same source. It would also be very helpful if users didn't have to create a potentially complicated data.yaml file, or any metadata at all, to create multiple endpoints (see #69).

Looking forward to your thoughts @ultrasaurus! cc @gbinal