CRDC API

[metzgr]

Gray, you ran into some challenges trying to create an API with it. How should we proceed?

[gbinal]

There's a few issues with turning the complete datasets into an API without further handling.

• The autoapi engine is struggling to process the full files, either for size or breadth reasons.
• The data is expansive to the point that it becomes unwieldy to developers if offered up all at once.

Even the Discipline file, a subset of the complete dataset, is still 137M in size and has over 570 columns.

There's a few ways to handle this. I went ahead and created an API from a pared down version of the Disciple file, with just the first 48 columns.

It'd be possible to do this a bunch of times, but I'd actually suggest that we try to further curate the data into more discrete datasets that convey fewer things at a time.

[gbinal]

We can see how https://api.ed.gov/data/crdc2013_14disc-x?api_key=DEMO_KEY1 plays with folks. It might be easiest to do the equivalent of hallway testing with a few friendly developers, just to get some initial thoughts.

In the meantime, I'm going to go ahead and close this issue. But we can reopen and reevaluate it anytime.