search

openknowledge-archive / datapackage-from-remote-js

Create a Data Package from data at a remote location.
https://www.npmjs.com/package/datapackage-from-remote
2 stars 4 forks source link

Custom remotes #2

Open pwalsh opened 9 years ago

pwalsh commented 9 years ago

User should be able to pass in a custom remote. Default remote is CKAN 3.

To do this we might need to change the structure of the remote config object a bit.

gvidon commented 9 years ago

@pwalsh we could do this by allowing user passing in optional callback which will be used for data mapping instead of datapackage-remote-js internal config. But this way the only role of datapackage-remote-js is just to download data and perform user callback over it — it's hard to believe that anyone will us it like this. I may be wrong and miss some points.

pwalsh commented 9 years ago

@gvidon I see your point.

The reason I have datapackage-from-remote as a distinct library is to include more "builtin remotes", like the one we have for ckan 3. So for example, we may add a "socrata" remote, or an "openspending 1.0" remote.

So then, the utility of the library is that it is aware of all the major open data portals, and has built-in mapping from those various formats, into Data Package format (and specifically, a particular Data Package Profile, like base, tabular, etc).

So, when I'm also suggesting we allow the user to add a custom remote, it is in the context of seeing this library developing to support a range of data portals so we can migrate that data into Data Package, but also allowing a user to add their own remote if there is one we don't have.

gvidon commented 9 years ago

@pwalsh ok, I see.