search

dat-ecosystem-archive / datproject-discussions

a repo for discussions and other non-code organizing stuff [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
65 stars 6 forks source link

data pipeline configuration proposals #16

Open max-mapper opened 9 years ago

max-mapper commented 9 years ago

entries we have so far:

the idea is to use a runtime like https://www.npmjs.org/package/gasket to run these. they are supposed to be less like full scripting languages and more a data structure/configuration that defines the relationship between unix commands in a pipeline

cc @karthik @sckott

max-mapper commented 9 years ago

Update

We have drafted a combined proposal here: https://github.com/datproject/gasket/issues/17

@melaniecebula has updated the hackfile parser to reflect the proposal: https://github.com/mafintosh/hackfile

Now @melaniecebula is working on a hackfile -> gasket converter https://github.com/melaniecebula/datscript#usage and will also be updating gasket to implement the new functionality required by the proposal.

max-mapper commented 9 years ago

food for thought http://vincebuffalo.com/2013/08/08/the-mighty-named-pipe.html

karthik commented 9 years ago

Good stuff!

On Tue, Mar 3, 2015 at 12:06 PM, Max Ogden notifications@github.com wrote:

food for thought http://vincebuffalo.com/2013/08/08/the-mighty-named-pipe.html

Reply to this email directly or view it on GitHub: https://github.com/datproject/discussions/issues/16#issuecomment-77023503