search

qri-io / qri

you're invited to a data party!
https://qri.io
GNU General Public License v3.0
1.1k stars 66 forks source link

data transformation engine for ingesting other data sources #812

Closed ghost closed 4 years ago

ghost commented 5 years ago

Benthos is a golang project can can transform any data from various sources, and i think a starlark wrapper would be hugely valuable.

https://github.com/Jeffail/benthos https://github.com/benthosdev/benthos-lab

vid: https://www.youtube.com/channel/UCjIYEhBrw3GQwpRWe1asufg/videos

This would make it easy to also watch the foreign data sources and the Benthos transformation can run and only pick up new data and pump it into qri as a new revision of the data. SO its all should be self running essentially.

Benthos is plugable also.

b5 commented 5 years ago

Looks really interesting. I'll have a look at Benthos & report back here with a set of possible paths to integration. Hope to get to this this week

b5 commented 4 years ago

Looked at Benthos, it's great! Not so sure we need to write an integration at this time tho