carverauto / threadr

🌎 OSS Real-time AI Data Analysis with GraphDB integration. 🔍
Apache License 2.0
17 stars 1 forks source link

data ingestion: need framework to pull in data #51

Open mfreeman451 opened 7 months ago

mfreeman451 commented 7 months ago

Currently we've built a connector for IRC and have plans to build more for other messaging platforms like Discord, Slack, etc, but we don't really have an answer for a customer that would be wanting to pull in data, structured or unstructured, from other sources.

We need to come up with some generic class/way to handle data in our database that aligns to a common schema, so it doesn't matter if the data came in from discord or a CSV that contains a million plus rows of tweets that were compiled by whatever means. Since we can't even anticipate that that format might be, we can create a framework for people to create their own transformation tools that put the data in the correct shape for threadr.