We need a data sink node service that runs on a small server device || phone

[truedat101]

Problem

1. We want a place to store either small mounts of data or large amounts of data (depending on the frequency of data sync and the kind of data)
2. We want the first choice to be a local data sink node
3. We want the data sink nodes to advertise themselves locally
4. We want the data sink nodes to use security for authenticating clients (acl list, PKI, or other mechanism using secure transport)
5. We want the data sink transactions to be reliable if required, however, default will be best effort reliability
6. We want the data sink nodes to have the ability to sync to upstream data services as an option (i.e. s3, onedrive, dropbox, etc...)
7. We want a data sink node to have the ability to visualize its data (#7 )

Definitions

• a data sink node: any destination that can store data on behalf of the user/device(s)
• data sink storage type: either blob or time series entry. Time series entry can be a metric or a log entry. A blob will be a data blob of some size, typically for larger text docs or binary such as an image.
• data sink capacity: the published capacity available for storage, typically will be finite, and style of storage may be a ring buffer if storage is extremely limited (i.e. might only store the most recent 100K of data)
• data sink api: the api exposed to applications to allow storage and retrieval of data
• data sink auth: the authentication method presented to applications to store/retrieve data
• data sink client: any device or app that sends data to sink

Open Questions

• How to sink smartly so that already previously sync'd data isn't sent
• Do we leave reliability of data sink to the client?

[truedat101]

Use fluentd apis and the GELF format (https://graylog.org/post/log-formats-a-complete-guide/) used in graylog. Gelf is super flexible for a data unit.

[truedat101]

We should focus on something that runs on linux first, and then worry about developing a phone based service.

[truedat101]

I am adding my forked repo of expresscpp to make it possible for us to bootstrap a C++ project easily.

[truedat101]

This needs some fixes, as there were upstream bugs. Will pull those in.