search

mcollina / hyperemitter

Horizontally Scalable EventEmitter powered by a Merkle DAG
ISC License
70 stars 2 forks source link

Are schemas shared #3

Closed mcdonnelldean closed 9 years ago

mcdonnelldean commented 9 years ago

If I have a schema A and Schema B on two different emitters that are connected to each other are they added together?

mcollina commented 9 years ago

No, it's currently like:

don't cross the streams

We might to handle that case somehow. Any ideas? I'm not writing the schemas in the database.

mcdonnelldean commented 9 years ago

Ha! That's what I thought. I need to come up with some use cases for this so we can discuss them.

mcdonnelldean commented 9 years ago

@mcollina

Could the emitter it's report it's schema to peers and then additively add it? I'm still not sure on this though because it would only make sense with two separate services as opposed to two horizontal ones (which would have the same schema).

So maybe this isn't the issue, maybe the real issue is when two HyperEmitters running on separate services want interface with each other. Perhaps the right way to do this is have some sort of hand over where they are not connected the same way horizontal ones are.

My concern is we are venturing outside of the original use case. It might be more astute to build a service on top of hyper emitter that manages this and never crosses streams.

mcollina commented 9 years ago

I think the case we should handle is that unknown events does not throw, but are replicated nevertheless.

mcdonnelldean commented 9 years ago

Hmm, so only use the schema some of the time? Outside of lovely small encoding are the schema's are used for anything else?

mcollina commented 9 years ago

Only lovely small encoding.

mcdonnelldean commented 9 years ago

So if we got lovely small encoding outside of this would you drop the schema as a hard requirement? As in if you have a protobuf schema great, we will validate against that. If you want to use a different encoder that's ok too?

Personally, for my projects I would rather not use protobuff as I don't really require schema validation. This would also allow us to handle an mark non protobuff messages with an unverified flag?

mcollina commented 9 years ago

The thing with protobuf is that's fast, really really fast. I'm ok for an optional schema for messages, I think we should support JSON and msgpack5.

mcdonnelldean commented 9 years ago

Awesome, I think we should close this in favor of the following tickets:

Thoughts?

mcollina commented 9 years ago

Go ahead, close this one and open two new ones.

mcdonnelldean commented 9 years ago

Awesome