Ideas for networking intelmq instances

certtools / intelmq

IntelMQ is a solution for IT security teams for collecting and processing security feeds using a message queuing protocol.

https://docs.intelmq.org/latest/

GNU Affero General Public License v3.0

975 stars 296 forks source link

Ideas for networking intelmq instances #901

Open aaronkaplan opened 7 years ago

aaronkaplan commented 7 years ago

Some ideas from @otmarlendl

What happens when we create a bigger network of intelmq instances? We might have loops.

Proposed solutions:

add UUID field to eventdb. Only add events to a local instance if the UUID does not exist yet.
add some kind of UUCP (she-bang) path . Example: !cert.at!bsi!nic.cz. This allows loop detection.

@otmarlendl thinks, both approaches should be taken in parallel.

dmth commented 7 years ago

regarding the use of UUID's https://github.com/certtools/intelmq/issues/543#issuecomment-233561823

waldbauer-certat commented 4 years ago

Talking about UUID's I would recommend something similar to https://discord.com/developers/docs/reference#snowflakes. Its built for concurrency and work pretty neat.

According to the discord documentation its built like		Timestamp	Internal worker ID	Internal process ID	Increment
Binary	111111111111111111111111111111111111111111	11111	11111	111111111111
From - To Bits	63 - 22	21 - 17	16 - 12	11 - 0

If we're exchanging data over network, it might break this system due to duplicates. Possible solution is to define a "transfer" protocol, which isnt built upon this id.

ghost commented 3 years ago

For UUIDs, maybe CyCat can be useful: https://cycat.org/ https://github.com/CyCat-project

ghost commented 3 years ago

Will be solved as part of IEP04