search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.63k stars 1.55k forks source link

Consider splitting Vector into many library crates #1941

Open ghost opened 4 years ago

ghost commented 4 years ago

@bruceg proposed in https://github.com/timberio/vector/pull/1924#issuecomment-591550535 to split Vector components into multiple isolated library crates, such as sinks, sources, etch to reduce built time of individual bits. This sounds interesting, although I think we need to spec out some kind of a plan before we get started.

binarylogic commented 4 years ago

If we do this I’d prefer that we split the code on logical boundaries that aren’t just Vector specific. We’ve been discussing with other Rust projects the possibility of sharing functionality, like file tailing.

MOZGIII commented 4 years ago

Related to https://github.com/timberio/vector/issues/1514

MOZGIII commented 4 years ago

If we do this I’d prefer that we split the code on logical boundaries that aren’t just Vector specific. We’ve been discussing with other Rust projects the possibility of sharing functionality, like file tailing.

I think it’s a different concern. It’ll still is a valuable thing to do though!

ghost commented 4 years ago

As a sidenote, an interesting question to explore is would Rust codegen LTO which we use for release builds work with project split into many library crates with the same efficiency as it does now.

MOZGIII commented 4 years ago

Do we do anything special with LTO? Can you link relevant repo parts here so I can follow?

ghost commented 4 years ago

@MOZGIII I mean this: https://github.com/timberio/vector/blob/0e4148e8d1150be536c032d2de8b33f5f5aab1df/scripts/build-archive.sh#L61-L75

Hoverbear commented 4 years ago

So the big danger here is that we end up spending more engineering time managing the cargo workspace than we save in build times on a monolith. :(

Cargo workspaces handle dependencies rather badly. That said, it might be an improvement over our use of features...

Hoverbear commented 4 years ago

@LucioFranco I know you've felt a lot of pain from feature flags, do you think this might trend us towards something better?

LucioFranco commented 4 years ago

I have a feeling that features flags will fall under the same thing as a workspace where we spend more time trying to make sure all the feature flag variations build correctly. I also feel like splitting everything into a workspace as well will be too much work.

I think for me from what I've seen is that linking is really the slow part for my workflow. I wonder if there are ways we can reducing linking. I am not sure but I would have to assume that we need to link in every dependency everytime we run test? Maybe reducing dependencies would help here?