search

tbillington / kondo

Cleans dependencies and build artifacts from your projects.
MIT License
1.75k stars 51 forks source link

Look into parallelisation #8

Open tbillington opened 4 years ago

tbillington commented 4 years ago

Look into parallelisation options such as rayon, crossbeam, or even tokio for parallelisation. Much of what kondo does is able to be done in parallel.

Noah-Kennedy commented 4 years ago

I think the best approach here would likely be to try using tokio or async-std to get asynchronous IO. I might try spinning up a fork with this later this week when I have time.

tbillington commented 4 years ago

Neat :) I'd recommend looking for inspiration in ripgrep (https://github.com/BurntSushi/ripgrep/tree/master/crates/ignore). It does parallel file searching without requiring an entire async runtime and gets better performance for it. Not sure how complex it is though.

derekdreery commented 3 years ago

Another reason that multithreading would be good: on wayland the compositor expects to be able to ping the UI often. If the UI thread blocks (as is the case with the clean method), then the compositor sometimes kills the app. The solution is to do all work in a separate thread and communicate using messages.

tbillington commented 3 years ago

Ah that's interesting, I didn't know that!

tbillington commented 1 year ago

@derekdreery kondo switched to using multiple threads for the CLI in https://github.com/tbillington/kondo/commit/1dcc3684c7e2b249d736d60a7e174fa7e8834359.

It now uses 3 threads, one for running ahead of the user input that "discovers" the projects, one for presenting them to the user and receiving the users action, and one for cleaning the projects the user has chosen.

They communicate via channels and don't block each other. The only limit I've set is the discover thread will only find up to 5 projects ahead of the UI thread, mostly to avoid unnecessary work if the user exits early.

tbillington commented 1 year ago

I'm open to more parallelism, but since the CLI is now not blocked by discovery or cleaning it feels very snappy to use.

It's also very fast as is for my use case now, so I have less motivation for a solution based on multithreaded async work, but I'm still not opposed to the idea.

Noah-Kennedy commented 1 year ago

FWIW I disagree with my original statement from three years ago, as I know a lot more about the subject now.

If you want to do some sort of async thing, io_uring on linux would allow for more speed on that platform, but other than that you won't see gains from just slapping async onto things.