Example implementation for the filters

dev-ardi commented 9 months ago

There's still a lot to do and think. This is just a draft to have an example to think about and work on.

This just is a list of composable filters that fit into another. This is really extensible, but there are a few things left to think.

There are many optimizations possible with this model.

I don't like having possible failures inside of the filters, everything should have been figured out by this time, but I've implemented it just in case

CosmicHorrorDev commented 9 months ago

I'll look at this more after #265 gets merged since I'm assuming that will change at least some of this PR

dev-ardi commented 9 months ago

@CosmicHorrorDev

will change at least some of this PR

Not really. This is atomic and only the "filter" part. There's still so much to do.

CosmicHorrorDev commented 9 months ago

I can take a look after I'm no longer sick

dev-ardi commented 9 months ago

Get better soon!

CosmicHorrorDev commented 8 months ago

Sorry for taking so long to give feedback

I think the main fundamental thing that will cause problems and have to be addressed is how to handle streaming in this model. Like in the case of something like

$ sd --pretend-we-have-a-line-mode 'bar' 'baz'
foo
bar
baz <-- output by `sd`

we need to be able to stream the output line by line which I think means that we'll need to have an api that looks more like the std::io apis where you dealing more with reading and writing and different layers can allocate as needed

Some various other thoughts:

The filter operations have to be fallible since we won't be able to resolve everything ahead of time. You already talked about how this is supported, but I just wanted to note that this a hard requirement
The fundamental type here should be bytes instead of utf8. We want to support being able to do byte operations for people and we want to be able to support different encodings as well
- That means that utf8 or any other kind of text decoding will have to be done on the fly instead of entirely up-front
- --preview is a special case where we want to only output utf8, so that could be represented as an extra Filter applied at the end

All that out of the way that doesn't mean we need something that can fit all of those needs now, but I think it does limit how Filter can be designed

chmln / sd

Example implementation for the filters #282