numtide / treefmt

one CLI to format your repo [maintainers=@zimbatm,@brianmcgee]
https://treefmt.com
MIT License
575 stars 35 forks source link

Improve the E2BIG detection #127

Open zimbatm opened 2 years ago

zimbatm commented 2 years ago

Is your feature request related to a problem? Please describe.

This is a follow-up on #10 . In ab23c9f73ef878e28f070698f8328e2e9f547a16 I limited the number of arguments to 1024, which is a conservative value that should work across all systems.

Describe the solution you'd like

Implement a mechanism that selects something closer to what is supported in the system. On my machine, getconf ARG_MAX returns 2097152 for example. I think that's the number of bytes total that argv can take, including the \0 separators.

Describe alternatives you've considered

Change the treefmt formatter interface to take the list of files to format from stdin instead. It might be good to do that in the future, but formatters don't generally support that method. I would still be up for allowing to select the interface per formatter.

Additional context

User report:

used to run in a few ms on my test tree now it's ~1s

Mic92 commented 2 years ago

Here is how busybox solves it: https://elixir.bootlin.com/busybox/1.34.0/source/findutils/xargs.c#L666

zimbatm commented 2 years ago

Thanks, I went hunting for a rust xargs and found this relevant issue: https://github.com/rust-lang/rust/issues/40384

zimbatm commented 2 years ago

Maybe a better heuristic is to invoke the command, and split it into two commands on E2BIG, recurse.