minio / mc

Unix like utilities for object store
https://min.io/download
GNU Affero General Public License v3.0
2.86k stars 548 forks source link

mc is way too verbose on operations on very large number of files (cp, mv, rm) #4804

Closed scherepanov closed 10 months ago

scherepanov commented 10 months ago

Expected behavior

Desired behaviour: mc operation on multiple files will output with --output=[all|files|summary|none] all - same as existing behavior files for each file and - no summary summary - only summary, no per-file output none - no output (name of parameter --output is not very good, you will come up with better name)

Actual behavior

mc display information for every file involved in operation and summary

Steps to reproduce the behavior

I just removed over million of files with mc rm --older-than mc output is unmanageable, with over million of lines in output. I would like to control what mc is producing on output. For operations on millions of files I would be interested only in summary, like number of files and their total size, both in JSON and regular output formats. In other case no output at all is desired, I am just using return code from mc. That include mc commands that can run on multiple files - rm, mv, cp I think it will speed up mc operations on very large number of small files, as output to stdout probably is a major resource consumer in operations on millions of files. Will you be open to PR with this change?

mc --version

System information

Linux

harshavardhana commented 10 months ago

Redirect the output to > /dev/null it you don't wish to know what the command is doing.

scherepanov commented 10 months ago

Quite opposite, I am very interested in getting command summary and not interested per-file info. Yes, I am redirecting to /dev/null. Current design would force to do some post-processing in pipe etc. Clean solution would be to allow users to control what mc is producing. I can see other users have same problems, see #4492