search

hpc / mpifileutils

File utilities designed for scalability and performance.
https://hpc.github.io/mpifileutils
BSD 3-Clause "New" or "Revised" License
169 stars 68 forks source link

Bash completions #244

Open gonsie opened 5 years ago

gonsie commented 5 years ago

It would be great to have tab-completion work for the various commands.

LiXi-storage commented 5 years ago

I am wondering whether it makes sense to implement this in Python language. If that is acceptable, I could implement one with relatively smaller effort due to the former work that I have done in the other projects. I choosed Python because it turns out similar auto-completion would need much more effort in C.

The auto-completion affect that can be implemented using the similar codes would be something like following. After running "./mpifileutils" to go to interactive mode. And we can select sub-command (e.g. dcp, dcmp etc) in the command. And of course, the sub-command can be auto-completed when typing tab. And after selecting sub-command, e.g. dcp, all the options (i.e. -d -g -i -p etc. for dcp) can be autocompleted. And not only command options, the file paths can be auto-compelted too.

This would need a lot of work. And quite some Python codes will be introduced. Thus need to know whether that is what everyone wants for this project. :)

gonsie commented 5 years ago

I don’t think the ‘interactive mode’ use case is very helpful for this project (except maybe for the dsh tool). I know there is some overhead of interaction with multiple MPI tasks, which is probably too much of a complication.

Out of curiosity, what is the name of the python tool?

Thank you very much for volunteering! We would welcome your contributions elsewhere in this project.

LiXi-storage commented 5 years ago

No problem. I will look for other things that I can help. :)

The tool is a series of commands for Lustre on Demand which is used for fetching and syncing the data between a global large (slow) Lustre and a small quick Lustre that are used for caching for each job.

adammoody commented 5 years ago

@LiXi-storage , thanks for offering to help.

We've had some people asking for improved support on sparse files, e.g., https://github.com/hpc/mpifileutils/issues/294. I just created some new issues of areas where I think we could benefit. At LLNL, we don't have a lot of users who write sparse files nor do we have the file systems needed to fully test the code, e.g., asking for file extents. Since you originally contributed this feature, I was wondering if you'd be willing to look to extend it to other tools and to test it.

Similarly, I think the group lock optimization could be extended to other use cases, perhaps it makes sense to add it to dbz2 or dstripe? Again, that's one that we can't test at LLNL, and so it's not easy for us to develop.