search

cea-hpc / clustershell

Scalable cluster administration Python framework — Manage node sets, node groups and execute commands on cluster nodes in parallel.
https://clustershell.readthedocs.io/
422 stars 85 forks source link

rcopy and strange file naming when using wildcard char #238

Open xdelaruelle opened 9 years ago

xdelaruelle commented 9 years ago

Hello,

I have just tried the --rcopy mode for the first time. Nice feature. However I use wildcard character to designate the remote file to copy and I end up locally with this wildcard character present in the resulting local file name.

$ clush -w $NODELIST --rcopy $VARDIR/-${date}.db --dest $VARDIR $ ls $VARDIR -20150218.db.node55

Regards, Xavier

thiell commented 9 years ago

That's not an easy one, as scp (more precisely scp2) supports remote wildcards expansion. But first, I should mention that the exact command should be :

$ clush -w $NODELIST --rcopy $VARDIR/\*-${date}.db --dest $VARDIR

...otherwise it could be expanded first on the local side.

Also, --rcopy was first intended to retrieve similar remote files and gather them locally. It becomes difficult to safely retrieve different files from different nodes, especially in term of error checking, eg. are you sure one .db file from node4424 is not missing?

As a workaround, for now, I can only suggest that you pre-build a list of remote files, something like:

clush -qS -w $NODELIST --rcopy  $(clush -N -w $NODELIST ls $VARDIR/\*-${date}.db) --dest $VARDIR

This command should work like the previous one, but remote filenames are kept. However, if remote files are different, error is printed on stderr, you could consider them as warnings.

thiell commented 9 years ago

We may have a solution: the idea is to pass a local directory destination to the underlying scp command, so that we can let scp do the wildcard globbing. However, it's not possible to tell scp to add a suffix to local files (like .host), so we plan to create temporary directories using node names during copy. All copied files should be renamed afterwards.

brianjmurrell commented 4 years ago

Any progress on this one?

thiell commented 4 years ago

No progress on this at this point, @brianjmurrell. But seeing interest from users can guide us to prioritize the resolution of such issues, so thank you for letting us know you would be interested in having this problem fixed. I'm keeping version 1.9 as the target milestone. If your use-case is different than the OP, please let us know.