search

pkolano / shift

High performance/reliability local/remote file transfers with sync and tar capabilities
https://pkolano.github.io/projects/shift.html
Other
51 stars 19 forks source link

shift-mgr usage on non-NAS NASA hosts #2

Closed ejm4567 closed 2 years ago

ejm4567 commented 2 years ago

Hi,

Can shiftc/shift-mgr be used to transfer files between 2 non-NAS NASA hosts?

I get the following error message:

[emansky@XXX~]$ shiftc ~/bin/parseJMDLogs.pl ~/Data Unable to execute "shift-mgr --host=XXX --put": The user_dir setting must be configured

where XXX is the host at GSFC I am testing on.

I have shiftc, shift-mgr, sup and shift-bin all installed on /usr/local/bin, and I've created a ~/.meshrc in my account containing the full paths to the source and target directories.

What else am I missing?

Thanks,

--Ed

pkolano commented 2 years ago

yes, but only use the versions here on github and not files you may have copied from NAS since the NAS versions have various tweaks that aren't generally applicable and won't match the instructions here. Similarly, don't rely on the NAS KB instructions as those again rely on assumptions specific to NAS (e.g. use of the SUP). sup and ~/.meshrc are likewise not needed. That error means you have not configured either ~/.shiftrc or /etc/shiftrc and told shift where to store the metadata (or you copied the NAS shift-mgr, which has a different location hard-coded). shift-aux is the only file needed on the destination host (make sure it is in the default path so "ssh host shift-aux" works). See the "INSTALL" document of the distribution for full instructions.

If you are planning to copy millions of files in a transfer, make sure to install the binary Data::MessagePack perl module on the shift-mgr host (as mentioned in INSTALL) and build/install shift-bin on the host with the source files as these significantly speed up certain aspects at scale. Also, for large remote transfers, you would want to move fish-tcp first in the remote transport order in shiftrc or reorder with --remote (assuming your remote host allows inbound TCP connections) as that has the fastest performance (note data stream is unencrypted in this case so do not use for sensitive data). May need to specify --ports in this case if remote host has only a certain range open.

ejm4567 commented 2 years ago

Ahhh, that was it. I was missing ~/.shiftrc in my $HOME. Got the local copy to work once I added .shiftrc.

Thanks!

--Ed

ejm4567 commented 2 years ago

Closing issue since adding .shiftrc resolved the problem.