Trouble with copying files from a remote machine

[yaaminiv]

To better inform our mass spectrometry, I'm using @Ellior2's data to create histograms of m/z ratios. The goal is to use the histogram to set a range of m/z ratios on the machine that would fit our data better.

I was able to successfully ssh into emu and I found Rhonda's files.

With the files I select, I need to add a column for an m/z ratio and export the file as a spreadsheet instead of an .xml. I figured the easiest way to do this would be to copy them over to my local machine first, then mess around with the files. I tried copying one file over using the following code:

ssh srlab@emu 'cat interact-20161205_sample_12.pep.xml > Documents/School/interact-20161205_sample_12.pep.xml

This however, seems to be taking a long time. Any suggestions on something that could expedite the process? Or is there any way I could manipulate the files in emu? Not too familiar with ssh so I'm not sure what I can/cannot do.

[seanb80]

How do you need to manipulate the files on Emu?

You can do most anything you can do in terminal over ssh, except for open graphical application.

Otherwise, there is an RStudio server instance, if you're familiar with R. You can open http://128.95.149.195:8787 in your local browser and use srlab/scallop as your login and password and it operate just like an RStudio server on your local machine.

If you're doing anything processor intensive, the slowness may be my fault, I'm working through Hollie's geoduck methylation data and that's fairly processor and I/O intensive.

[yaaminiv]

Theoretically, I need to calculate the m/z ratio using some information in the document. That m/z ratio needs to be its own column. I would then have to export the file as a spreadsheet and use that information to create a histogram (I was going to create the histogram in R Studio on my local machine). Emma had a way of doing this through her own system that just clicking a few buttons, so I'm not sure how that translates to file manipulation in emu.

[seanb80]

Oh! And scp works over ssh as well with a little change. ex.

scp srlab@emu:/home/Documents/rhonda/filename /path/to/local/directory

It will ask you for the srlab password and should happily copy.

[yaaminiv]

Hm.. just tried using the code scp srlab@emu: ~/Documents/rhonda/interact-20161205_sample_12.pep.xml /Users/yaaminivenkataraman/Documents/rhonda to copy over a file, but I got the following error: /Users/yaaminivenkataraman/Documents/rhonda: No such file or directory

However, the directory clearly exists!

Any suggestions?

[seanb80]

I just tried

scp srlab@emu:/home/srlab/Documents/rhonda/interact-20161205_sample_12.pep.xml ~/Desktop

and it worked. It's likely that space between the : and the ~

Otherwise, you might try using absolute paths (/home/srlab/Documents/rhonda/ as opposed to relative paths (~/Documents/rhonda).

[kubu4]

To help clarify some Terminal terminology.

SSH allows you to use Terminal on another computer. It's as though you're actually sitting at the remote computer (in this case, you'd be sitting at Emu). As such, there's no way to move files between your laptop and Emu directly; you need to use a special command. As @seanb80 pointed out, you need to use the secure copy command (scp). However, when using the scp command, you need to take note of which Terminal you're using (e.g. your laptop or Emu). This is likely what is causing your issue above.

If you run scp while you're in a SSH session (i.e. sitting at Emu), your command is now "backwards" because you're actually telling Emu to try to log in to itself (which it won't know how to do).

To utilize @seanb80 command, open a new Terminal window/tab on your laptop and then run the command (do not SSH into Emu).

[kubu4]

To continue with clarification of how to use the shell, your "cat" command is not the appropriate way to copy files. You should use the copy command (cp). Why? I don't know, it's just a how it should be done (why else would there be a dedicated command for copying files?). Actually, it has some technical aspects of preserving some file permissions as well as matching the exact bytes within a file. Additionally, it is also more computationally efficient. None of these factors will impact what you're doing, but it's good practice to use the proper tools.

[yaaminiv]

Alright I opened a new Terminal window and used the code

scp srlab@emu:/home/srlab/Documents/rhonda/interact-20161205_sample_12.pep.xml /Users/yaaminivenkataraman/Desktop

and got the following error:

ssh: Could not resolve hostname emu: nodename nor servname provided, or not known

My other Terminal window with the ssh is still open. Is there something else I need to do?

[kubu4]

Let's start at the beginning. Please paste the command you used to SSH into Emu.

[yaaminiv]

Used this to SSH into Emu: ssh -N -L localhost:8000:localhost:7000 srlab@128.95.149.195

Then opened a new window and pasted the following code ssh srlab@128.95.149.195

[kubu4]

OK, I thought that would be the issue.

In your scp command, you need to replace "emu" with the IP address for Emu (128.95.149.195).

[yaaminiv]

It works! Thanks @seanb80 and @kubu4 !

[kubu4]

To help clarify with what your commands are doing, here's a brief rundown:

ssh -N -L localhost:8000:localhost:7000 srlab@128.95.149.195 - This command creates a tunnel between your computer and Emu. Do you know why you ran this? (I have a guess, but want to get an idea of what you're trying to do.)

ssh srlab@128.95.149.195 - This is your standard SSH command and is what essentially has you "sitting" at Emu.

Finally, here's a "pro tip". If you want to use the word "emu" instead of the long IP address, add this line to the "hosts" file. The "hosts" file is located at: /etc/hosts.

128.95.149.195 emu

Once you've done that (and possibly restarted Terminal), you can type in ssh srlab@emu instead of ssh srlab@128.95.149.195

[yaaminiv]

Gotcha. I think I created the tunnel because I was following some instructions Rhonda had laid out in a previous Github issue. Is there no need to create this tunnel?

[kubu4]

If all you're doing is SSH, then no need for the tunnel.

The tunnel is sort of a "specialized" SSH technique. For our lab, we can use tunneling to operate a Jupyter notebook or RStudio Session on a remote computer, but use the web browser on our own computer to view, and use, Jupyter notebooks or RStudio sessions (even though they're actually running on a remote computer).

[yaaminiv]

Oh okay! Thanks for all of the help @kubu4!

[sr320]

To clarify for me - all you needed was access to a single file of Rhonda's on emu. You could run analysis on your own machine or did analysis need to occur on emu?

[yaaminiv]

@sr320 I just needed access to her files. No need to run any analysis on emu

[emmats]

Is there a public link to this server that can be accessed in a web browser? Or is it only accessible via command line?

[sr320]

There really needs to be a public link that can be accessed in web browser.

On Tue, Jan 17, 2017 at 12:45 PM emmats notifications@github.com wrote:

Is there a public link to this server that can be accessed in a web browser? Or is it only accessible via command line?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/sr320/LabDocs/issues/431#issuecomment-273293997, or mute the thread https://github.com/notifications/unsubscribe-auth/AEPHtz0Hld7q6RWiQx28Xpkun-KXuNoUks5rTSg8gaJpZM4LlMA_ .

[kubu4]

Isn't that what Owl is for?

[emmats]

I am used to working with a pep.xml and prot.xml viewer in a web browser (see attached). There are a lot of things that are useful about this, including ease of use and quality checking chromatograms.

[yaaminiv]

I couldn't access the files via Owl (see here: https://github.com/sr320/LabDocs/issues/430). Not sure if having a link through Owl would allow me to manipulate the files like Emma describes above

[emmats]

Also remember that the pep.xml and prot.xml files are not actually in that directory that Yaamini pointed to. I don't care about the raw files.

[kubu4]

@yaaminiv What do you mean you couldn't access the files? You marked that as closed (and thanked me) after I provided you with the link. I assumed that resolved your issue.

[yaaminiv]

@kubu4 I was able to access the .xml files that were in emu (they're now on my Desktop).

@emmats Just to clarify, the .raw files are on Owl (http://owl.fish.washington.edu/trochophora/) and the pep.xml and prot.xml files are on emu.

[kubu4]

@emmats - Oh, I see what you want. It's possible this could be done using tunnelling (this is how we can run Jupyter notebooks on a remote computer, but use/view them in a web browser on our own computer). However, it depends on the settings for the pep viewer program. What pep viewer are you using?

[emmats]

@kubu4, I have no idea. It just works when I click on a link :) I just emailed Jimmy Eng at UWPR to see if he has any pointers.

[kubu4]

You have no idea what pep viewer program you use?

[emmats]

I use google chrome!

Can we possibly move one of the pep.xml files to owl and see if that does anything useful?

[kubu4]

Also, all of this is way off topic and has nothing to do with difficulties copying files. A new issue should be opened.

[emmats]

True, but @yaaminiv shouldn't have to copy the file to begin with. And you did resolve it well much earlier by telling her to use cp :)

[yaaminiv]

I'm having some trouble using scp to copy the pepxml2txt script from my computer to emu. I used the following code:

scp /Users/yaaminivenkataraman/Downloads/pepxml2txt srlab@128.95.149.195:/home/srlab/Documents/rhonda/

And got the following error:

/Users/yaaminivenkataraman/Downloads/pepxml2txt: No such file or directory

The file is in my Downloads folder and this is the path to the file that terminal gave me, so I'm stumped.

@emmats There's no file extension, and I don't seem to be able to use tab autocomplete...? (I could be trying all the wrong ways)