workshop 4: Running programs on remote computers and retrieving the results

[ctb]

Thursday August 12 from 9 am - 11:30 PDT

Instructors: Moderator: Abhijna Helpers: Nick

Zoom link:

Description:

This two hour workshop will show attendees how to use remote computers to run their analyses, work with the output files, and copy the results back to their laptop and desktop computers. We will discuss input and output formats, where files are usually read from and written to, and how to use the ssh software to copy files to and from remote computers.

no draft lesson yet

owner: ???

---

potential hands-on topics:

• running something like FastQC on binder, downloading via RStudio Server
• ssh file transfer
• file paths and making life easy for yourself (e.g. a 'transfer' directory :)

[ctb]

• topics/learning objectives
  • where files live
  • where programs run
  • what resources programs use when they run remotely
  • where stdout/stderr goes, where stdin comes from
  • how to specify remote paths for ssh
  • transferring local to remote (push)
  • transferring remote to local (pull)
  • why you often can't push from remote to local, but have to request instead (NAT, firewalls, etc.) - maybe use a vampire analogy? or too cheesy?
• prerequisites
  • ssh/connecting to remote computers
• assessments
  • some basic questions about connecting to remote computers, remote resources, etc.
  • some file path strings, local and remote

[ctb]

I cannot make the first half hour of this. We might consider moving this back by 30 minutes since I suspect we'll need all hands on deck.

[nick-ulle]

I can be a helper for this.

[ctb]

I think I would probably point them at some other approaches, too - cat /proc/cpuinfo for the system they’re on; free; and df -h for disk space

[ctb]

for the e-mail that gets sent out for this one (presumably today?),

• make sure they have been able to log into farm as in workshop 3
• also ask people to drop out if they are not planning on attending

[ctb]

Let's tell people to show up at 9:30am, too; that's the earliest I'm available.

[ctb]

things to remind people of in announcements -

• we'll start at 9:30am, not 9; will still end at 11:30am
• please make sure you're able to log in per workshop 3's lesson

...what else?

assessment q:

I can use a key pair with ssh instead of a password to connect to a remote UNIX system

I can restrict others from being able to see my files on shared UNIX systems

I can directly transfer files from a Web address to a remote system at the command line

[jeremywalter]

Pre-Survey: https://forms.gle/e4izXSt3nfM9xYR48 Post-Survey: https://forms.gle/Hny9nogihwCvidq58

[nick-ulle]

Section 4.1

the private part of a key pair is NEVER shared, unlike with passwords where you have to type the password in.

This sort of sounds like it's normal to share passwords.

---

Section 4.2

Your private key for your datalab-XX account is kept in .ssh/id_rsa

Clarify which machine this is talking about (Farm, not the local machine).

[abhijna]

fix the docs to include .profile edit

[nick-ulle]

Most of the students having trouble with scp had a corrupted .bashrc on Farm as a result of #3.

So maybe reset their files between workshops or be careful about how we tell them to edit config files.

[ctb]

On Thu, Aug 12, 2021 at 10:25:25AM -0700, Nick Ulle wrote:

Most of the students having trouble with scp had a corrupted .bashrc on Farm as a result of #3.

So maybe reset their files between workshops or be careful about how we tell them to edit config files.

ugh didn't see that coming! how was it corrupted - just, like, bad content that broke on login?

[nick-ulle]

Mostly incorrect syntax in the .bashrc. Typos and missing quotes

[abhijna]

point people at hpc slack channel for hpc help

[nick-ulle]

Relevant for getting people to ask good tech questions: https://stackoverflow.com/help/how-to-ask

[ctb]

On Thu, Aug 12, 2021 at 10:58:04AM -0700, Nick Ulle wrote:

Mostly incorrect syntax in the .bashrc. Typos and missing quotes

ok - I ran into one or two of these while installing conda for everyone, do they actually prevent logins? I thought they just printed out errors.

[nick-ulle]

do they actually prevent logins? I thought they just printed out errors.

They don't prevent login with ssh but they do prevent scp from downloading files.

[ctb]

do they actually prevent logins? I thought they just printed out errors.

They don't prevent login with ssh but they do prevent scp from downloading files.

wow, I had no idea - finally followed this up,

https://stackoverflow.com/questions/12440287/scp-doesnt-work-when-echo-in-bashrc

"yay" I learn something new every time I teach!?!?!?

[ctb]

Most of the students having trouble with scp had a corrupted .bashrc on Farm as a result of #3.

So maybe reset their files between workshops or be careful about how we tell them to edit config files.

fixed in https://github.com/ngs-docs/remote-computing-base/pull/13 - once merged, we should update this repo, too.]

EDIT: done and now live! https://github.com/ngs-docs/2021-august-remote-computing/pull/42