Fetching, compiling, making and running on AWS

[jeremych1000]

Currently my partner and I are editing the source files on our local machines, commiting to git, fetching them using AWS, then running them using AWS as it's the only way to ensure both TBB and OpenCL work.

Does anyone have any ideas on a better workflow or any scripts or programs that automate this process? It gets messy having to restart instances (though working on t2 micro means we can leave it on but no actual GPU), and type all the git commands, while also having a billion commits each containing a single line change just because I want to debug something...

(plus we're both vim/emacs noobs)

[JHertz5]

You can edit source files on the AWS machine using the "nano" command line editor. Hopefully that helps at least part of your problem.

From the final section of aws.md in hpce-2016-cw2:

I would not recommend doing much editing on the remote instances, it should be more for testing, tuning, and experimentation. But inevitably you will need to change some source files, and need some way of editing the files remotely (you don't want to be pulling for each edit). There is a command line editor called nano installed on pretty much all unix machines which you can use to make small changes. For example, to edit the file wibble.cpp, just do:

nano wibble.cpp

You'll end up in an editor which behaves as you would expect. Along the bottom are a number of keyboard short-cuts, with ^ representing control. The main ones you'll want are:

^X (ctrl+x) : Quit the editor (it will prompt to save).
^O (ctrl+o) : Write the file out without changing it.
^G (ctrl+g) : Built-in documentation.

Other text editors can be used or installed (emacs, vim, ...), but I would suggest nano for most tasks you will encounter here.

[m8pple]

I associate myself with @JHertz5 's comments.

Generally speaking, you should try to develop a clear idea about what is currently limiting performance, and then pursue that locally. Only once you have a full idea up and running (and correct) on your local machine do you need to think about running things on AWS. So if you ever find yourself starting an AWS instance in order to debug, then your dev strategy probably needs tweaking.

[jeremych1000]

@JHertz5 Yup, thanks for reminding me of nano, seems like a good starting point to use while doing small edits.

@m8pple I'm in an awkward spot of WSL not being able to run OpenCL kernels on GPU and the time required to setup TBB + OpenCL on CLion, while learning CMake, is very relatively high? AWS was a fallback as it runs both without problem. Would you suggest developing locally on Windows (so spend time setting everything up) or developing locally on Windows but not compiling (as nothing is setup) and running on AWS?

[liafalk]

I've successfully gotten OpenCL (Nvidia in my case) to compile in g++ with MinGW, as it can link the .lib file required. I don't remember it taking too long to set up either. IIRC there's also a package for tbb for mingw/msys. I would definitely recommend doing it locally, as you're billed per hour rounded up. Even starting an instance for one second counts as an hour, as I found out the hard way... (let's say I've used a considerable chunk of my $150 credit)

[plus44]

You can copy files directly to your AWS instance using scp, thus working around git for small changes. For folders add the -r argument. scp -i mykey.pem somefile.txt root@my.ec2.id.amazonaws.com:/usr/local/hpce

[m8pple]

As @indigo- suggests, MinGW is pretty good as a windows platform, with good support for tbb and opencl. There is a 64-bit image that I linked to from the platform pages from the first coursework that should work well (tested in level-5 lab machines, at least).

While I like CLion for big projects, for this kind of work you probably don't need the complexity, so MinGW+make+text-editor is probably enough (it's how I developed it, except using a VM). When working on one puzzle there are only a few files you need for context, so an IDE like CLion doesn't help as much. (Well, I suppose context-sensitive help, but I tend not to use those).

[jeremych1000]

Thanks all.

[jeremych1000]

Settled on using FileZilla to connect via SFTP as I've used it before in the past. ref