Open xni opened 10 years ago
At first, you are to add yarn
user to the sudoers
file by appending a line yarn ALL=NOPASSWD:ALL
to the /etc/sudoers
file.
Then, install Docker: https://docs.docker.com/installation/ubuntulinux/#ubuntu-precise-1204-lts-64-bit
Create a test container with Python abroad:
$ sudo docker run -d ubuntu sudo apt-get install -y python
0630b6f59a20ce82e9d07ce9af9ef348a2837fbbac487b31c96bfda4fecf8dcc
$ sudo docker logs 0630b6f59a20ce82e9d07ce9af9ef348a2837fbbac487b31c96bfda4fecf8dcc
...
Setting up python (2.7.5-5ubuntu3) ...
$ sudo docker commit 0630b6f59a20ce82e9d07ce9af9ef348a2837fbbac487b31c96bfda4fecf8dcc
ee55efa7a5fff1c223db815b867d37400a53c69759dd205d958bc7579d876978
$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
<none> <none> ee55efa7a5ff 28 seconds ago 299.9 MB
ubuntu latest e54ca5efa2e9 4 weeks ago 276.5 MB
$ sudo docker run ee55efa7a5ff python -c "print('Hello from dockers guts.')"
Hello from dockers guts.
So, I will use ee55efa7a5ff
in my code.
Some modifications in nodes code:
private void run() throws IOException {
LOG.info("Running Container on {}", this.hostname);
// http://stackoverflow.com/questions/617414/create-a-temporary-directory-in-java
final File temp;
final File parent_dir = new File("/opt/docker-output");
temp = File.createTempFile("docker-", null, parent_dir);
temp.delete();
temp.mkdir();
LOG.info("Starting docker");
ProcessBuilder pb = new ProcessBuilder(
"sudo",
"docker", "run",
"-i",
"-v", temp.toString() + ":/opt/data",
"ee55efa7a5ff",
"python", "-c", "open('/opt/data/test.txt', 'w').write('Docker: Ok')");
Process docker_process = pb.start();
LOG.info("Docker is up. Waiting...");
try {
docker_process.waitFor();
LOG.info("Application is finished. Exit status: {}", docker_process.exitValue());
String stdout = slurp(docker_process.getInputStream(), 1024);
String stderr = slurp(docker_process.getErrorStream(), 1024);
LOG.info("Stdout: {}", stdout);
LOG.info("Stderr: {}", stderr);
} catch (InterruptedException e) {
e.printStackTrace();
}
try {
Thread.sleep(60000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
// http://stackoverflow.com/questions/309424/read-convert-an-inputstream-to-a-string
public static String slurp(final InputStream is, final int bufferSize)
{
final char[] buffer = new char[bufferSize];
final StringBuilder out = new StringBuilder();
try {
final Reader in = new InputStreamReader(is, "UTF-8");
try {
for (;;) {
int rsz = in.read(buffer, 0, buffer.length);
if (rsz < 0)
break;
out.append(buffer, 0, rsz);
}
}
finally {
in.close();
}
}
catch (UnsupportedEncodingException ex) {
}
catch (IOException ex) {
}
return out.toString();
}
Files are created,
$ head /opt/docker-output/*/test.txt
==> /opt/docker-output/docker-1435383504020414450.tmp/test.txt <==
Docker: Ok
==> /opt/docker-output/docker-2130967975752402865.tmp/test.txt <==
Docker: Ok
==> /opt/docker-output/docker-2887434490859823761.tmp/test.txt <==
Docker: Ok
==> /opt/docker-output/docker-5126749122784000365.tmp/test.txt <==
Docker: Ok
==> /opt/docker-output/docker-5945479457457188145.tmp/test.txt <==
Docker: Ok
But Java freezes on docker_process.waitFor();
for forever. So I'll perform some extra investigations.
I've changed -i
to -d
, and container start was successful:
14/07/20 09:30:41 INFO yarntest.MyContainer: Container just started on master/192.168.7.10
14/07/20 09:30:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/07/20 09:30:43 INFO yarntest.MyContainer: Running Container on master/192.168.7.10
14/07/20 09:30:43 INFO yarntest.MyContainer: Starting docker
14/07/20 09:30:43 INFO yarntest.MyContainer: Docker is up. Waiting...
14/07/20 09:30:44 INFO yarntest.MyContainer: Application is finished. Exit status: 0
14/07/20 09:30:44 INFO yarntest.MyContainer: Stdout: 3830f2f2236a9df06a4b516c12d5737345804b39578802c2109824e23bc82c1b
14/07/20 09:30:44 INFO yarntest.MyContainer: Stderr:
14/07/20 09:33:10 INFO yarntest.MyContainer: Container just started on master/192.168.7.10
14/07/20 09:33:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/07/20 09:33:14 INFO yarntest.MyContainer: Running Container on master/192.168.7.10
14/07/20 09:33:14 INFO yarntest.MyContainer: Starting docker
14/07/20 09:33:14 INFO yarntest.MyContainer: Docker is up. Waiting...
14/07/20 09:33:14 INFO yarntest.MyContainer: Application is finished. Exit status: 0
14/07/20 09:33:14 INFO yarntest.MyContainer: Stdout: 1fe97bd025b5229fc4ba52d1d40f28e684af77e94505bbed3088d96b6b4e036d
14/07/20 09:33:14 INFO yarntest.MyContainer: Stderr:
So, parsing this container ids and parsing docker ps
may be useful.
I don't know if starting Dockers directly from an application master is a good idea. If it is, okay, if no, we will start a Java application in Container, an it will start a Docker.