In this example, we are going to run lambdas that grab PNG files stored on S3 as
mybucket:sintel-1k-png16/%08d.png
, encode them 6 frames at a time as Y4M files,
and upload them to mybucket:sintel-1k-y4m_06/%08d.y4m
.
If you want more information on running xc-enc, see src/lambdaize/README_xc-enc.md.
I assume that you've already got the mybucket:sintel-1k-png16/%08d.png
files. You should
get these from Xiph and upload them to S3.
I also assume you're using a Debian-ish system of recent vintage (I'm running Debian testing as of September 2016).
You will need the following packages:
apt-get install build-essential g++-5 automake pkg-config \
python-dev python-boto3 libssl-dev python-openssl \
libpng-dev zlib1g-dev libtool libtool-bin awscli
You'll also need an AWS ID, both for the
AWS CLI
and for the mu scripts (after you've run aws configure
, your credentials will be in ~/.aws/credentials
).
You will also need a lambda
execution role.
Put these in your environment now so that you don't forget!
export AWS_ACCESS_KEY_ID=xxxxxx
export AWS_SECRET_ACCESS_KEY=yyyyyy
export AWS_ROLE=arn:aws:iam::0123456789:role/somerole
To start, let's build the mu repository:
mkdir -p /tmp/mu_example
cd /tmp/mu_example
git clone https://github.com/excamera/mu
cd mu
./autogen.sh
./configure
make -j$(nproc)
The other thing we'll need is the daala_tools repo,
which contains the png2y4m
tool we are going to run on each lambda worker.
Important: note STATIC=1
in the make
invocation. The
lambda environment
probably does not have the same system libraries as our machine, so to be safe, we should only
use statically linked binaries on lambda workers.
cd /tmp/mu_example
git clone https://github.com/excamera/daala_tools
cd daala_tools
make -j$(nproc) STATIC=1
The next step is preparing a lambda function. Our goal is for the lambda to execute a command
like ./png2y4m -o /tmp/somefile.y4m /tmp/%08d.png
, which will convert PNGs to a Y4M. (Don't
worry, we'll figure out how the PNGs get downloaded below.)
To do this, we'll invoke the lambdaize.sh
script in the mu
repo:
cd /tmp/mu_example
MEM_SIZE=1536 TIMEOUT=180 ./mu/src/lambdaize/lambdaize.sh \
./daala_tools/png2y4m \
'' \
'-i -d -o ##OUTFILE## ##INFILE##'
MEM_SIZE
and TIMEOUT
are configuration options for the lambda function. Note that this
command will use AWS_ROLE
(see above) as the role for executing the lambda function we've
just created. The command's output looks something like:
{
"CodeSize": 3996942,
"LastModified": "2016-09-01T00:00:00.000+0000",
"MemorySize": 1536,
"CodeSha256": "yv+mJC0/2hsjTcu3BpFwWyhix1YVRimph8O1y8Oy/Lw=",
"Description": "png2y4m",
"FunctionName": "png2y4m_cP4Mf5pn",
"Role": "arn:aws:iam::0123456789:role/somerole",
"Handler": "lambda_function.lambda_handler",
"Runtime": "python2.7",
"Timeout": 180,
"Version": "1",
"FunctionArn": "arn:aws:lambda:us-east-1:0123456789:function:png2y4m_cP4Mf5pn"
}
Your new lambda function's name is png2y4m_cP4Mf5pn
, and you will find a correspondingly-named
zipfile in /tmp/mu_example
. lambdaize.sh
generates a random suffix and appends it to the
lambda function name to avoid collisions with existing functions. If you forget the name
of your function, you can invoke aws lambda list-functions
.
Finally, we will run a server to launch and coordinate the lambda instances. The full script is in mu/src/lambdaize/png2y4m_server.py.
Usage: ./png2y4m_server.py [args ...]
You must also set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY envvars.
switch description default
-- -- --
-h: show this message
-D: enable debug (disabled)
-O oFile: state machine times output file (None)
-P pFile: profiling data output file (None)
-n nParts: launch nParts lambdas (1)
-f nFrames: number of frames to process in each chunk (6)
-o nOffset: skip this many input chunks when processing (0)
-v vidName: video name ('sintel-1k')
-b bucket: S3 bucket in which videos are stored ('excamera-us-east-1')
-i inFormat: input format ('png16', 'y4m_06', etc) ('png16')
-t portNum: listen on portNum (13579)
-l fnName: lambda function name ('png2y4m')
-r r1,r2,...: comma-separated list of regions ('us-east-1')
-c caCert: CA certificate file (None)
-s srvCert: server certificate file (None)
-k srvKey: server key file (None)
(hint: you can generate new keys with <mu>/bin/genkeys.sh)
(hint: you can use CA_CERT, SRV_CERT, SRV_KEY envvars instead)
We will need to generate SSL certs:
mkdir -p /tmp/mu_example/ssl
cd /tmp/mu_example/ssl
/tmp/mu_example/mu/bin/genkeys.sh
Now we're ready to go!
/tmp/mu_example/mu/src/lambdaize/png2y4m_server.py \
-n 5 \
-l png2y4m_cP4Mf5pn \
-b mybucket \
-c /tmp/mu_example/ssl/ca_cert.pem \
-s /tmp/mu_example/ssl/server_cert.pem \
-k /tmp/mu_example/ssl/server_key.pem
That's it! You're encoding files.
Coordinating servers use the pylaunch
module to launch many lambdas at once in parallel.
This module is an interface to liblaunch.
Usage:
pylaunch.launchpar(num_to_launch, lambda_function_name, \
access_key_id, secret_access_key, \
json_payload, [ region1, region2, ... ])
machine_state.py
overviewlibmu/machine_state.py provides general functionality for building coordinating servers.
At a high level, the idea is that we can build a state machine out of these generic classes, and that state machine drives the computation for each worker. Each state in the machine represents a pair, (expected client message, server command); the client always "goes first". Client responses depend on the prior command; all responses indicating success begin with "OK". (For more information on commands and responses, see libmu/handler.py.)
We represent state machines as subclasses of MachineState
, which is itself a subclass of
SocketNB
. SocketNB
is a wrapper around socket-like objects that handles non-blocking reads
and writes, a simple chunking protocol, etc.
MachineState
defines the general state transition framework, but one should probably not inherit
directly from MachineState
. Instead, most of the time a state will inherit from classes like
TerminalState
, CommandListState
, or ForLoopState
. These are the three subclasses we
use in png2y4m\_server.py
;
xcenc_server.py encodes
a more complex state machine that makes use of several other subclasses.
Immediately below I give a bit more background on each of the parent classes we use in building the
png2y4m_server.py
state machine; below, I discuss the state machine classes themselves.
TerminalState
TerminalState
is simple: it's a state from which the machine never transitions. In
png2y4m_server.py
, we have FinalState
, which simply overrides the extra
attribute to make
the string representation of the state more comprehensible in debug mode.
Another important subclass of TerminalState
is ErrorState
. If a state machine enters this
state, the server will report a corresponding error after execution.
CommandListState
A CommandListState
comprises a list of (client response, server command), and tracks the progress
through this command list. (One can think of a CommandListState
as a straight-line sequence
of independent states.)
The commandlist
attribute is a list of strings or tuples from which the CommandListState
builds the set of expected responses and the resulting commands. If an entry in commandlist
is a string, this is interpreted as the command that the server will send. The state will
automatically decide an expected response based on the previous command (or just "OK" for the
first command).
If an entry in commandlist
is a tuple, this is interpreted as (client_response, server_command)
.
This allows more explicit control over the client's expected response. A special case for both
client_response
and server_command
is None
. In the case of client_response
, None
means
that the state machine should immediately send the command and transition to the next state.
For server_response
, this means that there is no command, after a response is received.
We will see how both of these are useful later.
After a CommandListState
sends its last command, it transitions to the state whose constructor
is specified in the nextState
property.
ForLoopState
A ForLoopState
encodes a loop with an incrementing counter. iterKey
is a dictionary key
associated with the iteration counter; the counter is stored in the dictionary self.info
, which
is always carried from one state to the next. iterInit
is the first value given to the counter,
and iterFin
is the final value. If the value in self.info
corresponding to the key specified
by breakKey
is not None
, iteration ends the next time the machine reaches the ForLoopState
.
Each time the state machine enters the ForLoopState
, it consults the loop counter and decides
whether to transition to loopState
(continue looping) or exitState
(finish looping).
Most of the time, the expect
and command
properties are both None
for a ForLoopState
,
i.e., the state machine transitions to the next state immediately.
In this case, our state machine is pretty simple:
Because each state has to refer to the state that comes after it, the classes corresponding to each
state need to be defined in reverse order in the source file. Let's start with PNG2Y4MConfigState
,
which is the state machine's entry point.
PNG2Y4MConfigState
This state is a subclass of the CommandListState
(described above) that sets a few variables
in the worker. Its constructor first invokes the CommandListState
constructor, then computes
the commands to send based on the worker number and the video being transcoded.
Note that the final command is None
; the state machine will wait for the response from the
penultimate command (seti:nonblock:0
) and immediately transition to the next state.
PNG2Y4MRetrieveLoopState
This state is a subclass of the ForLoopState
that controls the number of frames that are
downloaded. (Note that the constructor is overridden here because the ServerInfo object might
be changed at run time.)
If the looping is not yet finished, this state goes to PNG2Y4MRetrieveAndRunState
, else it goes to
PNG2Y4MUploadState
.
PNG2Y4MRetrieveAndRunState
This is once again a CommandListState
subclass. It sets variables that determine which S3 object
to retrieve and the corresponding output filename, then retrieves the object. Here again we add
a final None
state to delay transition back to the loop header until the retrieve:
command
is complete.
Note that the first expect
is None
because every path leading to this state has already
waited for outstanding responses from the client; similarly, the final command is None
,
which makes this state wait for the client's response before transitioning back to the loop header.
Note also that we override the nextState
property after PNG2Y4MRetrieveLoopState
is defined
to prevent use-before-define errors.
PNG2Y4MUploadState
Another CommandListState
that runs the png2y4m conversion command and then uploads the result,
then transitions to the FinalState.