loader-engima

This repo will include loader components for both offline and online workloads. The offline will sort based. We will generate a randnom file and ask a server component (using PHP) to sort it. The random file will be based on the baseline we agree on. e.g.,

dd if=/dev/urandom of=1.log bs=5M count=2

Please note that the size of bs is determined by the VALUE the system needs to yield for the USER. i.e. if we anticipate a large files than bs will be bigger.

The server will load the generated file 1.log and sort it.

Our loader system will deploy the files across the compute system and log the processing time.

Assuming we run Apache2 with Documentroot at /var/www/html, we are going to copy the load.php to /var/www/html

sudo cp load.php /var/www/html/

sudo cp 1.log /var/www/html/

The load will be generated by a simple script e.g.

while (sleep 5)
do
curl $theipoftheserver/load.php
done

While we run it for couple of minutes we will observe ONE server only for general performance by executing on the SERVER:

tail -f /var/log/apache2/error.log

see if the test is valid and no outstanding errors are generated e.g., lack of memory.

Then execute the vmstat command on the SERVER and observe the reosurce utilization i.e. memory, swap, io, cpu etc.

Once the server is in optimal utilization i.e. 60-70% utilization of the reosurce than we start to measure the results.

Again, the results refers to VALUE we got from the system. The VALUE refers to the system utility i.e. user request per second.

The requests are logged in the apache access log under (per /etc/apache2/apache2.conf)

/var/log/apache2/access.log

Using the access log we measure the request per seconds or minutes.

Lets see how we can measure the request per minutes

We first filter the access log with the requests we wish to measure. e.g. if our app is /load.php we will filter that app call.

grep load.php /var/log/apache2/access.log

Then we clean the file so only the response time and the time the request was served remains

grep load.php /var/log/apache2/access.log|awk '{print $10,$4}'

Apache prints the character [ before the date. This can be modified in the log format configuration but we will remove it for now by adding sed 's/[//g'.

grep load.php /var/log/apache2/access.log|awk '{print $10,$4}' |sed 's/\[//g'

Now we are ready for the request per minutes calculation. We will use simple bucket sort and use awk arrays. To do it we need to create the bucket keys. We wish to create a single bucket for each minutes so we will concatenate the fields that form a miunte. i.e.,

17/Mar/2016 16:43

with the awk statement:

awk -F\: '{print $1"_"$2"_"$3}'

So far we have:

grep load.php /var/log/apache2/access.log|awk '{print $10,$4}' |sed 's/\[//g' | awk -F\: '{print $1"_"$2"_"$3}'

Now we are ready to popluate the minutes buckets (marked as arr)

grep load.php /var/log/apache2/access.log|awk '{print $10,$4}' |sed 's/\[//g' | awk -F\: '{print $1"_"$2"_"$3}' | awk '{arr[$2]+=$1}END{for(i in arr) print i,arr[i]/60}'

Finally we print the results for each minutes with the amount of served requests. e.g.,

17/Mar/2016_17_24 37.75
17/Mar/2016_17_25 35.2333
17/Mar/2016_17_26 35.2333