search

openenergymonitor / emonhub

Python service linking and decoding input to MQTT & Emoncms
90 stars 83 forks source link

Emoncms HTTP interfacer option to send names and send as compressed binary #164

Closed TrystanLea closed 2 years ago

TrystanLea commented 2 years ago

This pull request incorporates https://github.com/openenergymonitor/emonhub/pull/70 by @SeanDS and https://github.com/openenergymonitor/emonhub/pull/55 by @anbugge adding the ability to send input names alongside values to emoncms with gz compression.

Interestingly testing a couple of different formats for sending bulk data with compression suggests that, the at first glance, relatively inefficient json with names repeated for each frame actually ends up compressing better than alternative approaches. I will post the different tests I did below

TrystanLea commented 2 years ago

Test 1: Original bulk data format:

<?php

$data = array();

$len = 0;
$n = 0;

$start_time = time();
$time = $start_time;

while(true) {

    $dt = $time-$start_time;
    $time+=10;

    $node = "emontx";
    $nodeid = 10;
    $keyval = array("voltage"=>100.12,"current"=>200.45,"energy"=>300,"power"=>300,"flowT"=>300,"returnT"=>300,"flowrate"=>300,"volume"=>300);
    $line = [$dt,$nodeid];
    foreach ($keyval as $key=>$val) {
        $line[] = $val;
    }
    $data[] = $line;

    $last_len = $len;
    $len = strlen(json_encode($data));

    $n++;
    if ($n>100) break;
}

$json = json_encode($data);
echo $json."\n";

echo strlen($json)."\n";
$s = microtime(true);
echo strlen(gzcompress($json,9))."\n";
print microtime(true)-$s."\n";

JSON output:

[[0,10,100.12,200.45,300,300,300,300,300,300],[10,10,100.12,200.45,300,300,300,300,300,300],[20,10,100.12,200.45,300,300,300,300,300,300],[30,10,100.12,200.45,300,300,300,300,300,300],[40,10,100.12,200.45,300,300,300,300,300,300],[50,10,100.12,200.45,300,300,300,300,300,300],[60,10,100.12,200.45,300,300,300,300,300,300],[70,10,100.12,200.45,300,300,300,300,300,300],[80,10,100.12,200.45,300,300,300,300,300,300],[90,10,100.12,200.45,300,300,300,300,300,300],[100,10,100.12,200.45,300,300,300,300,300,300],[110,10,100.12,200.45,300,300,300,300,300,300],[120,10,100.12,200.45,300,300,300,300,300,300],[130,10,100.12,200.45,300,300,300,300,300,300],[140,10,100.12,200.45,300,300,300,300,300,300],[150,10,100.12,200.45,300,300,300,300,300,300],[160,10,100.12,200.45,300,300,300,300,300,300],[170,10,100.12,200.45,300,300,300,300,300,300],[180,10,100.12,200.45,300,300,300,300,300,300],[190,10,100.12,200.45,300,300,300,300,300,300],[200,10,100.12,200.45,300,300,300,300,300,300],[210,10,100.12,200.45,300,300,300,300,300,300],[220,10,100.12,200.45,300,300,300,300,300,300],[230,10,100.12,200.45,300,300,300,300,300,300],[240,10,100.12,200.45,300,300,300,300,300,300],[250,10,100.12,200.45,300,300,300,300,300,300],[260,10,100.12,200.45,300,300,300,300,300,300],[270,10,100.12,200.45,300,300,300,300,300,300],[280,10,100.12,200.45,300,300,300,300,300,300],[290,10,100.12,200.45,300,300,300,300,300,300],[300,10,100.12,200.45,300,300,300,300,300,300],[310,10,100.12,200.45,300,300,300,300,300,300],[320,10,100.12,200.45,300,300,300,300,300,300],[330,10,100.12,200.45,300,300,300,300,300,300],[340,10,100.12,200.45,300,300,300,300,300,300],[350,10,100.12,200.45,300,300,300,300,300,300],[360,10,100.12,200.45,300,300,300,300,300,300],[370,10,100.12,200.45,300,300,300,300,300,300],[380,10,100.12,200.45,300,300,300,300,300,300],[390,10,100.12,200.45,300,300,300,300,300,300],[400,10,100.12,200.45,300,300,300,300,300,300],[410,10,100.12,200.45,300,300,300,300,300,300],[420,10,100.12,200.45,300,300,300,300,300,300],[430,10,100.12,200.45,300,300,300,300,300,300],[440,10,100.12,200.45,300,300,300,300,300,300],[450,10,100.12,200.45,300,300,300,300,300,300],[460,10,100.12,200.45,300,300,300,300,300,300],[470,10,100.12,200.45,300,300,300,300,300,300],[480,10,100.12,200.45,300,300,300,300,300,300],[490,10,100.12,200.45,300,300,300,300,300,300],[500,10,100.12,200.45,300,300,300,300,300,300],[510,10,100.12,200.45,300,300,300,300,300,300],[520,10,100.12,200.45,300,300,300,300,300,300],[530,10,100.12,200.45,300,300,300,300,300,300],[540,10,100.12,200.45,300,300,300,300,300,300],[550,10,100.12,200.45,300,300,300,300,300,300],[560,10,100.12,200.45,300,300,300,300,300,300],[570,10,100.12,200.45,300,300,300,300,300,300],[580,10,100.12,200.45,300,300,300,300,300,300],[590,10,100.12,200.45,300,300,300,300,300,300],[600,10,100.12,200.45,300,300,300,300,300,300],[610,10,100.12,200.45,300,300,300,300,300,300],[620,10,100.12,200.45,300,300,300,300,300,300],[630,10,100.12,200.45,300,300,300,300,300,300],[640,10,100.12,200.45,300,300,300,300,300,300],[650,10,100.12,200.45,300,300,300,300,300,300],[660,10,100.12,200.45,300,300,300,300,300,300],[670,10,100.12,200.45,300,300,300,300,300,300],[680,10,100.12,200.45,300,300,300,300,300,300],[690,10,100.12,200.45,300,300,300,300,300,300],[700,10,100.12,200.45,300,300,300,300,300,300],[710,10,100.12,200.45,300,300,300,300,300,300],[720,10,100.12,200.45,300,300,300,300,300,300],[730,10,100.12,200.45,300,300,300,300,300,300],[740,10,100.12,200.45,300,300,300,300,300,300],[750,10,100.12,200.45,300,300,300,300,300,300],[760,10,100.12,200.45,300,300,300,300,300,300],[770,10,100.12,200.45,300,300,300,300,300,300],[780,10,100.12,200.45,300,300,300,300,300,300],[790,10,100.12,200.45,300,300,300,300,300,300],[800,10,100.12,200.45,300,300,300,300,300,300],[810,10,100.12,200.45,300,300,300,300,300,300],[820,10,100.12,200.45,300,300,300,300,300,300],[830,10,100.12,200.45,300,300,300,300,300,300],[840,10,100.12,200.45,300,300,300,300,300,300],[850,10,100.12,200.45,300,300,300,300,300,300],[860,10,100.12,200.45,300,300,300,300,300,300],[870,10,100.12,200.45,300,300,300,300,300,300],[880,10,100.12,200.45,300,300,300,300,300,300],[890,10,100.12,200.45,300,300,300,300,300,300],[900,10,100.12,200.45,300,300,300,300,300,300],[910,10,100.12,200.45,300,300,300,300,300,300],[920,10,100.12,200.45,300,300,300,300,300,300],[930,10,100.12,200.45,300,300,300,300,300,300],[940,10,100.12,200.45,300,300,300,300,300,300],[950,10,100.12,200.45,300,300,300,300,300,300],[960,10,100.12,200.45,300,300,300,300,300,300],[970,10,100.12,200.45,300,300,300,300,300,300],[980,10,100.12,200.45,300,300,300,300,300,300],[990,10,100.12,200.45,300,300,300,300,300,300],[1000,10,100.12,200.45,300,300,300,300,300,300]]

Uncompressed size: 4738 bytes Compressed size: 263 bytes (5.5%)

TrystanLea commented 2 years ago

Test 2: 

<?php

$data = array();

$len = 0;
$n = 0;

$start_time = time();
$time = $start_time;
while(true) {

    $dt = $time-$start_time;
    $time+=10;

    $node = "emontx";
    $nodeid = 10;
    $keyval = array("voltage"=>100.12,"current"=>200.45,"energy"=>300,"power"=>300,"flowT"=>300,"returnT"=>300,"flowrate"=>300,"volume"=>300);

    foreach ($keyval as $key=>$val) {
        if (!isset($data[$node])) $data[$node] = [];
        if (!isset($data[$node][$key])) $data[$node][$key] = [];
        $data[$node][$key][] = $dt;
        $data[$node][$key][] = $val;
    }

    $last_len = $len;
    $len = strlen(json_encode($data));

    $n++;

    if ($n>100) break;
}

$json = json_encode($data);
echo $json."\n";

echo strlen($json)."\n";
$s = microtime(true);
echo strlen(gzcompress($json,9))."\n";
print microtime(true)-$s."\n";

JSON output:

{"emontx":{"voltage":[0,100.12,10,100.12,20,100.12,30,100.12,40,100.12,50,100.12,60,100.12,70,100.12,80,100.12,90,100.12,100,100.12,110,100.12,120,100.12,130,100.12,140,100.12,150,100.12,160,100.12,170,100.12,180,100.12,190,100.12,200,100.12,210,100.12,220,100.12,230,100.12,240,100.12,250,100.12,260,100.12,270,100.12,280,100.12,290,100.12,300,100.12,310,100.12,320,100.12,330,100.12,340,100.12,350,100.12,360,100.12,370,100.12,380,100.12,390,100.12,400,100.12,410,100.12,420,100.12,430,100.12,440,100.12,450,100.12,460,100.12,470,100.12,480,100.12,490,100.12,500,100.12,510,100.12,520,100.12,530,100.12,540,100.12,550,100.12,560,100.12,570,100.12,580,100.12,590,100.12,600,100.12,610,100.12,620,100.12,630,100.12,640,100.12,650,100.12,660,100.12,670,100.12,680,100.12,690,100.12,700,100.12,710,100.12,720,100.12,730,100.12,740,100.12,750,100.12,760,100.12,770,100.12,780,100.12,790,100.12,800,100.12,810,100.12,820,100.12,830,100.12,840,100.12,850,100.12,860,100.12,870,100.12,880,100.12,890,100.12,900,100.12,910,100.12,920,100.12,930,100.12,940,100.12,950,100.12,960,100.12,970,100.12,980,100.12,990,100.12,1000,100.12],"current":[0,200.45,10,200.45,20,200.45,30,200.45,40,200.45,50,200.45,60,200.45,70,200.45,80,200.45,90,200.45,100,200.45,110,200.45,120,200.45,130,200.45,140,200.45,150,200.45,160,200.45,170,200.45,180,200.45,190,200.45,200,200.45,210,200.45,220,200.45,230,200.45,240,200.45,250,200.45,260,200.45,270,200.45,280,200.45,290,200.45,300,200.45,310,200.45,320,200.45,330,200.45,340,200.45,350,200.45,360,200.45,370,200.45,380,200.45,390,200.45,400,200.45,410,200.45,420,200.45,430,200.45,440,200.45,450,200.45,460,200.45,470,200.45,480,200.45,490,200.45,500,200.45,510,200.45,520,200.45,530,200.45,540,200.45,550,200.45,560,200.45,570,200.45,580,200.45,590,200.45,600,200.45,610,200.45,620,200.45,630,200.45,640,200.45,650,200.45,660,200.45,670,200.45,680,200.45,690,200.45,700,200.45,710,200.45,720,200.45,730,200.45,740,200.45,750,200.45,760,200.45,770,200.45,780,200.45,790,200.45,800,200.45,810,200.45,820,200.45,830,200.45,840,200.45,850,200.45,860,200.45,870,200.45,880,200.45,890,200.45,900,200.45,910,200.45,920,200.45,930,200.45,940,200.45,950,200.45,960,200.45,970,200.45,980,200.45,990,200.45,1000,200.45],"energy":[0,300,10,300,20,300,30,300,40,300,50,300,60,300,70,300,80,300,90,300,100,300,110,300,120,300,130,300,140,300,150,300,160,300,170,300,180,300,190,300,200,300,210,300,220,300,230,300,240,300,250,300,260,300,270,300,280,300,290,300,300,300,310,300,320,300,330,300,340,300,350,300,360,300,370,300,380,300,390,300,400,300,410,300,420,300,430,300,440,300,450,300,460,300,470,300,480,300,490,300,500,300,510,300,520,300,530,300,540,300,550,300,560,300,570,300,580,300,590,300,600,300,610,300,620,300,630,300,640,300,650,300,660,300,670,300,680,300,690,300,700,300,710,300,720,300,730,300,740,300,750,300,760,300,770,300,780,300,790,300,800,300,810,300,820,300,830,300,840,300,850,300,860,300,870,300,880,300,890,300,900,300,910,300,920,300,930,300,940,300,950,300,960,300,970,300,980,300,990,300,1000,300],"power":[0,300,10,300,20,300,30,300,40,300,50,300,60,300,70,300,80,300,90,300,100,300,110,300,120,300,130,300,140,300,150,300,160,300,170,300,180,300,190,300,200,300,210,300,220,300,230,300,240,300,250,300,260,300,270,300,280,300,290,300,300,300,310,300,320,300,330,300,340,300,350,300,360,300,370,300,380,300,390,300,400,300,410,300,420,300,430,300,440,300,450,300,460,300,470,300,480,300,490,300,500,300,510,300,520,300,530,300,540,300,550,300,560,300,570,300,580,300,590,300,600,300,610,300,620,300,630,300,640,300,650,300,660,300,670,300,680,300,690,300,700,300,710,300,720,300,730,300,740,300,750,300,760,300,770,300,780,300,790,300,800,300,810,300,820,300,830,300,840,300,850,300,860,300,870,300,880,300,890,300,900,300,910,300,920,300,930,300,940,300,950,300,960,300,970,300,980,300,990,300,1000,300],"flowT":[0,300,10,300,20,300,30,300,40,300,50,300,60,300,70,300,80,300,90,300,100,300,110,300,120,300,130,300,140,300,150,300,160,300,170,300,180,300,190,300,200,300,210,300,220,300,230,300,240,300,250,300,260,300,270,300,280,300,290,300,300,300,310,300,320,300,330,300,340,300,350,300,360,300,370,300,380,300,390,300,400,300,410,300,420,300,430,300,440,300,450,300,460,300,470,300,480,300,490,300,500,300,510,300,520,300,530,300,540,300,550,300,560,300,570,300,580,300,590,300,600,300,610,300,620,300,630,300,640,300,650,300,660,300,670,300,680,300,690,300,700,300,710,300,720,300,730,300,740,300,750,300,760,300,770,300,780,300,790,300,800,300,810,300,820,300,830,300,840,300,850,300,860,300,870,300,880,300,890,300,900,300,910,300,920,300,930,300,940,300,950,300,960,300,970,300,980,300,990,300,1000,300],"returnT":[0,300,10,300,20,300,30,300,40,300,50,300,60,300,70,300,80,300,90,300,100,300,110,300,120,300,130,300,140,300,150,300,160,300,170,300,180,300,190,300,200,300,210,300,220,300,230,300,240,300,250,300,260,300,270,300,280,300,290,300,300,300,310,300,320,300,330,300,340,300,350,300,360,300,370,300,380,300,390,300,400,300,410,300,420,300,430,300,440,300,450,300,460,300,470,300,480,300,490,300,500,300,510,300,520,300,530,300,540,300,550,300,560,300,570,300,580,300,590,300,600,300,610,300,620,300,630,300,640,300,650,300,660,300,670,300,680,300,690,300,700,300,710,300,720,300,730,300,740,300,750,300,760,300,770,300,780,300,790,300,800,300,810,300,820,300,830,300,840,300,850,300,860,300,870,300,880,300,890,300,900,300,910,300,920,300,930,300,940,300,950,300,960,300,970,300,980,300,990,300,1000,300],"flowrate":[0,300,10,300,20,300,30,300,40,300,50,300,60,300,70,300,80,300,90,300,100,300,110,300,120,300,130,300,140,300,150,300,160,300,170,300,180,300,190,300,200,300,210,300,220,300,230,300,240,300,250,300,260,300,270,300,280,300,290,300,300,300,310,300,320,300,330,300,340,300,350,300,360,300,370,300,380,300,390,300,400,300,410,300,420,300,430,300,440,300,450,300,460,300,470,300,480,300,490,300,500,300,510,300,520,300,530,300,540,300,550,300,560,300,570,300,580,300,590,300,600,300,610,300,620,300,630,300,640,300,650,300,660,300,670,300,680,300,690,300,700,300,710,300,720,300,730,300,740,300,750,300,760,300,770,300,780,300,790,300,800,300,810,300,820,300,830,300,840,300,850,300,860,300,870,300,880,300,890,300,900,300,910,300,920,300,930,300,940,300,950,300,960,300,970,300,980,300,990,300,1000,300],"volume":[0,300,10,300,20,300,30,300,40,300,50,300,60,300,70,300,80,300,90,300,100,300,110,300,120,300,130,300,140,300,150,300,160,300,170,300,180,300,190,300,200,300,210,300,220,300,230,300,240,300,250,300,260,300,270,300,280,300,290,300,300,300,310,300,320,300,330,300,340,300,350,300,360,300,370,300,380,300,390,300,400,300,410,300,420,300,430,300,440,300,450,300,460,300,470,300,480,300,490,300,500,300,510,300,520,300,530,300,540,300,550,300,560,300,570,300,580,300,590,300,600,300,610,300,620,300,630,300,640,300,650,300,660,300,670,300,680,300,690,300,700,300,710,300,720,300,730,300,740,300,750,300,760,300,770,300,780,300,790,300,800,300,810,300,820,300,830,300,840,300,850,300,860,300,870,300,880,300,890,300,900,300,910,300,920,300,930,300,940,300,950,300,960,300,970,300,980,300,990,300,1000,300]}} Uncompressed size: 7093 bytes Compressed size: 731 bytes (10.3%)

TrystanLea commented 2 years ago

Test 3: Format used in pull request above:

<?php
$data = array();
$len = 0;
$n = 0;

$start_time = time();
$time = $start_time;
while(true) {

    $dt = $time-$start_time;
    $time+=10;

    $node = "emontx";
    $nodeid = 10;
    $keyval = array("voltage"=>100.12,"current"=>200.45,"energy"=>300,"power"=>300,"flowT"=>300,"returnT"=>300,"flowrate"=>300,"volume"=>300);
    $data[] = [$time,$node,$keyval];

    $last_len = $len;
    $len = strlen(json_encode($data));

    $n++;
    if ($n>100) break;
}

$json = json_encode($data);
print $json."\n";

echo strlen($json)."\n";
$s = microtime(true);
echo strlen(gzcompress($json,9))."\n";
print microtime(true)-$s."\n";

JSON output: [[1640878399,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878409,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878419,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878429,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878439,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878449,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878459,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878469,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878479,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878489,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878499,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878509,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878519,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878529,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878539,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878549,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878559,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878569,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878579,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878589,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878599,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878609,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878619,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878629,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878639,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878649,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878659,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878669,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878679,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878689,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878699,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878709,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878719,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878729,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878739,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878749,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878759,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878769,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878779,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878789,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878799,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878809,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878819,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878829,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878839,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878849,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878859,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878869,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878879,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878889,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878899,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878909,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878919,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878929,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878939,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878949,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878959,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878969,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878979,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878989,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640878999,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879009,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879019,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879029,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879039,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879049,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879059,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879069,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879079,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879089,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879099,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879109,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879119,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879129,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879139,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879149,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879159,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879169,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879179,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879189,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879199,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879209,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879219,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879229,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879239,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879249,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879259,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879269,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879279,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879289,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879299,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879309,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879319,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879329,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879339,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879349,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879359,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879369,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879379,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879389,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}],[1640879399,"emontx",{"voltage":100.12,"current":200.45,"energy":300,"power":300,"flowT":300,"returnT":300,"flowrate":300,"volume":300}]]

Uncompressed size: 13838 bytes Compressed size: 403 bytes (2.9%)

TrystanLea commented 2 years ago

The last format whilst producing a much larger JSON seems to compress a lot better. It's also a format that requires less processing pre and after compression in order to make use of it.

What I havent worked out yet is if compression on a busy emoncms server with lots of different devices connecting will result in more or less server load? It may be that compression reduces the time spent with an apache2 connection open per device and that this will provide more of a performance advantage than the penalty of decompression, testing ongoing..

TrystanLea commented 2 years ago

Example of emonhub log sending compressed data with input names:

2021-12-30 15:59:02,655 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1640879942&cb=1 (301 bytes of data, 1 frames, compressed)
2021-12-30 15:59:02,656 INFO     emoncmsorg compression ratio: 48%
2021-12-30 15:59:02,786 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org (127 ms)
2021-12-30 15:59:32,878 DEBUG    emoncmsorg Buffer size: 6
2021-12-30 15:59:32,881 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1640879972&cb=1 (526 bytes of data, 6 frames, compressed)
2021-12-30 15:59:32,881 INFO     emoncmsorg compression ratio: 21%
2021-12-30 15:59:33,017 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org (134 ms)
2021-12-30 16:00:03,105 DEBUG    emoncmsorg Buffer size: 6
2021-12-30 16:00:03,110 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1640880003&cb=1 (543 bytes of data, 6 frames, compressed)
2021-12-30 16:00:03,112 INFO     emoncmsorg compression ratio: 22%
2021-12-30 16:00:03,276 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org (163 ms)
2021-12-30 16:00:33,373 DEBUG    emoncmsorg Buffer size: 6
2021-12-30 16:00:33,375 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1640880033&cb=1 (545 bytes of data, 6 frames, compressed)
2021-12-30 16:00:33,375 INFO     emoncmsorg compression ratio: 22%
2021-12-30 16:00:33,545 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org (169 ms)
2021-12-30 16:01:03,552 DEBUG    emoncmsorg Buffer size: 6
2021-12-30 16:01:03,555 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1640880063&cb=1 (543 bytes of data, 6 frames, compressed)
2021-12-30 16:01:03,556 INFO     emoncmsorg compression ratio: 21%
2021-12-30 16:01:03,714 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org (156 ms)
2021-12-30 16:01:33,765 DEBUG    emoncmsorg Buffer size: 6
2021-12-30 16:01:33,768 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1640880093&cb=1 (535 bytes of data, 6 frames, compressed)
2021-12-30 16:01:33,769 INFO     emoncmsorg compression ratio: 21%
2021-12-30 16:01:33,933 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org (163 ms)
2021-12-30 16:02:04,019 DEBUG    emoncmsorg Buffer size: 6
2021-12-30 16:02:04,024 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1640880124&cb=1 (559 bytes of data, 6 frames, compressed)
2021-12-30 16:02:04,026 INFO     emoncmsorg compression ratio: 22%
2021-12-30 16:02:04,190 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org (162 ms)
2021-12-30 16:02:34,282 DEBUG    emoncmsorg Buffer size: 6
2021-12-30 16:02:34,286 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1640880154&cb=1 (530 bytes of data, 6 frames, compressed)
2021-12-30 16:02:34,287 INFO     emoncmsorg compression ratio: 21%
2021-12-30 16:02:34,449 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org (159 ms)
2021-12-30 16:03:04,539 DEBUG    emoncmsorg Buffer size: 6
2021-12-30 16:03:04,545 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1640880184&cb=1 (547 bytes of data, 6 frames, compressed)
2021-12-30 16:03:04,546 INFO     emoncmsorg compression ratio: 22%
2021-12-30 16:03:04,708 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org (159 ms)
2021-12-30 16:03:34,793 DEBUG    emoncmsorg Buffer size: 7
2021-12-30 16:03:34,796 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1640880214&cb=1 (546 bytes of data, 7 frames, compressed)
2021-12-30 16:03:34,797 INFO     emoncmsorg compression ratio: 20%
2021-12-30 16:03:34,946 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org (147 ms)
2021-12-30 16:04:05,039 DEBUG    emoncmsorg Buffer size: 6
2021-12-30 16:04:05,042 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1640880245&cb=1 (535 bytes of data, 6 frames, compressed)
2021-12-30 16:04:05,043 INFO     emoncmsorg compression ratio: 21%
2021-12-30 16:04:05,203 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org (159 ms)
2021-12-30 16:04:35,298 DEBUG    emoncmsorg Buffer size: 6
2021-12-30 16:04:35,301 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1640880275&cb=1 (531 bytes of data, 6 frames, compressed)
2021-12-30 16:04:35,302 INFO     emoncmsorg compression ratio: 22%
2021-12-30 16:04:35,472 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org (168 ms)
TrystanLea commented 2 years ago

Compression and names turned off:

2021-12-30 16:06:36,921 INFO     MainThread Setting emoncmsorg senddata: 1
2021-12-30 16:06:36,921 INFO     MainThread Setting emoncmsorg sendstatus: 1
2021-12-30 16:06:36,921 INFO     MainThread Setting emoncmsorg sendnames: 0
2021-12-30 16:06:36,921 INFO     MainThread Setting emoncmsorg compress: 0
2021-12-30 16:07:06,613 DEBUG    emoncmsorg Buffer size: 6
2021-12-30 16:07:06,615 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1640880426 (749 bytes of data, 6 frames, uncompressed)
2021-12-30 16:07:06,778 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org (162 ms)
2021-12-30 16:07:36,870 DEBUG    emoncmsorg Buffer size: 6
2021-12-30 16:07:36,872 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1640880456 (780 bytes of data, 6 frames, uncompressed)
2021-12-30 16:07:37,019 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org (146 ms)
2021-12-30 16:08:07,118 DEBUG    emoncmsorg Buffer size: 6
2021-12-30 16:08:07,120 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1640880487 (702 bytes of data, 6 frames, uncompressed)
2021-12-30 16:08:07,285 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org (164 ms)
2021-12-30 16:08:37,377 DEBUG    emoncmsorg Buffer size: 6
2021-12-30 16:08:37,380 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1640880517 (753 bytes of data, 6 frames, uncompressed)
2021-12-30 16:08:37,547 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org (165 ms)
2021-12-30 16:09:07,630 DEBUG    emoncmsorg Buffer size: 6
2021-12-30 16:09:07,632 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1640880547 (759 bytes of data, 6 frames, uncompressed)
2021-12-30 16:09:07,798 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org (163 ms)
2021-12-30 16:09:37,883 DEBUG    emoncmsorg Buffer size: 6
2021-12-30 16:09:37,886 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1640880577 (747 bytes of data, 6 frames, uncompressed)
2021-12-30 16:09:38,055 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org (167 ms)
2021-12-30 16:10:08,145 DEBUG    emoncmsorg Buffer size: 6
2021-12-30 16:10:08,148 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1640880608 (734 bytes of data, 6 frames, uncompressed)
2021-12-30 16:10:08,315 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org (166 ms)
2021-12-30 16:10:38,400 DEBUG    emoncmsorg Buffer size: 6
2021-12-30 16:10:38,402 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1640880638 (752 bytes of data, 6 frames, uncompressed)
2021-12-30 16:10:38,564 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org (160 ms)
TrystanLea commented 2 years ago

Average emoncms.org response time of compressed test: 155.5ms

Average emoncms.org response time of uncompressed test: 161.6ms

Not a large difference but we can see that compressed data with names takes up a little less room than uncompressed data without names..

bwduncan commented 2 years ago

Looks good, thanks Trystan. If we want to try to lighten the load on the emoncms servers I think we could do better. Reusing connections, compressing the entire stream (i.e. HTTP/2), binary formats would probably have a bigger impact. Something like BSON or Protocol Buffers would allow us to represent the data in the most compact format, so that compression isn't necessary. (A float is 8 bytes, but if you represent it as text then it's up to 19 bytes!).

Given that all of this is happening over HTTPS, the benefit to compressing the data in terms of round-trip time will be negligible. The size of the TLS certificate chain alone is nearly 7kB. https://www.ssllabs.com/ssltest/analyze.html?d=emoncms.org&hideResults=on

I tried using a requests.Session in EmonHubEmoncmsHTTPInterfacer.py to avoid a full HTTP round-trip every time, but it looks like the server is dropping the connection each time. In Firefox I can see that the server is sending connection: close with each reply, even for a simple page, forcing the client to open a whole new TCP/HTTPS/TLS connection again. Keep-alive in Apache is on by default, maybe it's turned off for some other reason?

The other thing to consider is the time between updates. Because the data are so regular, the bigger the payload the better the compression ratio. Plus, you get to save all the above-mentioned HTTP costs, too. If you don't stare at emoncms.org, sending updates half-hourly might be enough. Just a thought.

Stopping short of that, it might make more sense to use the built-in compression that HTTP/1 provides. Setting the Content-Encoding: gzip header and using gzip would give better gains because Apache will decompress for us (I think), and the entire body is compressed. I don't know if PHP makes that transparent, though.

Having said all that, the code above has now been written, so I see no reason not to merge it. My only comment is it the code is clever enough to decide whether or not to use compression, does it really need a config option? It should try compression by default.

TrystanLea commented 2 years ago

Thanks for your detailed thoughts on this @bwduncan much appreciated.

Good points re https overhead and apache settings, it's been a while since I have looked at the apache settings, but I think the keep alive was reducing performance.. cant remember exactly... I should probably revisit that setting?

bwduncan commented 2 years ago

Maybe getting a bit off-topic with this comment but you might want to look at a different MPM (Multi-Processing Module) as well. I seem to recall that the default is not particularly optimised for modern kernels. I believe the default MPM should be event, but if you needed prefork or worker for some other reason that might limit performance.

Or maybe you can skip all that and just enable http/2 :smile:

TrystanLea commented 2 years ago

Thanks @bwduncan, looking at the keep alive setting here, I think the original reason that I turned it off is that most of the http requests on emoncms.org server are devices posting data to the server at spaced apart intervals e.g 10-30s rather than user page loads that loading a lot of different resources within a short time e.g <1s.

There are on average 215 http&s requests per second posting data to the server. 271 total requests per second, 16 get data requests per second. 18,000 active feeds over ~4250 active users. So I think that's something like an average time between input posts of around 20s per user but over 4250 separate devices..

Would I need MaxRequestWorkers therefore to be over 4250? with a keep alive of 30-60s? Is that practical I wonder? or is there a different way to achieve the same result? Im using mpm_prefork..

I enabled KeepAlive yesterday with a timeout of 1s and that increased the number of concurrent apache2 processes from 129 to 205. I will experiment with increasing the timeout and increasing the MaxRequestWorkers setting and keep an eye on memory consumption..

TrystanLea commented 2 years ago

Increasing KeepAlive to 2s increases the number of apache processes to ~265 and 5s: ~410

bwduncan commented 2 years ago

That's amazing Trystan, I had no idea emoncms was so widely deployed! Well done!

Yes, you would need a lot of prefork workers, each one will cost some RAM, but if the number of workers is too low, connections will just be closed in order to service new connections.

Everything I've read suggests you should switch to event, instead of prefork, unless you need process separation for some scripts which aren't thread-safe. (I'll have to bow to the PHP expert on that one though). Certainly if our goal is to reduce the load on the server, event is much less resource intensive than prefork.

Thinking about HTTP/2 as well, https://httpd.apache.org/docs/2.4/howto/http2.html suggests that if you use prefork, it just destroys the performance.

bwduncan commented 2 years ago

Well that was fun... It seems like the default php module only works with prefork, and you need to use fpm to make it work with the newer MPMs. There are lots of people saying that using the ubuntu default php module with prefork is a bad idea in production. This worked for me:

https://www.digitalocean.com/community/tutorials/how-to-configure-apache-http-with-mpm-event-and-php-fpm-on-ubuntu-18-04

The only thing that was missing is the nonstandard location of the mysql socket (it's in /home/pi/data/mysql/mysql.sock on my emonpi). You need to set mysqli.default_socket = /home/pi/data/mysql/mysql.sock in `/etc/php/7.4/fpm/php.ini

TrystanLea commented 2 years ago

Thanks @bwduncan! really appreciate your tips on this, il do some more reading and hopefully get back to you on this soon!

One other thought is that I did implement a AES-128-CBC encryption option for use with ESP8266 hardware that struggle with the full https stack https://github.com/emoncms/emoncms/blob/master/param.php#L66 it works because the apikey can be used as a pre shared key to encrypt the data being sent, makes me wonder if using that with the compressed binary above might be a good light weight option using code that we've already developed...

TrystanLea commented 2 years ago

I've put together some example code that first compresses and then encrypts using AES-128-CBC emoncms input data to hopefully give a more concrete example of what I discussed in my previous post. This could allow us to use HTTP in order to reduce bandwidth without requiring a very long keep alive setting.

import json, zlib

from base64 import b64encode
from Crypto.Cipher import AES
from Crypto.Random import get_random_bytes
from Crypto.Util.Padding import pad
import hmac, hashlib
import binascii

import requests

userid = 3
apikey = "2d717ef8fd03fc8dc581e0b6f247db2e"
json = json.dumps([[0,"emontx",{"power1":123.45,"power2":500,"power3":300,"power4":123456}]])
data = "data="+json+"&sentat=0"

# Compress
data = zlib.compress(data.encode('utf8'),9)

# Encrypt
key = binascii.unhexlify(apikey);
iv = get_random_bytes(AES.block_size)
cipher = AES.new(key, AES.MODE_CBC, iv)
base64EncryptedData = b64encode(iv + cipher.encrypt(pad(data,AES.block_size)))

# Header hmac
hmac_str = hmac.new(key,data, hashlib.sha256).hexdigest()

reply = requests.post("http://localhost/emoncms/input/bulk", base64EncryptedData, headers={'Authorization':str(userid)+":"+hmac_str,'Content-Type':'aes128cbcgz'}, timeout=60)
print(reply.text)

sha1 = b64encode(hashlib.sha256(data).digest()).decode("utf-8") 
sha1 = sha1.replace('/','_')
sha1 = sha1.replace('+','-')

print(sha1)

if reply.text==sha1:
    print ("ok")

It would require the following changes on the emoncms end, but mostly reusing existing code developed for ESP8266 devices, in particular IotaWatt:

https://github.com/emoncms/emoncms/compare/aes128cbcgz

TrystanLea commented 2 years ago

Id be interested in hearing what you think of this idea @bwduncan is it any good as an alternative to changing the way we use apache2? or do I really need to go through the learning curve on the apache side of things? maybe it's not so hard?