Open glynhudson opened 8 years ago
The only modification I would suggest to this plan is using a raspbian based image (potentially with php5/apache already setup) for the emoncms container, just cause that rpi-alpine-scratch one is busybox so is going to involve more work. The hypriot project use resin/rpi-raspbian(:jessie?) for their base image where they are using raspbian so that might be a good place to start.
Im going to try and get your dockerfile for emoncms working on top of that (or a similar image). Ill fork this repo and put changes in there with the goal of achieving same setup on rpi3.
EDIT1: Also the rpi-mysql/rpi-redis images are using resin/rpi-raspbian as base image.
Sounds good! I'll wait to hear how you get on before I get my hand in the pie ;-)
On 12 Aug 2016 17:27, "Ben Aylott" notifications@github.com wrote:
The only modification I would suggest to this plan is using a raspbian based image (potentially with php5/apache already setup) for the emoncms container, just cause that rpi-alpine-scratch one is busybox so is going to involve more work. The hypriot project use resin/rpi-raspbian:wheezy for their base image where they are using raspbian.
Im going to try and get your dockerfile for emoncms working on top of that (or a similar image). Ill fork this repo and put changes in there with the goal of achieving same setup on rpi3.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/emoncms/emoncms-docker/issues/5#issuecomment-239493206, or mute the thread https://github.com/notifications/unsubscribe-auth/AAuUPDh6OQX4lR4a3TmelWzmjf4nv783ks5qfJ7tgaJpZM4JiPKO .
Super positive reply from @DieterReuter regarding running hypriot image with RO FS
https://github.com/hypriot/image-builder-rpi/issues/119#issuecomment-239665501
I now have working emoncms on the rpi3 on our Nobel Grid SMX SD image. It was almost drop in rpi replacements for the base images (with some version changes). My working configuration files can be found here:
https://github.com/beaylott/emoncms-docker/commit/9cdf2951cd30fc47afaeeaffeea6e844be67cbbc
There are some issues which now need to be addressed...
The Dockerfile for the modified rpi-mysql is here: https://github.com/beaylott/emoncms-docker/tree/master/rpi-mysql
I could also push my modified docker images to the public registry if that helps?
We do have a new SD image with Docker v1.12.1-rc1 now which is only 252M in size. So I would be happy if you could start with this one, as we're currently fixing last bugs and hoping to get more feedback from your project too.
Ok that is certainly another possibility ... I would be quite interested in basing off of your image but both my project (an EU research project) and OEM already have SD card images which we are working off of which makes it more difficult. However, I think we are already using a lot of your work in the form of base images etc. so it may make sense to try and bring them a bit closer together ...
So ... maybe multiple lines of attack in order of priority (for me anyway):
Nice work @beaylott , good progress. I have updated my fist post above to better present to options available.
I would like to get @DieterReuter input on what the pros / cons are of installing docker ontop of stock Jessie rather than using pre-built hypriot SD card. My feeling is that we would get better docker support and a more optimised system for docker if we used to hypriot SD card to start with.
However, whatever the outcome of the Jessie Vs Hypriot SD card image is this does not affect the building of an emoncms raspi docker container. Nice working getting emoncms up and running using nidorpi/rpi-php, its' nice how little changes is needed from the standard emoncms-docker setup. I will create a branch on this repo called raspi and give you RW access so you can push. Update: I've just merged your changes: https://github.com/emoncms/emoncms-docker/pull/6
I think we are agreed that for the purpose of SMX integration a single Docker image will be best. For this we should build up emoncms in a single container based on raspi-alpine, agree?
I may be wrong (and @DieterReuter can help here) but I think by basing the images off of resin/rpi-raspbian:(wheezy/jessie) we get most of the work of Hypriot (at least on the container side rather than host OS). I guess the benefit of apline (busybox) is that it is lighter on resources compared to debian but it will probably require more work to get the stuff working compared to building something on jessie where we can use all the existing instructions without many changes?
@glynhudson probably I should write some details about the similarities and the differences between Raspbian and HypriotOS first:
2016-05-27-raspbian-jessie-lite
has 419, HypriotOS 1.0.0-rc1
has 291 packages installed. As you guess, you can easily install additional Raspbian packages if you need them (on the RPi we'd like to be 100% compatible to Raspbian) !root
user available for security reasonsAbout 18 months ago there was just no real Docker support at all on ARM (only a few hacks exists and some outdated tutorials) and we've built everything from scratch. As we're working closer together with Docker we were able to convince them, to officially support ARMv6 and ARMv7 - so the future Docker Engine packages will be officially build and supported by Docker directly, this starts with Docker Engine v1.12.1 in a few days only. And honestly you can use them even on a standard Raspbian, but then you loose all the kernel optimisations we've done.
This were just a few details, I could talk for hours. But if you'd like to get more details please ask or we could setup a Skype call.
@beaylott you're right. Using resin/rpi-raspbian:jessie
is the best starting place if you're using some Pi related software and tools like accessing the Pi hardware etc. Alpine is extremely slim and is the best option if you'd like to get super-small Docker images and saving system resources, I would recommend this for network services and so on - but it takes more work to build. So if you want to go fast, start with resin/rpi-raspbian
and optimise later for the important or larger images.
HypriotOS has a Docker-optimised kernel, based upon the standard Raspbian Kernel but with some container related kernel settings.
I think this is the part of HypriotOS we could most be interested in. Does this mean docker containers will run 'better' (faster? stable?) on HypriotOS rather than installing Docker on Jessie?
Agree, using resin/rpi-raspbian
will be better for us than alpine. Emoncms has been tested to work well on Debian. We currently use Jessie as our base image therefore docker setup with a single container resin/rpi-raspbian
should not be too difficult. I have updated my initial post above to referance resin/rpi-raspiban rather than alpine
This were just a few details, I could talk for hours. But if you'd like to get more details please ask or we could setup a Skype call.
Good idea, however I am tied up for the next few days with other work then going away next week for a couple of weeks so I can't really get stuck into this right now. However would @beaylott be interested in a skype with @DieterReuter ?
Just a side note: developing Docker images directly on a Pi could be somewhat slow, this is the reason we're doing almost all of our work on Scaleway cloud servers (C1 instance, 4x ARMv7 core, 2GByte memory, approx. 3Euro/month!) with super-fast internet access, which is great for build jobs.
Great idea, a simulated raspi build environment sounds awesome :+1:
I have made a lot of progress on some of the above... I updated all the base images to jessie and got MQTT working. Required some small modifications to emoncms setup. Unfortunately I forgot to push and dont have access to the machine as Im not in the office. On Monday ill sort it out. I think I will also do as Dieter suggested and get one of those build machines as doing it on an rpi3 is a PITA!
Nice work! Have a good weekend :-)
I pushed the jessie updates on monday. Couldnt quite get everything I wanted (cant bump MySQL version).
I have MQTT working on the container and talking to the host MQTT broker that we have ...just trying to get the emoncms MQTT daemon to play nice and then hopefully it will be done.
I have been slowed down by having to prepare for my thesis exam tomorrow and the raspberry pi grinding to a halt under the builds (Im currently migrating the builds to the Scaleway servers).
I put the MQTT stuff so far in a repo on my account (https://github.com/beaylott/emoncms-docker) and ill create a pull request when its done.
Awesome!
What was the issue with bumping MYSQL version? Are you limited by the base image version? What version would you like to use? What improvements does the newer version have?
Good effort getting MQTT up and running. Emoncms uses mqtt_input script https://github.com/emoncms/emoncms/blob/master/scripts/phpmqtt_input.php
Getting PHP MQTT working was a bit of a pain, there does not seem to be a super well supported PHP MQTT library. We used https://github.com/mgdm/Mosquitto-PHP. Setup guide here, https://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md. Keen to hear if you find any improvements. As I mentioned I would like to use PHP composer to install dependencies.
Let me know how you get on. I'm abroad in Switzerland at the moment so can't do much testing I'm afraid. Excited to get stuck in when I'm home.
On Wednesday, 24 August 2016, Ben Aylott notifications@github.com wrote:
I pushed the jessie updates on monday. Couldnt quite get everything I wanted (cant bump MySQL version).
I have MQTT working on the container and talking to the host MQTT broker that we have ...just trying to get the emoncms MQTT daemon to play nice and then hopefully it will be done.
I have been slowed down by having to prepare for my thesis exam tomorrow and the raspberry pi grinding to a halt under the builds (Im currently migrating the builds to the Scaleway servers).
I put the MQTT stuff so far in a repo on my account ( https://github.com/beaylott/emoncms-docker) and ill create a pull request when its done.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/emoncms/emoncms-docker/issues/5#issuecomment-242143019, or mute the thread https://github.com/notifications/unsubscribe-auth/AAuUPO5gKEVz-5kC-oKCofxdq__gX4YNks5qjH1ogaJpZM4JiPKO .
Glyn Hudson
http://megni.co.uk http://adventuresplusnorthwales.blogspot.com
Well one issue is that there isnt a MySQL 5.7 package readily available for ARM. It looks like there used to be on Oracle's website judging by what is in some Dockerfile's I have seen, but not anymore. There are also differences in the configuration which I am not familiar with.
The PHP MQTT library is working, I wrote a test script which was producing output from the broker on the host. It is the running of the mqtt_input script as a service I am having trouble with but I think this is a Docker thing which I need to work out for myself. I may need to write a systemctl service file for it as well (which may be useful elsewhere in emoncms anyway).
Oh ...and just generally on the version stuff... im not that bothered about the version, I was mainly taking my cue from the emoncms requirements and your expressed desire to bump versions of things (but maybe you just meant php?). 5.5 works fine for me!
Nice,a systemctl mqtt_input service would be great. I'm not fully up to speed with the advantages over /etc/init.d but it seems to be the way forward i.e everyone is using it a init.d replacement.
Thanks,
On Thursday, 25 August 2016, Ben Aylott notifications@github.com wrote:
Well one issue is that there isnt a MySQL 5.7 package readily available for ARM. It looks like there used to be on Oracle's website judging by what is in some Dockerfile's I have seen, but not anymore. There are also differences in the configuration which I am not familiar with.
The PHP MQTT library is working, I wrote a test script which was producing output from the broker on the host. It is the running of the mqtt_input script as a service I am having trouble with but I think this is a Docker thing which I need to work out for myself. I may need to write a systemctl service file for it as well (which may be useful elsewhere in emoncms anyway).
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/emoncms/emoncms-docker/issues/5#issuecomment-242319456, or mute the thread https://github.com/notifications/unsubscribe-auth/AAuUPN3zaARfBOBumcyHLmdZwtgdCOvrks5qjVZJgaJpZM4JiPKO .
Glyn Hudson
http://megni.co.uk http://adventuresplusnorthwales.blogspot.com
Sorry this has been slow, have been side tracked a lot in past few weeks. I am just replicating the environment on the Scaleway servers as I got fed up with the rpi3 grinding to a halt.
@glynhudson : Okey dokes I have got MQTT working in the container (or at least showing up in the logger with my smart meter readings). I have pushed this to my own branch and will create a pull request. There are several things you will want to review there.
I have a question now about how emoncms parses the output. I understand your format for MQTT as published by emonhub is something like
[base topic]/[node]/[key?] [value]
I have a topic I want to subscribe to with the following format:
G3M40000001/SMX/LD01/[OBIS Code 1]/[OBIS Code 2] [value]
Despite the fact I put G3M40000001/SMX/LD01 as the base topic it is showing the Node as 'SMX' and the Key as 'LD01' in the Input view.... my expectation is it would take [OBIS Code 1] as the Node and [OBIS Code 2] as the Key and then pull in all the values (it is about 48 values at a time). Any ideas what might be going on?
I guess I am asking what happens here https://github.com/emoncms/emoncms/blob/master/scripts/phpmqtt_input.php#L170-L216 ....
I dont really know PHP ....but by the looks of it if the top level topic is not equal to the base topic that you set then it will just return an empty array and hence only one input without a value (which is what I am getting)? So in other words the base topic cannot include any sub-topics as is...
EDIT: I have realised the above description is almost gibberish... to illustrate what I mean with an example ... I think you should be able to have a base topic which is made up of sub-topics ...sort of like a file path. At the moment if you specify the base topic as
head/sub1/sub2/sub3
and the inputs have the topic
head/sub1/sub2/sub3/sub4_/tail
then ecomcms will only check if
head/sub1/sub2/sub3 == head_
so that even if head=head , sub1=sub1, sub2=sub2 , sub3=sub3 ...it will not pass and you will get an empty array. Whereas in this case I would expect it to match and pass sub4 as the Node and tail as the key (along with associated values)... is this desirable behaviour?
If so I would be happy to produce a patch.
Hi Ben,
That's awesome! Nice work :+1
Emoncms currently uses the mqtt_input
script to process MQTT input.
Correct, MQTT inputs are passed:
[base topic]/[node]/[key?] [value]
Where the base topic is set in setting.php
(default emon
). e.g. mqtt
input emon/emontx/power1
will result in an Emoncms input from
node=emontx
and key=power1
.
I'm not sure the 'basetopiccan be a sub topic e.g. we use
emonas the default base topic. I have not tested using something like
emon/nobelas the basetopic. I see no reason why it it could not be made to work but since we have not designed or tested with this in mind there might well be an issue. In your case mqtt_input is only looking accepting
G3M40000001 as the base topic rather than the whole
G3M40000001/SMX/LD01/`
An easy hack could be just to re-publish the data as emon/[OBIS Code 1]/[OBIS Code 2] [value].
Or alternatively we could try and investigate what is stopping mqtt_input from accepting multiple base-subtopics.
IMO mqtt_input script needs some work, very occationally it stops for no reason we can detect. It could be the PHP mqtt library we are using or it could be how the script is being ran (currently as /etc/init.d), a community remember did port the script over the run as a systemd. I need to look into this and try and add some better logging...
On Tuesday, 13 September 2016, Ben Aylott <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:
@glynhudson https://github.com/glynhudson : Okey dokes I have got MQTT working in the container (or at least showing up in the logger with my smart meter readings). I have pushed this to my own branch and will create a pull request. There are several things you will want to review there.
I have a question now about how emoncms parses the output. I understand your format for MQTT as published by emonhub is something like
[base topic]/[node]/[key?] [value]
I have a topic I want to subscribe to with the following format:
G3M40000001/SMX/LD01/[OBIS Code 1]/[OBIS Code 2] [value]
Despite the fact I put G3M40000001/SMX/LD01 as the base topic it is showing the Node as 'SMX' and the Key as 'LD01' in the Input view.... my expectation is it would take [OBIS Code 1] as the Node and [OBIS Code 2] as the Key and then pull in all the values (it is about 48 values at a time). Any ideas what might be going on?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/emoncms/emoncms-docker/issues/5#issuecomment-246720313, or mute the thread https://github.com/notifications/unsubscribe-auth/AAuUPGYGR4270b-FJXbjQEBBNxkdfNMEks5qpsElgaJpZM4JiPKO .
Glyn Hudson
http://megni.co.uk http://adventuresplusnorthwales.blogspot.com
Hi yes I was digging around in all this. mqtt_input is the init script. I am not actually using this anymore as it isnt really right to use the init system or systemctl in the container to determine runtime behaviour (as I have discovered!). Instead I am calling the PHP script which does the MQTT for emoncms (phpmqtt_input.php) directly under supervisord running in PID1. This works pretty well. I havent seen any stops of phpmqtt_input.php under supervisord yet (at least after the database has been initialised).
If the behaviour I want is ok with you (i.e. it wont break something) then I will try and patch phpmqtt_input.php so it can deal with multiple base-topics.
The alternative is to re-publish as you suggest and I was looking into this, although this then requires some logic to be implemented somewhere. Ideally I dont want to make changes in our 'Trusted Zone' on the host (which is where the MQTT is posting from). The alternative is to run another daemon which re-publishes etc.
Awesome. Yes running phpmqtt_input.php is fine. Sorry, I slowly learning how things work differently in a container based system.
Yes, I'm happy for you to patch phpmqtt_input.php to allow base subtopics, obviously as long as standard operation with a single base topic is not effected. Happy to review and test a pull request. I'm still away from home (in Germany!) therefore it might take me a day or two to review and test.
@trystanlea wrote phpmqtt_input.php, please direct any specific questions regarding the scrip to him.
On 14 Sep 2016 16:34, "Ben Aylott" notifications@github.com wrote:
Hi yes I was digging around in all this. mqtt_input is the init script. I am not actually using this anymore as it isnt really right to use the init system or systemctl in the container to determine runtime behaviour (as I have discovered!). Instead I am calling the PHP script which does the MQTT for emoncms (phpmqtt_input.php) directly under supervisord running in PID1. This works pretty well. I havent seen any stops of phpmqtt_input.php under supervisord yet (at least after the database has been initialised).
If the behaviour I want is ok with you (i.e. it wont break something) then I will try and patch phpmqtt_input.php so it can deal with multiple base-topics.
The alternative is to re-publish as you suggest and I was looking into this, although this then requires some logic to be implemented somewhere. Ideally I dont want to make changes in our 'Trusted Zone' on the host (which is where the MQTT is posting from). The alternative is to run another daemon which re-publishes etc.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/emoncms/emoncms-docker/issues/5#issuecomment-247033068, or mute the thread https://github.com/notifications/unsubscribe-auth/AAuUPAhceRD-0vMrPDb9JN0jhjepTRAcks5qqAXngaJpZM4JiPKO .
Sure thing will do! Sorry about my stream of consciousness!
We have decided to post our data into emoncms using the API now using a custom script for various reasons so wont be going down the MQTT route now. This has the advantage of decoupling our needs from emoncms, although I have posted the patch for multiple base topics there in case it is considered useful.
Aside from the custom script, I am currently creating a single container application which merges the work on the composed application and should be more suitable for running on the pi. Ill post back here once it is done.
Hi Ben, Sounds great. I've see your pr for the mqtt input script. I'll test and review later tonight.
Thanks a lot.
On 3 Oct 2016 15:48, "Ben Aylott" notifications@github.com wrote:
We have decided to post our data into emoncms using the API now using a custom script for various reasons so wont be going down the MQTT route now. This has the advantage of decoupling our needs from emoncms, although I have posted the patch for multiple base topics there in case it is considered useful.
Aside from the custom script, I am currently creating a single container application which merges the work on the composed application and should be more suitable for running on the pi. Ill post back here once it is done.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/emoncms/emoncms-docker/issues/5#issuecomment-251126449, or mute the thread https://github.com/notifications/unsubscribe-auth/AAuUPIgNb2Dr0HLaI7dw_qYLm3lmLQMYks5qwRXUgaJpZM4JiPKO .
Hi Ben,
I've shown the work you have been doing on the raspi docker to my friend @spikeheap who helped me origionally setup emoncms-docker. His thoughts are:
The work on the raspberry pi docker image looks promising. A couple of things I’d consider early on to prevent fragmentation are:
- extract the common lines from Dockerfile into a script, so you can have debian and raspian versions sharing the build steps.
- remove the duplicate
smx…
env files and have a single environmental config wherever possible.
It would be great to merge this raspi into master after we make the suggested changes to managed the two platforms. It's an interesting challange managing cross-platform docker.
Will the 'single container application' be based on a fork of this repo or are you starting again from scratch? Does this mean that you don't be doing any more work on this branch? If we did choose to run docker on our pre-build raspi SD card images to run our enviroment it may well be better to use a single container application as your suggest for simplicity. Let me know how you get on.
Hey I basically just smushed all the Dockerfiles together and stuck all the services needed under supervisord! Ideally it wouldnt be a fork but it is sort of hard to chain Dockerfiles together in this way and there are differences between rpi and normal server versions reflected in these. Most of it is common though as you say so there must be a way to have a single build system. Its almost working as well just having some bash scripting issues which should be resolved soon!
That SMX env is just me iterating, i agree it shouldnt be there!
Sorry its coming along in fits and starts, lots of other things going on at work. Hopefully I can set some time aside for this later in the week
I mean you could even just have a makefile which generates either one Dockerfile or three depending on arch.
Ill try and push what I have got as it has changed quite a bit so at least the branch isnt stale .
Hey Ben, sorry for resurrecting an old thread. I'm keen to kickstart RasPi docker Emoncms, emonhub etc implementation. What's the current status of docker raspi Emoncms? Have you got a working image I could use as a starting point? I'm keen to explore moving to using docker on our pre-built emonSD RasPi SD card images
yes me too.. i have actually produced (in last few months for nobel grid project) a docker image for both emonhub and emonpi-emoncms stack for alpine linux (which is now much more popular than when we first started looking at this).
That's great! Nice work. Do you have a github repo with the docker files so I can try and replicate?
On Tue, 24 Jul 2018 at 19:13, Ben Aylott notifications@github.com wrote:
yes me too.. i have actually produced (in last few months for nobel grid project) a docker image for both emonhub and emonpi-emoncms stack for alpine linux (which is now much more popular than when we first started looking at this).
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/emoncms/emoncms-docker/issues/5#issuecomment-407501627, or mute the thread https://github.com/notifications/unsubscribe-auth/AAuUPEL4ge2x51fnRxzq_V3gMMYLyfSVks5uJ2OvgaJpZM4JiPKO .
-- Glyn Hudson
https://openenergymonitor.org http://zerocarbonadventures.co.uk
Do you run Alpine OS on the Pi, I see alpine is now available for the Pi. Or are you still running raspbian on the Pi with docker running alpine inside docker? Do you have a link to your docker files?
I have a problem using docker and emoncms: WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested
Where can i find arm/v7 platform for emoncms?
Run Emoncms docker on a Raspberry Pi
- Requirement to run root file-system in read-only for long term SD card life span as mentioned on Hypriot docker issues https://github.com/hypriot/image-builder-rpi/issues/119#issuecomment-239665501
Two options:
1. Start with Hypriot RasPi SD card
Either build up Emoncms in a single image using resin/rpi-raspbian/ as base
Or use docker-compose multi container setup with resin/rpi-raspbian/ or nidorpi/rpi-php as the base
2. Start with Raspbian Jessie image
Instructions to install Docker on Raspbian Jessie :
/home/pi/data