debian package emoncmsHub

[Dave-McCraw]

@Jerome-github / @pb66 - you should be able to try installing emoncms-hub from the apt repo and it should run through basic configuration questions and get up and running...

I had to make a couple of slightly odder changes to get this to work, but should hopefully be possible for someone to try this out.

For now I've just bundled the existing master as 0.1 (but partly rebadged as emoncmsHub where I had to make a choice).

I probably need to spend another day or so tweaking this and writing up new documentation...

[Dave-McCraw]

Bah, emonHub of course - typical late night error :facepunch:

At least the name is easily enough tweaked...

[pb66]

I gave the 'package of many names' a spin, very impressed by the menu system and all seemed ok until the final install steps,

No need to rename oemgateway.init.dist (are we reconfiguring?) Start automatically on boot... update-rc.d: using dependency based boot sequencing update-rc.d: error: unable to read /etc/init.d/oemgateway To update emoncmsHub configuration, run 'dpkg-reconfigure emoncms-hub' To start now, run 'sudo service oemgateway start [log]' Remember that you can't run emoncmsHub and the emoncms 'raspberrypi' module at the same time!

I copied /etc/init.d/emoncmsHub to /etc/init.d/oemgateway and was able to start after sudo chmod 755 /etc/init.d/oemgateway && sudo update-rc.d oemgateway defaults.

I will carry on testing emonHUB, let me know if there is anything else I can help with, readme's etc maybe.

Paul

[pb66]

Is there any reason emonHUB should reject a packet via socket on 50011 ? works ok with rfm2pi though, I will look into it tomorrow as it's late, maybe something I've overlooked. I will swap back for now as i need the packet listener and i'm just testing on sd not hdd, so will setup properly and report back. Paul

[Dave-McCraw]

Unfortunately I couldn't get the debian installer to copy oemgateway.init.dist into /etc/init.d/ (I think it may apply validation to the file types and locations) so instead I renamed the file emoncmsHub.

You can just substitute emoncmsHub for oemgateway in the commands and it should still work - although obviously it should be called emonHub, or perhaps literally emonHUB (all caps makes me expect it to be an acronym though).

The installer needs a bit of a clean up as it's still trying to do (unnecessary) stuff with the init.dist file. I also need to update it to add in the socket. I was just trying to get the basics to work and forgot to go back and add that in at the end.

I wanted to minimise the changes I made to the source as we should really rename it etc, in the upstream repo.

I should have a new 0.2 version later that is a bit more consistent.

[lafrech]

I suggest we settle quickly on the name (emonHUB or emonHub, I don't care), so that we can change the code in OEM Gateway dev branch, do all renaming correctly (perhaps even rename the repo, unless it breaks too many links), get a v2.0 will heopfully definitive names, and then publish a package without the naming mess.

At some point, I'll publish a message on the forum to let everyone know about the renamings. We may also have to update the doc/wiki.

My point is, before communicating on the brand new package, let's ensure we won't break everything two weeks later with a new name, so I'd rather wait a bit, even if it delays the package. No rush, anyway.

I try to work on the dev branch asap.

(If we're unsure about the mysql buffer, we can remove it from the v2 release but keep the rest (renaming of dispatchers, new abstract buffer) and put it back in dev branch for v3 or v2.1, so that we can release v2 quickly.)

[lafrech]

Dave, for the numbering, do you want to do use the emonHUB version number and append -1, -2, -3 for deb package version number ?

[Dave-McCraw]

For the name, my vote is emonHub so it matches emonBase.

I agree on your approach - get everything fixed up on the dev branch then "launch" the package into apt at the same time (the final numbered versions will supercede these 0.x test versions anyway).

Have you considered just archiving the oemgateway repository and creating a new emonHub repository? (Just edit the readme to have a preface saying "no longer recommended, see the emonHub repo")

That way we can have a real clean start with no worries about someone doing a git pull and twisting themselves in knots. At the moment if someone has edited any of the files and we do an extensive rename / rebrand to "dispatcher" it will give them a hell of a merge.

My vote is to supply only in-memory with the initial release so we don't have to support two things at the same time - installing emonHub and making the database buffer work! It can just be added as 2.1, 2.2, whatever.

[lafrech]

Have you considered just archiving the oemgateway repository and creating a new emonHub repository? (Just edit the readme to have a preface saying "no longer recommended, see the emonHub repo")

Good point. Let's do that.

My vote is to supply only in-memory with the initial release so we don't have to support two things at the same time - installing emonHub and making the database buffer work! It can just be added as 2.1, 2.2, whatever.

Yes.

[Dave-McCraw]

In terms of how the packaging will work, I envision that you will tag your master with whatever version number, and then I will import that using a script, build the deb and push it to apt.

For emoncms I've been pushing up version 8.0.9 etc. as just 8.0.9 and saving the -1, -2 for any extra changes I need to make. I think the correct approach really is to append -1 from the beginning, so that's what I'd propose going forward.

The first thing we'd announce on the forum could be emonHub v1.0 (in APT as 1.0-1)?

[lafrech]

Or we continue on oemgateway's numbering. 1.0.2 already, I was expecting to move to 2.0 for emonHub. But if everything is so new, we can reset the numbering, after all. I don't care. OK for 1.0.

[pb66]

This is all good, /etc/init.d/oemgateway filename issue expected & temporary - check socket listener is not yet complete (ie not broken) - check

Regardless of how it is "branded" outside the code I think no capitalization or hyphens etc for user accessible commands, files & folders for simple consistency, deeper in the code normal coding rules can apply, but anything a general user encounters should be just "emonhub".

Regards the repo if you want we could make it its own repo, this will accomadate any future emonHub modules or associated files and allow multiple maintainers, I have snapped up emonhub repo as it was available I can transfer ownership to one of you if you want to go that route.

The default configuration maybe better set as in-memory regardless, initial install my be to sd card only which may shorten life if user unaware db created, the extended functions could then be enabled by user for a full hdd installation.

Paul

[Dave-McCraw]

I'm hoping I can learn enough about debconf to offer the user an initial choice: are you just forwarding to emoncms.org? If so, it would just ask for the API key and the RFM12B settings.

If not, it would ask you for dispatcher config and at the end ask for optional emoncms.org API key (as at present).

[pb66]

Something just occurred to me about the packaging of emonhub, should the rfm2pi be an optional module? not everyone using a pi will use rfm2pi board and other debian platforms won't support the pi board. There was also a comment made on the forum about oemgateway being able to run on windows, so should the emonHub be independent? Probally the rfm2pi "module" should be optional, in its own repo altogether. Although the rfm2pi is currently the dominant source of data that could change, the pi is able to interface & process data itself, there is usb, urls and sockets already available but not fully explored probally due to limited interfacing which emonHub is going to overcome. Also if down the line rfm2pi gets replaced by something not directly compatible it may cause problems in an effort to support both options from within emonHub rather than just installing one or other module. This would again, simplify the standard emonHub install (no serial to contend with) and compartmentalize the rfm2pi to its own repo with its own readme. I Know it's more work for you guys but it may make sense to do it sooner rather than later, I'm just thinking about avoiding ending up where we are with the emoncms - rfm2pi relationship currently. I'm good with whatever you guys decide but I felt it should be discussed at least.

[Dave-McCraw]

I think the difficulty would be the configuration. Would you have two config files, one for emonhub-rfm12pi and one for the main emonhub?

I like separation of concerns a lot, and I guess the use case of having this installed on windows (or anywhere that isn't a Pi) does suggest that interfacing with the rfm12b might not be core functionality.

On the other hand, we have already separated that functionality into a modular listener concept. Is it decomposing one step too far to make separate systems for each listener?

Good question.

[lafrech]

We could have a one file per listener/disp/buff policy, and package them separately, then let the user do the config in the config file according to which packages he actually installed. This is how things are done with plugins in Roundcube (php webmail), for instance.

I don't think RFM2Pi is so irrelevant, even on windows. Jan Becker uses a Jeelink USB module (http://openenergymonitor.org/emon/node/4947) that may be compatible (not tested, but I'd like this confirmed or fixed).

I think we can package the whole stuff altogether and let the user pick his choice through the configuration file. And if one listener / buffer / dispatcher has a dependency with something important, we'd add this as a suggest, not a depends. For instance mySQL.

It might not be that obvious.

The imports are made globally, not per item (listener, dispatcher, etc.). So if we want to be able to install only the dependencies actually used, we may have to think of a way to import at item level, or separate items in different files.

If we create a listeners/ directory, for instance, and want all listeners to be imported by import *, we need a file in there with the list of the files. It is not just about adding the file through apt. Not sure I'm clear. Anyway, I had been thinking about this a long time ago and left it aside. Maybe it's obvious for someone used to python distribution.

Importing at instantiation may be inelegant, but it should work. Along with an exception and a CRITICAL error message about missing dependency.

[pb66]

Sorry if I suggested RFM2PI was irrelevant, if anything I was hoping independence would promote its use by allowing the "module" to develop at its own speed without being possibly held back by compulsory inclusion in emonHub. Currently RFM2Pi is by far the most relevant input source but that may change if users have other easily interfaced options and actually a separate RFM2PI module may deter any "as its already built in" thoughts and promote other avenues.

Also Glyn has his emonPi arriving soon which emonHub needs to accommodate or things may splinter, features like OOK & in-circuit AVR programming will need other dependencies, avrdude etc, the emonPi will probably need its own module? if so will it have its own RFM12B support and will that clash with the emonHub's in-built RFM2PI support ?

Permanent RFM2Pi inclusion, to me also highlights the current bias towards raspberrypi which may not be healthy for advancing on other platforms,

I think any bias towards platform or delivery method should be minimized but where decisions have to be made in favor of easier for git or for apt the fact apt is specifically aimed at simplifying & standardizing installs whilst git is primarily for more advanced users and development, should mean easier for apt to keep apt as easy as possible despite a (currently) smaller user base.

Maybe a "Windows" module (not package) would help Jan Becker and other windows users, in the same way the serial port is monitored by the RFM2Pi module, maybe other windows sources could be injected by a "windows" module.

Currently we have only discussed data collection, how does this work having separate modules that may also need to be called upon to issue commands or data?

Oh dear ! does this mean a listener should also be able to speak? how does the OemGatewayRFM2PiListenerRepeater work?

I suppose at the very least we need be offer a way to disable the RFM2Pi stuff to avoid trying to hog the serial port when a RFM2Pi board is not present.

Paul

[pb66]

@Dave-McCraw are you sticking with the package name pkg-emon-hub or changing it to pkg-emonhub ? I only ask as earlier I went to write comment on oemgateway repo and wanted to ref this thread but didn't want a dead link if you change later :-)

[Dave-McCraw]

It's up to @Jerome-github really, he is the upstream for this repo. I'm guessing I should rename to the latter?

The example of the MySQL buffer actually makes me think that separate packages would be a great way forward. This way we can have a hard dependency on mysql-server etc. when that module is installed, and if you don't install it, nice and clean in-memory implementation with few dependencies.

What we'd want to do is have some kind of self-registration mechanism whereby the main Python class invokes each file and in turn they supply human-readable name for config purposes etc.

I will try to make time this evening to investigate some of this.