Create pip installer for mqttwarn

[metbril]

Improvement: create pip installer for mqttwarn

[jpmens]

If we do this we should probably package the services without external dependencies along with mqttwarn (e.g. smtp, file, syslog, and several others). For those with external library / package dependencies we should probably offer mqttwarn-<service> packages? Is that the way it's done?

So, who's going to volunteer to do this? :-)

[jpmens]

If somebody who has experience with Python packaging reads this (and for some inexplicable reason doesn't want to volunteer to do the actual work), maybe you can give me tips on how to best start. I know roughly what to do, but perhaps you can help me move this monolith to something package-able.

[amotl]

Hi there,

i will be happy to take this job - it has always been on my agenda ;]. Regarding "dynamic dependencies", there is actually a good way to handle this appropriately without making thousands of "mqttwarn-" modules which would drain our sanity.

I will send a proposal as PR and then we can discuss it further. Do you have any plans on how quick all this should happen?

Cheers, Andreas.

[jpmens]

I was hoping you would take this on. Make it happen, and make it happen when it's convenient to you. As for plans on how quick? Well, I think you know me a bit: right now is fine with me. :-)

At your convenience of course.

[jpmens]

I note that this exists for FreeBSD; I haven't looked at it though.

[amotl]

I note that this exists for FreeBSD; I haven't looked at it though.

Thanks! Unfortunately, this won't give us PyPI source distribution (sdist) compatibility, where we should head first before thinking about full distribution packages.

As for plans on how quick? Well, I think you know me a bit: right now is fine with me. :-)

I have scheduled the improvements to mqttwarn for the upcoming weekend. Cheers!

[amotl]

Dear @jpmens, @sumnerboy12, @ckrey and the whole mqttwarn community,

Introduction

We were finally able to catch some time to wrap our minds around the gentle refactoring and packaging improvements we wanted to apply to mqttwarn and wanted to tell you about the progress we made so far. Please also just consider this as a proposal into a possible direction mqttwarn might evolve. We are happy to hear back from you.

Status quo

We started our work in the "develop" branch of this repository and it will stay there until we are satisfied to make it the new default. We outlined possible tasks for the upcoming releases in a todo file, which might progressively be transferred into GitHub issues on demand.

As the line count of the former mqttwarn.py already crossed the 1000 line mark, we tried our best to split and refactor the functionality into different modules appropriately to encourage future growth and further refactorings of the code base.

mqttwarn has become the new designated main entrypoint to the program, which is defined in setup.py and will get materialized by Python package setup mechanisms. For the curious: The actual code being executed is located at mqttwarn.commands:run.

Getting started

Installing mqttwarn using pip directly from the GitHub repository should work so far, so it would be cool if someone of you could give this a testdrive.

As the environment obviously will lack appropriate mqttwarn.ini and samplefuncs.py starter files after installing mqttwarn as a Python package (in contrast to using mqttwarn from the Git repository), users should follow the usage documentation to create the example files interactively.

Further information

Along the lines, we started adding some neat additional features like capabilities to launch notification service plugins interactively from the command line. For further information about that, see also the section Running notification plugins in the new README file.

Thanks for listening and have fun!

With kind regards, Andreas.

---

P.S.: As installing this directly from the Git repository would unconditionally install the still existing folders "examples" and "vendors" into the global top-level Python library directory at /usr/local/lib/python2.7/site-packages, we recommend an isolated installation inside a Python virtualenv - at least until the package is ready for prime time and has been published on PyPI.

So, just issue

virtualenv .venv27
source .venv27/bin/activate

before proceeding with the installation using pip install.

[jpmens]

First notes after installing in virtuallenv and then pip install from github repository:

1. six module is missing; fix with pip install six
2. make-config and make-samplefuncs are excellent, but these should produce much simpler examples. In particular the latter might create a very easy function which transforms some data. I think all the OwnTracks stuff in there is, at least for beginners, a bit intimidating.
3. Logging looks great
4. We (I) should probably also think about creating a simple test harness. Maybe just a short shell script which publishes topics/payloads that match a configuration created by make-config so that people can see how it all fits together.
5. Your name, @amotl, needs to go into __init__.py authors!

[jpmens]

Traceback (most recent call last):
  File "/Users/jpm/tmp/mqttwarn-temp/venv/bin/mqttwarn", line 7, in <module>
    from mqttwarn.commands import run
  File "/Users/jpm/tmp/mqttwarn-temp/venv/lib/python2.7/site-packages/mqttwarn/commands.py", line 13, in <module>
    from mqttwarn.core import bootstrap, connect, cleanup, run_plugin
  File "/Users/jpm/tmp/mqttwarn-temp/venv/lib/python2.7/site-packages/mqttwarn/core.py", line 16, in <module>
    from mqttwarn.context import RuntimeContext, FunctionInvoker
  File "/Users/jpm/tmp/mqttwarn-temp/venv/lib/python2.7/site-packages/mqttwarn/context.py", line 6, in <module>
    from mqttwarn.util import sanitize_function_name, load_function
  File "/Users/jpm/tmp/mqttwarn-temp/venv/lib/python2.7/site-packages/mqttwarn/util.py", line 11, in <module>
    from six import StringIO
ImportError: No module named six

[amotl]

Dear @jpmens and @juzam,

thanks for trying the current state of the refurbished code base already - good to hear the thing works in general modulo the minor hiccups you encountered.

@jpmens I totally agree with all your notes and will take them into the next wave of improvements. As i am currently travelling, i humbly ask for your patience.

Thanks also for addressing #81 and #309, i am happy about their outcomes!

With kind regards, Andreas.

[jpmens]

Something ocurred to me as I was stepping out of the train a moment ago: How about we have make-config create a configuration which can be tested with

mqttwarn make-pubs

That command would use the MQTT broker config from mqttwarn.ini and create three or four publishes which would allow a quick turnaround test that everying works. Maybe messy. Maybe not.

[amotl]

Absolutely agree with that. Let's have exactly this as a basic quick turnaround test harness for the upcoming 0.15.0 release and let's build a full-fledged and thorough test environment for the 1.0.0 release later this year.

I would eventually call some offspring of that mqttwarn selftest, see also https://github.com/daq-tools/kotori/commit/6222ddde just added recently for the very same purpose: Have a neat wrapper around mosquitto_pub for doing some basic MQTT publishing in order to quickly see everything works.

[amotl]

The issue regarding the missing dependency to the "six" package has been addressed with release 0.10.1, thanks again for testing!

[jpmens]

@amotl is there any reason we should not merge this into master?

[amotl]

Hey @jpmens,

thanks for following up on this.

The codebase refactoring is pretty much finished and there are just minor leftovers: I wanted to also converge the "vendor" directory as well as examples/amqp-puka-get.py into the mqttwarn namespace but haven't been able to catch up on this.

I will see what i can do to this until the end of this week but otherwise i also consider this to be ready so we can go ahead with merging to master and finally bake Python packages if everyone is fine with the overhaul.

Cheers, Andreas.

[amotl]

Dear mqttwarn community,

mqttwarn 0.11.3 is the first release of mqttwarn on PyPI [1]. Enjoy installing it by just typing pip install mqttwarn from now on.

While this has been taken from this repository's "develop" branch, it shouldn't do anyone any harm.

However, before further announcing it, we should all test the release thoroughly and I am humbly asking for support on this.

This is really a milestone for mqttwarn and I would like to thank @jpmens, @sumnerboy12 and all others who contributed over the last five years of development time. You know who you are.

With kind regards, Andreas.

[1] https://pypi.org/project/mqttwarn/

[jpmens]

@amotl thank you very much for all the work you’re putting into mqttwarn!

[dlangille]

I tried to build mqttwarn-0.11.3 for FreeBSD and Python 3.6: I started patching... I have patches for 68 files, mostly to do with Exceptions in mqttwarn/services .... Thing is: I'm not sure I have patched the exceptions correctly.

This is an example, pasted so tabs may be wrong:

--- mqttwarn/services/ifttt.py.orig 2019-11-24 23:59:13 UTC
+++ mqttwarn/services/ifttt.py
@@ -30,7 +30,7 @@ def plugin(srv, item):
         url = "https://maker.ifttt.com/trigger/" + event + "/with/key/" + apikey
         requests.post(url, data=payload)
         srv.logging.debug("Successfully sent ifttt event")
-    except Exception, e:
+    except Exception as e:
         srv.logging.warning("Cannot send ifttt event: %s" % (str(e)))
         return False

If wanted, I'll send in a merge request. These are all over the place.

[amotl]

Dear Dan,

a pull request for getting the codebase closer to Python3 compatibility would be very welcome. Please make sure you are sending it against the "develop" branch of the repository.

Either JP will merge it right away or I will pick it up when coming back from my travels.

Thanks already!

With kind regards, Andreas.

On 25 November 2019 00:16:50 WET, Dan Langille notifications@github.com wrote:

I tried to build mqttwarn-0.11.3 for FreeBSD. I started patching... I have patches for 68 files, mostly to do with Exceptions in mqttwarn/services .... Thing is: I'm not sure I have patched the exceptions correctly.

This is an example, pasted so tabs may be wrong:

--- mqttwarn/services/ifttt.py.orig    2019-11-24 23:59:13 UTC
+++ mqttwarn/services/ifttt.py
@@ -30,7 +30,7 @@ def plugin(srv, item):
url = "https://maker.ifttt.com/trigger/" + event + "/with/key/" +
apikey
        requests.post(url, data=payload)
        srv.logging.debug("Successfully sent ifttt event")
-    except Exception, e:
+    except Exception as e:
        srv.logging.warning("Cannot send ifttt event: %s" % (str(e)))
        return False

If wanted, I'll send in a merge request. These are all over the place.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/jpmens/mqttwarn/issues/127#issuecomment-557944816

-- Sent from my mind. This might have been typed on a mobile device, so please excuse my brevity.

[amotl]

Dear @jpmens, @sumnerboy12, @dlangille and the whole mqttwarn community,

if nobody has any objections on that, I would merge the "develop" branch into the "master" branch in order to continue working on #336 there.

With kind regards, Andreas.

[jpmens]

@amotl please do so!

[amotl]

Hi there,

the latest and greatest mqttwarn-0.13.2 has just been released [1].

In order to mitigate eventual fallout coming from this process, we want to encourage everyone to install mqttwarn from PyPI, test it using their favorite plugins and report the outcome back to us.

Thanks already!

With kind regards, Andreas.

[1] https://pypi.org/project/mqttwarn/

[jpmens]

Thank you, @amotl, for the incredible job you're doing during this transition! Hugely appreciated!

[amotl]

Likewise, thanks for all the work you've put into mqttwarn, I appreciate becoming part of the journey.