ConfigurableFirmata does not work in Arduino 1.5.7

[soundanalogous]

Arduino keeps changing the way they recursively include libraries. This seems to break something in configurable firmata with each new Arduino 1.5.x beta release. Currently the compiler throws an error on the Ethernet.h import EthernetClientStream.cpp. This is not an issue in Arduino 1.5.6 so there was a regression in the most recent version of Arduino.

I'll provide updates when I have time to look into this further.

[mcinnes01]

Hi,

I just wondered if there is any update on this, I am new to arduino and am finding it quite difficult getting firmata working over ethernet. I am using an arduino nano v3 and ENC28J60 ethernet shield. Are there any workarounds or pointers you can give me? My aim is to get this working with my raspberry pi as the controller running node-red and using mqtt as the message transport.

Many thanks

Andy

[ntruchsess]

I just testet, Arduino-IDE would not include the Ethernet-library dir in include-path while compiling EthernetClientStream.cpp. (That is the same with all versions of Arduino-IDE that I have used so far). Other includes (e.g. Servo.h) will do the same, you have to include them all in the ino when you compile the sketch for the very first time. You may then configure ConfigurableFirmata.ino by removing all includes you do not want to be part of the sketch and compile again without getting the error. If you change the board to other arduino-model you have to re-do the whole procedure.

I don't think we can solve this without either design, write and contribute a change to the Arduino-IDE that will resolve Include-directories for utility-classes that are not used in the sketch or move all Firmata-features into their own Arduino-library so they don't get compiled when not being used in the sketch.

Seems this issue is very simmilar to #132

library-format and directory-structure of ConfigurableFirmata seems to match Arduino 1.5.7 expectations. It even picks up Firmata from library-directory without removing it from the libraries that come with the IDE.

[ntruchsess]

@mcinnes01 what problems do you run into? Just get and install UIPEthernet from https://github.com/ntruchsess/arduino_uip. Then uncomment the include UIPEthernet and configure ethernet-settings (mac, ip etc...), compile once (you will get an error the sketch is to big), then remove all features you do not neet and compile again.

[mcinnes01]

Hi @ntruchsess

So I have freshly installed the arduino ide 1.5.7, updated firmata with the configurable branch as described in the readme, downloaded UIPEthernet to the same C:\Program Files (x86)\Arduino\libraries\UIPEthernet and then ensured all includes are uncommented for first build.

This gives me:

UIPEthernet\Dhcp.cpp.o: In function DhcpClass::presend_DHCP()': C:\Program Files (x86)\Arduino\libraries\UIPEthernet/Dhcp.cpp:30: multiple definition ofDhcpClass::reset_DHCP_lease()' Ethernet\Dhcp.cpp.o:C:\Program Files (x86)\Arduino\libraries\Ethernet\src/Dhcp.cpp:32: first defined here UIPEthernet\Dhcp.cpp.o: In function __static_initialization_and_destruction_0': C:\Program Files (x86)\Arduino\libraries\UIPEthernet/Dhcp.cpp:130: multiple definition ofDhcpClass::presendDHCP()' Ethernet\Dhcp.cpp.o:C:\Program Files (x86)\Arduino\libraries\Ethernet\src/Dhcp.cpp:132: first defined here UIPEthernet\Dhcp.cpp.o: In function DhcpClass::printByte(char*, unsigned char)': C:\Program Files (x86)\Arduino\libraries\UIPEthernet/Dhcp.cpp:445: multiple definition ofDhcpClass::getLocalIp()' Ethernet\Dhcp.cpp.o:C:\Program Files (x86)\Arduino\libraries\Ethernet\src/Dhcp.cpp:447: first defined here UIPEthernet\Dhcp.cpp.o: In function DhcpClass::beginWithDHCP(unsigned char*, unsigned long, unsigned long)': C:\Program Files (x86)\Arduino\libraries\UIPEthernet/Dhcp.cpp:450: multiple definition ofDhcpClass::getSubnetMask()' Ethernet\Dhcp.cpp.o:C:\Program Files (x86)\Arduino\libraries\Ethernet\src/Dhcp.cpp:452: first defined here UIPEthernet\Dhcp.cpp.o: In function DhcpClass::checkLease()': C:\Program Files (x86)\Arduino\libraries\UIPEthernet/Dhcp.cpp:455: multiple definition ofDhcpClass::getGatewayIp()' Ethernet\Dhcp.cpp.o:C:\Program Files (x86)\Arduino\libraries\Ethernet\src/Dhcp.cpp:457: first defined here UIPEthernet\Dhcp.cpp.o: In function DhcpClass::checkLease()': C:\Program Files (x86)\Arduino\libraries\UIPEthernet/Dhcp.cpp:460: multiple definition ofDhcpClass::getDhcpServerIp()' Ethernet\Dhcp.cpp.o:C:\Program Files (x86)\Arduino\libraries\Ethernet\src/Dhcp.cpp:462: first defined here UIPEthernet\Dhcp.cpp.o: In function DhcpClass::parseDHCPResponse(unsigned long, unsigned long&)': C:\Program Files (x86)\Arduino\libraries\UIPEthernet/Dhcp.cpp:465: multiple definition ofDhcpClass::getDnsServerIp()' Ethernet\Dhcp.cpp.o:C:\Program Files (x86)\Arduino\libraries\Ethernet\src/Dhcp.cpp:467: first defined here UIPEthernet\Dhcp.cpp.o: In function DhcpClass::printByte(char*, unsigned char)': C:\Program Files (x86)\Arduino\libraries\UIPEthernet/Dhcp.cpp:471: multiple definition ofDhcpClass::printByte(char, unsigned char)' Ethernet\Dhcp.cpp.o:C:\Program Files (x86)\Arduino\libraries\Ethernet\src/Dhcp.cpp:473: first defined here UIPEthernet\Dhcp.cpp.o: In function DhcpClass::send_DHCP_MESSAGE(unsigned char, unsigned int)': C:\Program Files (x86)\Arduino\libraries\UIPEthernet/Dhcp.cpp:134: multiple definition ofDhcpClass::send_DHCP_MESSAGE(unsigned char, unsigned int)' Ethernet\Dhcp.cpp.o:C:\Program Files (x86)\Arduino\libraries\Ethernet\src/Dhcp.cpp:136: first defined here UIPEthernet\Dhcp.cpp.o: In function DhcpClass::parseDHCPResponse(unsigned long, unsigned long&)': C:\Program Files (x86)\Arduino\libraries\UIPEthernet/Dhcp.cpp:252: multiple definition ofDhcpClass::parseDHCPResponse(unsigned long, unsigned long&)' Ethernet\Dhcp.cpp.o:C:\Program Files (x86)\Arduino\libraries\Ethernet\src/Dhcp.cpp:254: first defined here UIPEthernet\Dhcp.cpp.o: In function DhcpClass::request_DHCP_lease()': C:\Program Files (x86)\Arduino\libraries\UIPEthernet/Dhcp.cpp:34: multiple definition ofDhcpClass::request_DHCPlease()' Ethernet\Dhcp.cpp.o:C:\Program Files (x86)\Arduino\libraries\Ethernet\src/Dhcp.cpp:36: first defined here UIPEthernet\Dhcp.cpp.o: In function `DhcpClass::beginWithDHCP(unsigned char, unsigned long, unsigned long)': C:\Program Files (x86)\Arduino\libraries\UIPEthernet/Dhcp.cpp:11: multiple definition of DhcpClass::beginWithDHCP(unsigned char*, unsigned long, unsigned long)' Ethernet\Dhcp.cpp.o:C:\Program Files (x86)\Arduino\libraries\Ethernet\src/Dhcp.cpp:13: first defined here UIPEthernet\Dhcp.cpp.o: In functionDhcpClass::checkLease()': C:\Program Files (x86)\Arduino\libraries\UIPEthernet/Dhcp.cpp:391: multiple definition of DhcpClass::checkLease()' Ethernet\Dhcp.cpp.o:C:\Program Files (x86)\Arduino\libraries\Ethernet\src/Dhcp.cpp:393: first defined here UIPEthernet\Dns.cpp.o: In functionIPAddress::operator=(IPAddress const&)': C:\Program Files (x86)\Arduino\hardware\arduino\avr\cores\arduino/IPAddress.h:28: multiple definition of DNSClient::begin(IPAddress const&)' Ethernet\Dns.cpp.o:C:\Program Files (x86)\Arduino\hardware\arduino\avr\cores\arduino/IPAddress.h:28: first defined here UIPEthernet\Dns.cpp.o: In functionDNSClient::inetaton(char const, IPAddress&)': C:\Program Files (x86)\Arduino\libraries\UIPEthernet/Dns.cpp:58: multiple definition of DNSClient::inet_aton(char const_, IPAddress&)' Ethernet\Dns.cpp.o:C:\Program Files (x86)\Arduino\libraries\Ethernet\src/Dns.cpp:59: first defined here UIPEthernet\Dns.cpp.o: In functionDNSClient::BuildRequest(char const)': C:\Program Files (x86)\Arduino\libraries\UIPEthernet/Dns.cpp:175: multiple definition ofDNSClient::BuildRequest(char const_)' Ethernet\Dns.cpp.o:C:\Program Files (x86)\Arduino\libraries\Ethernet\src/Dns.cpp:176: first defined here UIPEthernet\Dns.cpp.o: In functionDNSClient::ProcessResponse(unsigned int, IPAddress&)': C:\Program Files (x86)\Arduino\libraries\UIPEthernet/Dns.cpp:255: multiple definition ofDNSClient::ProcessResponse(unsigned int, IPAddress&)' Ethernet\Dns.cpp.o:C:\Program Files (x86)\Arduino\libraries\Ethernet\src/Dns.cpp:256: first defined here UIPEthernet\Dns.cpp.o: In functionDNSClient::getHostByName(char const_, IPAddress&)': C:\Program Files (x86)\Arduino\libraries\UIPEthernet/Dns.cpp:118: multiple definition of `DNSClient::getHostByName(char const, IPAddress&)' Ethernet\Dns.cpp.o:C:\Program Files (x86)\Arduino\libraries\Ethernet\src/Dns.cpp:119: first defined here collect2: error: ld returned 1 exit status

When I comment out SPI.h and Ethernet.h I get the following:

C:\Program Files (x86)\Arduino\libraries\Firmata\utility\EthernetClientStream.cpp:22:22: fatal error: Ethernet.h: No such file or directory

include

                  ^

compilation terminated.

Thanks for the help

Andy

[ntruchsess]

don't include both Ethernet.h and UIPEthernet.h at the same time.

just fixed the error on Ethernet.h not being included: https://github.com/firmata/arduino/commit/c84f58dfa6ce6a1563d320c2848a6b00db18afcf so you don't need to include Ethernet.h or UIPEthernet.h at all when you do the initial compile and do not intend to to Ethernet at all. EthernetClientStream actually does not have a real dependency on Ethernet as it includes Client.h and IPAddress.h directly.

[mcinnes01]

Hi @ntruchsess

Thanks for your quick response :) I take this would mean everything is working as expected and now just to remove some un-required elements?

Sketch uses 39,366 bytes (128%) of program storage space. Maximum is 30,720 bytes. Global variables use 2,354 bytes (114%) of dynamic memory, leaving -306 bytes for local variables. Maximum is 2,048 bytes. processing.app.debug.RunnerException: Sketch too big; see http://www.arduino.cc/en/Guide/Troubleshooting#size for tips on reducing it. at processing.app.Sketch.size(Sketch.java:1676) at processing.app.Sketch.build(Sketch.java:1590) at processing.app.Sketch.build(Sketch.java:1509) at processing.app.Editor$DefaultRunHandler.run(Editor.java:1915) at java.lang.Thread.run(Unknown Source)

Many thanks

Andy

[soundanalogous]

verified

[mcinnes01]

Hi @ntruchsess

I have managed to load this on to my nano v3 now, but wondered if you have any pointers on how I can go about connecting to the arduino over ethernet using node-red?

I have node-red running on a raspberry pi and have installed firmata on that, but I get no arduino node in the advanced category of node-red and all the blog posts and tutorials I find seem to only describe the scenario of using firmata in node-red via serial or usb? I.e. they reference the name of the arduino board such as "device":"/dev/ttyUSB0" opposed to the ip of the board.

Thanks again,

Andy

[rwaldron]

@mcinnes01 you may also be interested in https://github.com/octoblu cc @chrismatthieu @virgilvox @iamruinous

[chrismatthieu]

Thanks @rwaldron!

@mcinnes01 We have an Arduino sketch called Microblu (https://github.com/octoblu/microblu_mqtt) which allows you to control and Arduino (with a WiFi or Ethernet shield) with our drag & drop Chrome app called Nodeblu (https://chrome.google.com/webstore/detail/nodeblu/aanmmiaepnlibdlobmbhmfemjioahilm) which is powered by Johnny-Five!

[ntruchsess]

@mcinnes01: from what I know there's no support for Ethernet in Node.js arduino-firmata library (https://github.com/shokai/node-arduino-firmata/blob/master/src/arduino-firmata.coffee) yet, it's capable to connect via serial (and USB). Therefore no Firmata over Ethernet support in Node-red.js either. Maybe you'd like to contribute that? (not my project, just looked into the code... I use https://github.com/ntruchsess/perl-firmata for Firmata over Ethernet)

[ntruchsess]

@rwaldron, @chrismatthieu : did you try to microblu_mqtt with https://github.com/ntruchsess/arduino_uip to run with enc28J60-based shields? I have reports that pubsubclient does work, so maybe microblue_mqtt does as well?

[chrismatthieu]

We are able to connect ethernet-powered Arduinos to Meshblu (https://github.com/octoblu/meshblu) and Nodeblu and control their pins via Johnny-Five. You will need to generate a new UUID on Meshblu first and add the new UUID and Token to the Arduino sketch in order for it to connect. Here are API docs for Meshblu: http://developer.octoblu.com.

[rwaldron]

@ntruchsess I've never seen that implementation of Firmata before, the most commonly used library is https://github.com/jgautier/firmata . It stands to reason that this library should be updated to support data over Ethernet

[soundanalogous]

I second adding Ethernet support to https://github.com/jgautier/firmata. I think Julian had done some work on it an may exist already in one of the branches. However I'm sure that implementation needs to be updated. I unfortunately don't have time to take it on for a while.

[mcinnes01]

Hi @ntruchsess, @rwaldron, @chrismatthieu

Some very interesting stuff thanks! Maybe you can tell me the best route for my scenario...

I am doing a home automation project, to control everything from sockets to lights and I will also be adding a whole host of sensors and logic. I am using mqtt, currently mosquitto running on windows server 2012. I have around 70 relays split across 2 distribution boards so I/O is important and there is a requirement for distributed devices to work together with a central controller coordinating things. I have run about 1Km of network cable around my house for this project and again have 2 racks with POE switches.

For the mains control in the distribution boards I intend on having 2 raspberry pi's, these will run node-red and also wiring Pi to provide I/O via some mcp23S17s, but there will also be some arduinos distributed around the house providing sensor readings back for the pi's to pick up and process. Where a device isn't using mqtt e.g. an arduino running firmata, its "master" would publish any readings to mqtt on its behalf.

I like the idea of using arduino's as distributed devices, they can perform timing specific activities far better than the pi or something like a netduino; which is great for LED dimming, 1 wire temperature readings etc. Also I can pick an arduino nano v3 and an ENC28J60 ethernet shield up for about £5. This has got to be the cheapest way of getting networkable I/O. Throw in a Linksys POE splitter for £10 and for £15 I can have a relatively small networkable device anywhere I can run some cat 5 to.

So far I have had little experience of writing code directly for the arduino, and I really like the flexibility of writing a bit of js in node-red, deploying it and being able to remotely I/O with an arduino running some firmware over ethernet . I find the idea of leaving the arduino completely logic free, literally a "dumb" device, very appealing.

So in a nutshell if I can find some way of keeping the arduino "dumb", and being able to interface with it over ethernet, ideally through node-red and if not, then using mqtt, that would be great. Again if it is possible to get away without hard coding logic on to the arduino that would be great.

Thanks for all your help,

Andy

[ntruchsess]

@mcinnes01 don't know what the 'best' way is. For me I'm using http://fhem.de It drives Firmata over ethernet and supports all features build into ConfigurableFirmata. It's perl and doesn't interface to mqtt (yet, but I'm working on this). I have not used any of the Node.js based stuff (yet ;-) ), so I'm not in a position to give you advise there. In respect to Firmata for home-automation: It has some pros and cons. It provides tight integration with the automation software which results in very fast (nearly instant) response times. So as an easy example you might dim your lights jusing a rotary-encoder with no noticable delay. But it also is to 100% reliant on the automation-software to run and network being stable. So if you use it to meassure your power-consumption by counting impluses from the power-meter you will not gather data while you reboot your server or unplug a switch. In respect to ConfigurableFirmata and ENC28J60: Due to the size of UIPEthernet-lib it does fit on a Nano v3 only when you deselect some features. E.g. digital+analog i/o+1-wire will work, but the combination of 1-Wire and I2C does not.

In respect to mqtt: have a look at http://mysensors.org It builds a wireless mesh-network using ardunio and nRF24L01 that you integrate with mqtt using an wireless<->mqtt gateway (Build on Adruino too...)

[chrismatthieu]

@mcinnes01, @ntruchsess, @rwaldron,

Here's how I would do this...

1. Run our open source IoT Gateblu gateway (https://github.com/octoblu/gateblu) on one of your home computers (or Raspberry Pis).
2. Connect Gateblu to our free Meshblu IoT platform (http://developer.octoblu.com) or deploy your own private Meshblu cloud in your home (https://github.com/octoblu/meshblu).
3. Our gateway has an open plugin architecture and already supports Phillips Hue, Belkin Wemo, Insteon, Lockatron, Blink1, Punchthrough Bean, and various AllJoyn and BLE devices). These plugins can be configured via the gatewayConfig API detailed on the Gateblu readme. You can write other plugins in Node.JS as needed.
4. Connect Arduinos and Spark devices to Meshblu using our Microblu MQTT sketch (https://github.com/octoblu/microblu_mqtt) for connecting these micro-controllers to our public Meshblu cloud or your private Meshblu cloud.
5. Use our Nodeblu drag & drop Chrome app (https://chrome.google.com/webstore/detail/nodeblu/aanmmiaepnlibdlobmbhmfemjioahilm) to control all of these devices in your home.

We have a new cloud-based Octoblu designer being release very soon...

[chrismatthieu]

BTW, this is the same configuration that we used in the LocalMotors' $100k RallyFighter Connected Car!

Here's a video - http://www.youtube.com/watch?v=3-swFFero-8

[mcinnes01]

Hi @ntruchsess

fhem looks very interesting, most certainly a way to get my automation up and running without reinventing the wheel. Also pearl is something I haven't really looked at before so that would also be an interesting avenue to explore!

Is fhem easy to api with, I notice it exposes lot of interfaces: simple text, JSON, XML, each of them over plain TCP/IP, SSL or HTTP. Do you have any examples of say using a webservice to turn a light on via fhem or perhaps send a push notification? I am one of those strange people who loves microsoft technology so being able create windows phone apps and my own web interfaces using mvc, etc is really interesting for me. I also have my own server 2012 running at home running 3 VMs (db, app, web) and even AD.

Hi @chrismatthieu

That looks really cool, I love the way you have built on the node-red platform. An interesting question, does this still allow you to use all the typical node-red functionality? For example I was planning on using wiring Pi (http://wiringpi.com/) on a couple of Pi's to do the relay control in my distribution boards, can I still use wiring Pi and the various "plugins" within it to use MCP23S17s etc?

I have been meaning to play with mongo for a while now so it is interesting to see you using it in meshblu. Is it possible to also use other databases?

Finally (I say finally I'm sure I could ask a lot more questions), with your arduino sketch I've had a quick look through the readme, does this support ENC28J60 boards? I am planning on using a ton of these throughout my house as I can get a nano and ENC28J60 board for about £5.

Many thanks

Andy

[chrismatthieu]

We were definitely inspired by Node-red but decided to write our own multi-tenent designer using AngularJS on the frontend and Docker sandboxes for each flow on the backend. The client and server sides of the app are connected with real time websockets allowing us to pulse the flows as a live data streams across the nodes. All flows and nodes each have a UUID (like devices do) on our Meshblu IoT platform. We did try to implement a msg object like Node-red supports to allow for flow compatibility across platforms.

We have talked about connecting Meshblu to other databases. For now you can launch the platform without Mongo configured in your environment variables and we fallback to NEDB (a file/memory-based database with a mongo-like API). If you are feeling inspired to attach Meshblu to another database, we would love a pull request.

I am not sure about the ENC28J60 boards but we use the SainSmart Ardunio boards at $16/board for testing - http://www.amazon.com/SainSmart-ATmega328P-CABLE-Included-Arduino/dp/B006GX8IAY/ref=sr_1_3?ie=UTF8&qid=1411839203&sr=8-3&keywords=sainsmart+arduino

[mcinnes01]

Hi @chrismatthieu

Just having a play at setting up your recommendation and with your chrome plugin which is really nice. Are you able to use other nodes that work with node-red or do they need to written specifically for your platform?

Once I have things up and running I would be interested to have a look at other databases definitely!

The arduino nano I am using is: http://www.ebay.co.uk/itm/Compatible-Nano-V3-0-ATmega328-Mini-USB-Controller-Board-Cable-For-Arduino-/191257218511?pt=UK_BOI_Electrical_Components_Supplies_ET&hash=item2c87d18dcf

Then I use the ENC28J60 such as: http://www.ebay.co.uk/itm/1PC-New-MiNi-ENC28J60-Ethernet-LAN-Network-Module-For-Arduino-/141086339330?pt=UK_Computing_Other_Computing_Networking&hash=item20d966b102

One of the great things about the above arduino setup is the foot print, not to mention the price. @ntruchsess has a UIPEthernet library which you may have seen https://github.com/ntruchsess/arduino_uip ? This implements ethernet for the ENC28J60 on the arduino.

Many thanks

Andy

[chrismatthieu]

Nodeblu is written specifically to run on our Meshblu (formerly SkyNet.im) IoT platform.

That nano Arduino should be able to support our Microblu sketch.

If the ENC28J60 board is compatible with the Arduino ethernet shield, it should work as well.

Hope this helps... Chris