Make using this as a library easier

[simonvanderveldt]

Hi Tuan,

First of all thanks for the effort you've put into this awesome project! Very happy we have a decent MQTT library for the ESP8266.

I've been looking around for projects that use the ESP8266, MQTT and the DHT22 and all of the projects I've found contain copies of parts of this repo. This already indicates that it's difficult to use this repo in a modular way.

Then I tried to create a project myself with this repo as a git submodule, but that obviously doesn't work well either since the necessary files are not in the root of the repo.

I'd like to propose/discuss splitting this repo up in a library and an example project that uses the library, so the library can be used as a drop-in library for other projects. FYI I'd like to add a section to clib for the esp8266 with drop-in libraries.

The main things that would need to be done for this are:

• Rename this repo to esp_mqtt_example or something similar
• git filter-branch as documented here the mqtt directory into a new esp_mqtt repo
• Move the include/driver uart files to driver/include and then git filter-branch the driver directory into a new esp_uart repo
• Add a submodule to the example project for both the esp_mqtt and esp_uart repos in the correct location

This is the structure I'm using for my example project right now:

.
├── Makefile
├── lib
│   ├── esp_mqtt
│   └── esp_uart
└── src
    ├── include
    │   ├── config.h
    │   ├── user_config.h
    │   └── wifi.h
    ├── config.c
    ├── user_main.c
    └── wifi.c

I have no clue where the drivers, modules and user directories convention in a lot of esp8266 projects comes from, but we could use that as well if that would make adoption easier.

.
├── Makefile
├── drivers
│   └── esp_uart
├── modules
│   └── esp_mqtt
└── user
    ├── include
    │   ├── config.h
    │   ├── user_config.h
    │   └── wifi.h
    ├── config.c
    ├── user_main.c
    └── wifi.c

[Namphibian]

Hi Simon, Tuan

I have modified and worked with the library/project for a while now and have added some new commands. Maybe we should get designs ideas and other improvements and put heads together.

Besides its been a while since I have visited HCMC Tuan and I am going to see my family in Tuy Hoa soon so we could meet in person.

Simon I will look at your proposals but we should include OTA upgrades and smartconfig as well.

Kind Regards Neil Franken On 05 Dec 2015 10:43 PM, "Simon van der Veldt" notifications@github.com wrote:

Hi Tuan,

I've looking around for projects that use the ESP8266, MQTT and the DHT22 and all of the projects I've found contain copies of parts of this repo. This already indicates that it's difficult to use this repo in a modular way.

Then I tried to create a project myself with this repo as a git submodule, but that obviously doesn't work well either since the necessary files are not in the root of the repo.

I'd like to propose/discuss splitting this repo up in a library and an example project that uses the library, so the library can be used as a drop-in library for other projects. FYI I'd like to add a section to clib https://github.com/clibs/clib for the esp8266 with drop-in libraries.

The main things that would need to be done for this are:

• Rename this repo to esp_mqtt_example or something similar
• git filter-branch as documented here https://help.github.com/articles/splitting-a-subfolder-out-into-a-new-repository/ the mqtt directory into a new esp_mqtt repo
• Move the include/driver uart files to driver/include and then git filter-branch the driver directory into a new esp_uart repo
• Add a submodule to the example project for both the esp_mqtt and esp_uart repos in the correct location

This is the structure I'm using for my example project right now:

. ├── Makefile ├── lib │ ├── esp_mqtt │ └── esp_uart └── src ├── include │ ├── config.h │ ├── user_config.h │ └── wifi.h ├── config.c ├── user_main.c └── wifi.c

I have no clue where the drivers, modules and user directories convention in a lot of esp8266 projects comes from, but we could use that as well if that would make adoption easier.

. ├── Makefile ├── drivers │ └── esp_uart ├── modules │ └── esp_mqtt └── user ├── include │ ├── config.h │ ├── user_config.h │ └── wifi.h ├── config.c ├── user_main.c └── wifi.c

— Reply to this email directly or view it on GitHub https://github.com/tuanpmt/esp_mqtt/issues/79.

[tuanpmt]

Hi Neil, Simon, I've made this library can work as libmqtt.a but haven't published yet. I need to fix some critical bug and optimize queue work as outbox before publish it. Ps: Maybe while you come to Vietnam, I am in South Korea, Neil. 1 week from today.

[simonvanderveldt]

@Namphibian @tuanpmt great to hear you guys are trying to make this library more modular as well :)

@Namphibian Do you have other/more improvements than improving the modularity? Also, what do you mean exactly with we should include OTA upgrades and smartconfig? Do you mean those are things that this library should support?

@tuanpmt Np :) I'm not going to be able to visit Vietnam :P Do you have a local branch that includes all those fixes together? I've done a quick git filter-branch as I described above and building from an example project works as it should, haven't had the time to test the firmware files yet, but will do so ASAP to let you know if it works as it should.

[Namphibian]

@simonvanderveldt @tuanpmt this project has some great potential however I think we need to focus on not just a improvement on the current structures/modularity, but also set some goals which will make this a awesome project.

Here is some of my thoughts. Please note these are very rough ideas but hopefully enough to stimulate some discussion.

Ease of use:

At the moment this library is not difficult to use at all. The thing is we live in a world where people press a button and the thing works. We need to make sure this library/framework can be easily deployed and used where needed.

More API styled calls:

Handling the REST API Calls and MQTT in Arduino can be pretty memory intensive. There is a lot of strings that need to be passed. For example when doing a REST or MQTT Calls you have to specify the topic/JSON string. For example /my_id/hello/helloworld in MQTT or worse in JSON

{ "client": {"ID":"1234567890", Event:"ATWORK"} }

The ESP has much more memory than the arduino and thus offloading these type of functions onto the ESP would make more sense. I currently have a board with 6 different sensors reporting from a ATMega328-PU and getting the memory usage down to acceptable levels was not very difficult but the problem was why should a device with less memory be burdened to handle details a device with more memory can handle with ease? With the new ESP32 around the corner there is some scope here apparently this will improve the current ESP greatly with better ADC converters etc. Hopefully two UARTS as well I hope.

Solution: Create a API style interface in addition to the current methods. It occurred to me that most people will be sending sensor or input data via ESP to a server.

Thus for example instead of:

REST: void post(const char* path, const char* data);

MQTT: void publish(const char* topic, char* data);

We could also have:

REST:

`void sensordata(const int* sensortype, const char* data);

MQTT void sensordata(int sensortype, char* data);

The sensortype variable could be some sort of array, enum etc that contains some common sensors for example PIR, Sound Sensor(Digital), Sound Sensor(Analogue), Lidar, Ultrasound etc. This will allow the ESP to handle some of the other heavy lifting. The list of sensors could be in a header file etc. Lets face it things like PIR sensor and MQ135, no matter the manufacturer, will report the data the same way no matter who made it. It is quiet possible to describe in structure what the sensor is and what data it is sending e.g PIR, DIGITAL 0,1 or Sound Analogue 1-1024 for example.

Another goal that one can achieve here is to have the devices(Arduino and ESP) query each other. For example the arduino can ask the ESP what the MAC address is so that it can use that information in say authentication to REST services or use in some topic level on MQTT.

This is just a very rough example and would need some refining of course but I hope you understand that we can possibly leverage more of the ESP and use the arduino as more of a auxiliary processing unit.

SmartConfig: @simonvanderveldt the following idea has working examples on the internet.

There are several examples of smartconfig available for the ESP8266. The biggest problem people will face when deploying these applications on a large scale is catering for configuration of every device(wifi access point details). For example if the Wifi AP is not found the ESP can listen to multicast packets for new configuration. A technician can then send these UDP/multicast packets via an application and the ESP could reconfigure itself. Much more thought is needed here but you get the idea.

OTA upgrades:

@simonvanderveldt the following idea has been implemented on the ESP SDK.

I have read that the ESP can do over the air upgrades thus allowing the system to upgrade itself. It is critical that all devices can be updated with minimum effort and this feature will be killer. It would be great if the ESP could act as an ICSP for the Arduino and you could upload a new sketch to the Arduino using this feature. Complete remote upgrades.

Documentation:

More tutorials, WIki documentation and how to get started articles needs to be published. I firmly believe this framework has great potential however potential not realized is a great waste. While not a very difficult project to start with I do believe there can be a bigger market if there was more clear documentation and maybe some easy to implement circuits etc.

Anyways it is getting late here I just thought I would make some notes on the items I had on my mind,

[Namphibian]

@simonvanderveldt ek sien jy is van die nederland. Het gedink jy is van daardie deel van die wereld.

[vaibhav93]

Any updates on this ? @tuanpmt

[someburner]

In my opinion, OTA should not be integrated into the core part of the library. OTA is already possible with this project as it is. But if/when the app code is separated out from the transport code an OTA example could be added. The problem is that the OTA is implementation-specific, depending on what bootloader you're using, how you are formatting the messages, and the server-side code.

@tuanpmt : Not sure where you're at on the conversion to a library, but I already do this in my projects. In this project I have the core MQTT code in the "app_common" folder. Then In app_setup and app_main I have different MQTT app code.

For SmartConfig, I believe it can be done, but it's outside of the MQTT spec. I think it would be better suited as a separate project since MQTT is meant for TCP.