Decide on the log format

[TheCodeSummoner]

The proposed logging format is as follows:

1. Severity - Debug, Info, Warning and Error levels. Critical level will not be used.
2. Time stamping - yyyy-MM-dd HH:mm:ss,SSS, e.g. 2017-06-26 02:31:29,573.
3. Message - no format specified.

Additionally, for more verbose logging:

1. Calling module, function and specific line of code
2. Hardware diagnostics where needed, such as processor usage

@thisiswhereitype suggested Mqtt and asked to share further thoughts. This is where you type.

[thisiswhereitype]

MQTT I think MQTT can be a good application layer to build messaging around. Payloads of bytes are sent on topics via TCP. Topics are expected to be used periodically with a single connection to the broker remaining open.

In our case topics can be used to organize messaging to various consumers and processors of information. We may update power measurements periodically or only use some sensors briefly.

Coordination is handled by a broker server forwarding topic publications to subscribers, in this case probably the surface server is also the broker.

The power system for example has measurements of voltage, power and temperature we may want to monitor. For each of these we can organize them in a topic:

'monitor/bus-12v/' To get all measurements for a bus we can use: 'monitor/bus-12v/+' i.e. subscribe to all children of bus-12v For all things monitored: 'monitor/#' to get all descendants of 'monitor'.

A use case to log all publications: Subscribe too '#' and write too a log file.

Schemas and Payloads The actual payload to send on a topic can be json or apache thrift. The schema should be channel/topic appropriate:

Using MQTT here can help you organize schemas for various groups of sub topics. /alarm/... sub topics may have be a user facing warning with an appropriate fields in JSON /monitor/... may be for raw sensor data again with appropriate fields.

I don't have much to say about logging from software but I can see this being useful for some hardware sensor logging.

[TheCodeSummoner]

Hmmm, do you think we could use software logging for the program to access offline (because we will have access to the computer onboard and I don't want to use the bandwidth), and possibly use the MQTT stack for hardware debugging "offline" too? But then surely we can use the system logs (?)

[TheCodeSummoner]

It would make sense to use the hardware debug over serial when debugging actually.

[thisiswhereitype]

Hmmm, do you think we could use software logging for the program to access offline (because we will have access to the computer onboard and I don't want to use the bandwidth), and possibly use the MQTT stack for hardware debugging "offline" too? But then surely we can use the system logs (?)

All MQTT traffic goes through the broker, so with a broker on the ROV it will not send anything to the surface, except to any subscribers which can be restricted to low bandwidth topics. A permanent storage subscriber on the ROV itself ought to be fine, the broker may have options for logging traffic+payloads itself.

It would make sense to use the hardware debug over serial when debugging actually.

MQTT should be ok for Arduino with Ethernet shield, but serial is an option we have used ok in the past, the Pi can forward serial messages onto MQTT if needed.

[TheCodeSummoner]

@antoinepetty would you mind having a look at it? It might be a bit of an overkill I think. It is described as a great tool for IoT which I believe makes sense, but we don't have an excessive amount of communication. I do believe however that it would be good to build our messaging/logging upon an existing protocol.

[antoinepetty]

I have just had a brief look but it seems like a viable option. We could use it for the normal data transfer we had last year, but also for logging which could easily be switched on from the surface (but only using bandwidth when required). I think it's worth trying to implement MQTT, can always switch to a custom protocol if it ends up not fitting our needs. (+good experience for members going on to IoT)

[TheCodeSummoner]

So I spoke with some IBMers and they generally say it's a viable option for logging because of its pub/sub schema. However, it would make no sense for us to use it as a standard communication protocol.

[thisiswhereitype]

So if I understood:

• Logging to surface of sensed values i.e. power.
• Logging to surface of program execution.
• Not for debugging/general communications i.e. replacing ssh or file system activities.

@antoinepetty @TheCodeSummoner

[TheCodeSummoner]

@thisiswhereitype Exactly that. We don't want to use it as a way to transfer data normally, just the logging bits.

[TheCodeSummoner]

Moving to "High priority" to pick up later when low-level logging comes into place and we've tried out MQTT.

[TheCodeSummoner]

Moving to blocked because this issue waited too long to be marked as 'high priority'