Augmented reality

[8kdesign]

// Still in development, do not merge

Description

Added communication and AR modules.

The communication module allows for real-time interaction between different users, creating new possibilities for what users can do on Source Academy. Meanwhile, the AR module allows users to run an instance of AR on their device (currently Android only, can run in simulator on desktop browser) and add objects to it.

Dependencies Added:

• MQTT : For communicating between devices.
• React Three Fibre: For AR capabilities.
• React Spring: For animating movements.
• Uniqid: For generating unique identifiers.

Type of change

• [x] New feature (non-breaking change which adds functionality)
• [x] This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration.

• Added testing for communications module logic
• Created sample scenarios for usage of modules in Source Academy

Checklist:

• [ ] My code follows the style guidelines of this project
• [ ] I have performed a self-review of my own code
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] My changes generate no new warnings
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] New and existing unit tests pass locally with my changes
• [x] Any dependent changes have been merged and published in downstream modules

[RichDom2185]

// Still in development, do not merge

Ok, I will mark it as draft in that case.

[martin-henz]

@8kdesign: you use a public mqtt server: broker.hivemq.com, see https://github.com/source-academy/modules/pull/278/files#diff-4dbd666d1953774f4cba4f92d36d320ce3c967d68ff4b21b8a0e7a11117d1bd1R18 What are the security implications? Are the messages encrypted?

@RichDom2185 @chownces @angelsl should we use our own MQTT server for this? Are there any security concerns? If we use our MQTT server in a Source Academy communication module, would that expose it to the public, or is it already protected? It only allows connections via sourceacademy.nus.edu.sg, right?

[angelsl]

should we use our own MQTT server for this

Definitely.

For what it's worth the EV3 setup uses the AWS IoT-provided MQTT broker with authentication of the EV3 devices done using TLS client certificates provisioned when a user adds a device in Source Academy. The frontend itself authenticates using a temporary AWS token generated on-the-fly by the backend.

So doing something like that is an option and might let you reuse some of the already-existing infrastructure for that although I imagine some refactoring will be needed to expose the functionality to modules.

I think @RichDom2185 will be able to talk about that setup in more detail if needed.

Alternatively you could spin up a separate MQTT broker on an EC2 instance. But you'll have to handle authentication somehow. For example (from a cursory search) it looks like Mosquitto (a popular MQTT broker and the one you would probably use) has a nice authentication plugin https://github.com/iegomez/mosquitto-go-auth that accepts JWTs so maybe you could just configure that to use the JWT token we already use for authentication to the main backend.

[martin-henz]

should we use our own MQTT server for this

Definitely.

For what it's worth the EV3 setup uses the AWS IoT-provided MQTT broker with authentication of the EV3 devices done using TLS client certificates provisioned when a user adds a device in Source Academy. The frontend itself authenticates using a temporary AWS token generated on-the-fly by the backend.

So doing something like that is an option and might let you reuse some of the already-existing infrastructure for that although I imagine some refactoring will be needed to expose the functionality to modules.

I think @RichDom2185 will be able to talk about that setup in more detail if needed.

Alternatively you could spin up a separate MQTT broker on an EC2 instance. But you'll have to handle authentication somehow. For example (from a cursory search) it looks like Mosquitto (a popular MQTT broker and the one you would probably use) has a nice authentication plugin https://github.com/iegomez/mosquitto-go-auth that accepts JWTs so maybe you could just configure that to use the JWT token we already use for authentication to the main backend.

OK, so we are considering adding a module to Source Academy that requires an MQTT server. How about making that dependency explicit? We could have an init function in the module that explicitly specifies the MQTT server. Anyone who wants to start a user community that communicates with the module will need to tell the members of the community what server to use and how to authenticate.

Not sure if this goes in the right direction. Any comments welcome.