Companion app for mic input

[achimmihca]

Actual behaviour

One of the obstacles for players is the need for a mic, especially when using different devices or mobile.

For example, I bought USB mics just for the game.

However, these mics cannot be used to play the game on a smartphone / tablet.

Expected behaviour

It would be very nice if a smartphone app could be used to provide mic input for UltraStar Play. Everyone has a smartphone so people can start singing duets without the need for additional hardware.

Partially, there are existing solutions such as WO Mic. However, these do not work across all platforms and sometimes only for a limited number of devices.

Mic access Accessing the mic is already implemented in UltraStar Play. This part can be copied for the companion app.

Network discovery What is missing is some sort of automatic network discovery.

I think it is important that no setup is needed. It should "just work" when the devices are in the same local network. If people have to enter an IP-address, then it is too complicated.

A port scan could suffice for this.

Network lib First, a network lib is required. This is also relevant for #68.

This blog post names possible libs. Out of these, free open source solutions seem to be MLAPI and Mirror.

Note that MLAPI will be the starting point for Unity's upcoming networking solution.

Own project

The companion app is a separate (GitHub) project.

The companion app is used to provide input for the main game and maybe control it (see also #176 for ideas).

The companion app has nothing to do with the multiplayer feature of #68 (except that the same networking lib should be used).

Example (PlantUML):

@startuml
node "LAN1" {
  [Tablet]
  [Smartphone1]
  [Smartphone2]

  Tablet <--> Smartphone1
  Tablet <--> Smartphone2
}

node "LAN2" {
  [PC]
  [Smartphone3]
  [USBMic]

  PC <--> Smartphone3
  PC <--> USBMic
}

Tablet <--> PC
@enduml

• PC and Tablet are running the main game in different networks.
  • They receive mic input from smartphones, which run the companion app.
  • PC and Tablet communicate and exchange data for multiplayer. However, this is independent of the companion app.

Names Any suggestions for names? I would just name it "UltraStar Play Companion"

[basisbit]

"Microphone for UltraStar Play Desktop" or what you suggest. I don't know if having "UltraStar Play Companion" might confuse people who are searching for UltraStar Play? How do other games handle this?

[achimmihca]

The companion app could be extened with other features (e.g. #176). Thus, focusing too much on "microphone for..." might not be optimal. It is not necessary that the main game is running on Desktop. Thus, "... for UltraStarPlay Desktop" is not optimal.

[achimmihca]

Alternatives for "companion" might be partner, fellow, sidekick, extra, associate, augmentation, etc.

But I think "Companion" is the best because a companion on its own makes little sense, it needs two. Also I think it sounds good, simple (not techie), and not too generic. However, I am not a native English speaker so my understanding might be off.

Anyway, at least from a coding perspective I think we can name the stuff "Companion". On the app store the name could also be "UltraStar Play Mic and Companion" or similar to emphasize the awesomeness of the use-your-smartphone-as-mic feature :)

[basisbit]

okay, Companion won. Your arguments are too good 😄

[achimmihca]

I just created a second repo with the companion-app-prototype I have so far. I found that C# network API (UdpClient, TcpClient) is perfectly fine for this.

Before, I had a look at the networking libs for Unity. They seem very nice to implement multiplayer feature (#68). However, for the companion app I want to avoid this "magic" introduced by "hidden" protocols of specialized libs and frameworks. A simple protocol based on standard network sockets allows other developers to create native Android / iOS apps if they desire.

So, current approach:

• Udp broadcast is used to find the main game in the local network.

  • Client sends ConnectRequest. This ConnectRequest has a flag to ask for a dedicated port for the mic-data.
  • I think a dedicated port for the real-time critical mic-data could be beneficial

• The main game listens for the UDP broadcast and answers. This way main game and companion app exchange their IP-address

  • If the flag is set, the main game opens a socket on a new port and includes this port information in the ConnectResponse

• Now, the main game knows the client and vice versa.

• In my main game branch, I have it ready that the companion app and its mic-data can be used as normal recording-device.

• Also, I added a HttpServer (C# standard HttpListener) to implement simple REST API for the main game. This can be extended easily.

  • Again, using standard approach for this instead of Unity specific networking lib. REST is very common, testable and supported by many tools. This makes it easier for other developers to join and extend or create software around it (such as native companion app clients).
  • At the moment, only the loaded song list can be queried. I want to show the song list in the companion app.

Only problem I have is the mic-data when send over WLAN. It is pretty "jumpy". Maybe I can fix this easily by using Tcp instead of Udp. Anyway, I will refine what I have and then create a new release of both, main game and companion app.