Rebuild networking to use TCP vs UDP

[dmssargent]

The UDP systems requires bonding of ports to a selected number, and if/when the app crashes the port may get leaked. The TCP system requires only one socket to run, (two, if multicast is enabled) one in UDP mode for listening for a multicast. The UDP is also using hardcoded strings for IP addresses in stead of relying on validated user input.

What I have done:

• [x] Re-build the networking components to use the TCP mode, instead of UDP
• [x] Re-factor out the old systems
• [x] Write a new data exchange protocol
• [x] Add options about simulation to the settings menu

What doesn't work:

• [x] Thread safety is apperently not so good (the Android App is throwing a ConcurrentModificationException, which shutdowns the server) EDIT: Not quite: according to Android App thread dumps I am getting caught in a dead spin when I reimplemented the storage queue with a proper object designed for this usage (the old invalidated the iterator in my foreach, when the data got changed from a different thread)
• [x] Console is overflowing - the amount of data being sent and received is causing the console output to noticeably slowdown the application, it is taking its sweet time to catch back up after running for more than a few seconds (right now, one minute of running takes the console writer ten minutes to do) Code to display full data is commented out, otherwise it has an adepquate speed now
• [x] The device list getter/setter doesn't work just yet (probably because of problem one) Is XML Deserializer working?
• [ ] Commenting (a need to add quite a few of those)

What we need to do:

• [x] figure out what is happening when a Concurrent Modification Execption gets fired
• [x] see what happens when above is fixed see what happens when if it works
• [x] Log the packet data and only display the important details if necessary
• [ ] Add a way of turing debug mode on/off*
• [ ] Get and validate user input for an IP address if multicasting is off or fails*
• [ ] Add an option to turn multicasting on or off*
• [x] Add more ways to Stop / restart the Simulator*
• [x] Fix any leaking memory (when the client is left running it caches everything until it gets processed) I have trimmed the memory usage of my side down from 2.5GB to 100MB (client) and 100MB to 16MB (my side) (that's what I get for forgetting to remove a section of the mirror protocol, plus having high data retention policy)

• [ ] Figure out why the networking is acting buggy

  *since you are maintaining the interface, I would recommend you the do the following

I opened this early so I could get feedback from you and to request changes, it is not ready to be considered alpha yet

[dmssargent]

I resolved the issue with the ConcurrentModificationException, so that a dead spin could show up.

[dmssargent]

Dead spin fixed, plus a few minor touchups to identify now your code is throwing: org.xml.sax.SAXParseException: Unexpected token (position:TEXT ?@1:2 in java.io.InputStreamReader@376a700e). I will go see if the data is valid in the parser, or just the code running the XML is expecting a new format of XML.

[HagertyRobotics]

Thanks. I'll start working on merging it into a branch to try it out. I just committed a lot of changes trying to get the GUI working better along with some restructuring.

[dmssargent]

Sorry, about any delays, any ideas about any problems or have suggestions?

[dmssargent]

I am working out why the XML is getting corrupted by the exchange. How would the Device List exchange work best for you?

[HagertyRobotics]

Hi, I'm a little confused. When you were working on your new networking code, I was also restructuring some of the classes and also I changed the package names to ftccommunity. I then merged my new code into master. When you issued your 2nd pull request I took a look but saw that because I make so many changes that I had to do some work to merge it with my latest develop branch. I haven't had a chance to merge it because I was working on getting the new ftc_app release working with the existing code. I guess the XML parser stuff above is just added to your original networking code. My question is, when you are working on your networking code, is it also merged with my latest master branch?

[dmssargent]

Some of the times, I am trying to stay somewhat recent, but I had to take a break (from coding) when you slightly before you started pushing, so I am trying to catch up. My side of Git went slightly nuts on this repo, so thats why there is multiple commits with the same comments.

[dmssargent]

Can I try to repackage some common items in it a module that both use? The current problem is that when the OpMode is switched it doesn't switch and it is stuck in the init phase.

[HagertyRobotics]

I am finally looking at your code. I have to catch up with all the work you did! I did get the new SDK updated and working in a separate ftc_app repository along with changes on the PC side in my develop branch.

[dmssargent]

Okay, I figured out why it gets stuck in init, its completing a send then the client tries to reply back but a) the packet is getted corrupted b) the system for saving the packets is getting blocked or is not functioning, my bet is on the former. Right now commit a9ea7e0 is the best bet for looking at somewhat working code, the latest additions have the GUI closing cleanly, but the networking sections went all crazy, when I tried to optimize memory usage even more. I will try to get all of that protocol stuff functioning before I get the new app components folded in (so I don't have to work out both problems)

[HagertyRobotics]

I tired your code but it seems I am missing the SimulatorData class. Did you check that in or am I doing something wrong?

[dmssargent]

That is a problem. That file is generated from the protobuf/Data.proto. You can use the copy of it in the App, otherwise you can download protoc and rebuild the file, or you can wait a tad a bit for me to force the file to be there.

[HagertyRobotics]

No problem. I'll just get it from the App. Thanks.

[dmssargent]

I am still figuring out why the client is not replying back to the server, but I improved/polished lots of things

[dmssargent]

The bug is apparently within the server receive Non-Heartbeat type. I validated the following components:

Server

• [x] Server - send
• [x] Server -> Network Manger - send
• [-] Server -> Network Manger - receive
• [ ] Server - receive
• [ ] Server - Heartbeat receive (strange things are happening here)
• [x] Encoder
• [x] Decoder

Client

• [x] Client - send
• [x] Client - receive
• [x] Client -> Network Manager - send
• [x] Client -> Network Manager - receive
• [x] Client -> Heartbeat receive
• [x] Client -> Heartbeat send
• [x] Decoder
• [x] Encoder

[HagertyRobotics]

Thanks. I've been working on making sure I understand the communication between an actual Legacy Module and the Phone Still trying to understand switching from read and write modes.

[dmssargent]

I am going to try to re-read the Netty and Protobuf documentation, and see if our software mentor can spot the bug. If all else fails, I am going to need fresh to eyes to spot that bug.

[dmssargent]

I seem to have fixed the problem bug, now we can see it gets stuck on init with waiting on the Simulator Device feedback. So, I probably missed something when I updated the simulator to the latest version, I designed the networking to be backwards-compatible with your code (as in the functions that handle specific things should have the exact same behavior as before). I did add the throwing of an Interrupted Exception to the functions that wait for user input, to allow safe termination. Can you specify an overview of how each component is networked, or evaluate your module code to see if I only partially changed something.

[HagertyRobotics]

Sure, I will look at it right now. After a lot of tracing with an actual device, I now understand more about what is going on in the modules. Thanks.

[dmssargent]

Let me know, if you find the mistake. Do you want a layout of how everything works in the networking?

[HagertyRobotics]

yes, if it is not too much trouble. Thanks.

[dmssargent]

The Network Manager class takes care of keeping track of every send, and the received data. The received data is getting dumped into the Network Manager, then gets sorted by a "process queue" thread that moves everything to the right place. The Server/Client classes are Runnable classes that taking care of the bootstrapping the communications, it also handles the configuration options. The *Handler classes take on creating sends, and receiving things. The Decoder/Encoder classes change the Data class to a ByteBuf (an extension of the ByteBuffer) , and it gets written to the data stream, and vice versa. Sending: NetworkManager.requestSend (builds the Data object) - Handler -> NetworkManager.getNextSend() -> Encoder -> Data gets flushed to the stream

Receiving: New data appears in the stream -> Decoder -> Handler -> NetworkManager.add - processQueue - getNextMessage (callable) (-> getNextData (gets the data encoded (used ASCII to encode)

• -> Next object gets called immediately
• - This may or may not happen immediately

There should be a protocol.md that goes into the communcation

[dmssargent]

Have you gotten anywhere? I have been busy trying to get 4 teams started on the learning the new platform.

[dmssargent]

I am thinking that I already need to do a major rewrite of my own code to increase readability and performance.

[HagertyRobotics]

Same here. Working on getting started. I have more time now. I'll see what I can do. I like what you did but I need to make some changes to merge it with the current code.

[dmssargent]

Okay, though what changes need to be made to push it to the current code, even you get settled down with this keep this pull request open, so we can collaborate on the optimization and rewriting the things that need to be rewritten

[dmssargent]

If you are still looking at this feed, can you rewrite and point me towards the MR USB communication protocols on the PC side? I can re-implement the hooks that you had previously to emulate devices, and rebuild the TCP stack to work correctly.

[HagertyRobotics]

Hi! I just saw your note. Yes, I would like to get back into the simulator. I need a few more weeks to see where we end up this season. Thanks for your interest.

[dmssargent]

Sorry for responding 23 days after your reply. I'm busy with FTC and FRC as well, so my time is limited right now. But I should be free to help out sometime soon.