Discussion about the LoggingManager

[Vincent-SD]

Before we start updating the Logging Manager, let's discuss a few things.

Optimization

Dictionaries, Lists and types

Sometimes, in the code, dictionaries with Integers as key are used. Maybe we could use Lists instead.

Lists in C# keep the order and we don't have to get one specific element. We only need to iterate over the data and save it to a file or create a string, so I think a list is suitable for this purpose.

For instance, this line :

public Dictionary<string, Dictionary<int, object>> log = new Dictionary<string, Dictionary<int, object>>();

could be replaced by this :

public Dictionary<string, List<object>> log = new Dictionary<string, List<object>>();

Data type

The data is currently saved in memory as objects. We could probably save this data directly as strings, it will prevent the program from converting it later when logging to csv or sql.

Logging

Now, we are logging data over time using a dictionary. Each time we log a line, we only add it to the dictionary. There are several potential features that we can discuss here (with the ability to enable/disable features):

• We could log the data directly in the file over time (performance issues ?).
• We could create the dataString (which will be send to sql) over time while the program is running.
• We could create only one string to log into csv and sql. The string could be formatted as following:
  • "value11;value12;value13\nvalue21;value22;value23\n"
  • The php file that saves the data in sql will have to be adapted to this new string format
  • If we log all the data to csv and sql at the end, we could create an asynchronous process to make it faster.

    Refactoring

We could put a lot of code directly in the LogCollection class to organize the manager better. I think that the manager only have to be the interface between the user and the logger.

• We could put all the code about logging into LogCollection
  • Saving of the logs in memory
  • Writing to csv file
  • Creation of sql string
• ConnectToMysql will only have to connect to the database and send the request using the string provided by LogCollection.
• Then, in the manager, we will only have the list of all the LogCollection and functions that the user will have to call in order to log. These functions will call the functions of LogCollection and ConnectToMysql.

[bastianilso]

Thanks @Vincent-SD !

RE: Dictionaries, Lists and types

• This sounds like a good step 1. Let's convert public Dictionary<string, Dictionary<int, object>> to public Dictionary<string, List<string>> then (this is what you propose here, right?).

RE: Logging "We could create only one string to log into csv and sql. The string could be formatted as following: "value11;value12;value13\nvalue21;value22;value23\n""

• I think this sounds quiet effective. As a starting point, maybe we can log to both Dictionary<string, List<string>> and raw string - and the phase out the Dictionary later, if we discover that we have no use case for it anymore.

• When adapting the php file to the new format, we can create a different URL, e.g. logtodbV2021.php - and keep backwards compatibility this way (for a period, both logging formats might be in use while we transition apps).

  "If we log all the data to csv and sql at the end, we could create an asynchronous process to make it faster."

• Are you thinking it will be faster because we can use multi-threading here? Or do you mean, that it will feel faster because we don't block the main loop (e.g. so we can show a progress bar)?

RE: Refactoring "We could put all the code about logging into LogCollection "

• Interesting suggestion. Could you list what private and public functions then would remain in the Manager interface then?
• If the LogCollection includes saving to memory, writing and uploading, then we may need a different name. "Collection" does not imply that it is a variable that can do anything to the logs - except collecting them. This is why right now, the collection is just a data structure and the manager, is responsible for functional aspects of it.

[Vincent-SD]

RE: Dictionaries, Lists and types "This sounds like a good step 1. Let's convert public Dictionary<string, Dictionary<int, object>> to public Dictionary<string, List<string>> then (this is what you propose here, right?)."

• Yes, this is exactly what I was meaning, so let's do this.

RE: Logging

"As a starting point, maybe we can log to both Dictionary<string, List<string>> and raw string - and the phase out the Dictionary later, if we discover that we have no use case for it anymore."

• I agree, so let's log in the dictionary as before and also log to a formatted string as explained in the issue ? ("value11;value12;value13\nvalue21;value22;value23\n")

"Are you thinking it will be faster because we can use multi-threading here? Or do you mean, that it will feel faster because we don't block the main loop (e.g. so we can show a progress bar)?"

• Both in fact. If we use multi threading to create the "big string" from the dictionary, we could for instance :
  • Divide the dictionary into 4 dictionaries of the same size
  • Concatenate the string for each dictionary on 4 separate threads, so that we have 4 strings at the end
  • Concatenate together the 4 strings to get the big string

I don't now if it is really possible, but it could be a good approach to optimize the system. We could also increase the number of threads depending on the size of the dictionary.

RE: Refactoring

Here are the functions that will be in the LoggingManager. Actually, the functions will be almost the same, we will just change what happens in these functions. The idea is just to organize the code to make it more readable.

We could rename the LogCollection class into LogWritter or something like this.

[bastianilso]

RE: Logging I agree, so let's log in the dictionary as before and also log to a formatted string as explained in the issue ? ("value11;value12;value13\nvalue21;value22;value23\n") Sounds good.

Both in fact. If we use multi threading to create the "big string" from the dictionary, we could.. Cool. Let's keep this as a backup solution, in case creating the "big string" on the fly affects performance too much.

RE: Refactoring Looks good. But we could still let users send data to Logging Manager in any format (object) and we could parse it to a string when storing it in LogCollection. Just so users can continue to pass float, int etc to the logging manager. This way we can always ensure that numbers are properly formatted with decimals, e.g. always use 4.123 and never to 4,123 (this commonly happened in danish computers) etc.

In terms of delegating responsibility, here is my proposal:

let me know if you have questions about it, or counter proposal :-)

[bastianilso]

Test in the following applications:

• https://github.com/med-material/Whack_A_Mole_VR
• https://github.com/med-material/ArduinoLogger
• https://github.com/med-material/HandStrengthener