IP & file naming clarification

[ltorrespr]

Your new code is working just fine here! GREAT job Jeremy. I was able to get data back once I changed the ip on the sniffer.py & telemetryCapture.py.

Quick questions:

1. Within color_setter.py, there are many lines with "client = OpenRGBClient('127.0.0.1', 6742, 'My client!')", does this particular ip stay as-is? Just curious what it represents. I left it w/o change.
2. There is a file named "telemtryFiller.py", is this a misnamed file? Should it actually be "telemetryFiller.py"?

BTW, Keyboard Interrupt for Mac is also "control + c".

Thank you!

Luis

[jeremySchriver]

You should only have to run the telemtryCapture script to do your pulls. I will be looking to make the data pull more robust, by adding the additional missing variables but my first focus was on the dyno type data and the normalization it needed.

1. Within color_setter.py, there are many lines with "client = OpenRGBClient('127.0.0.1', 6742, 'My client!')", does this particular ip stay as-is? Just curious what it represents. I left it w/o change.

• This whole module is more than likely not ready for wider consumption. It leverages the OpenRBG application which has a built in SDK server that can be run. That IP information is OpenRBG's default information and should not need to be changed so long as the server within the app is running. There is some base level hardware capturing I need to perform before trying to allow for configuring the lighting conditions, and the rules in place now are based on my LianLi case fans rather than any real dashboard add on.

2. There is a file named "telemtryFiller.py", is this a misnamed file? Should it actually be "telemetryFiller.py"?

• This file was originally intended to perform the normalization of the data but will most likely be deleted instead of renamed. My original calculation for finding optimal shifting points did not work with blank data along the curves so this was meant to help interpolate and fill in those gaps. I have been working on updates to this process in a jupyter notebook on the side and just pulled updates into the csvFIller.py file on the dev branch. I will be looking to merge these updates up into the master this week as I have time to start building the UI for displaying the plots.

[ltorrespr]

Thank you very much for the detailed explanations Jeremy. This all makes sense to me now.

Regards, Luis

[jeremySchriver]

@ltorrespr Just wanted to let you know there has been a major release update pushed to the main branch of this repo. I suggest working from the new branch moving forward.

The process for capturing telem data has not changed:

• Launch the game
• Run the telemetryCapture.py script via terminal
• Perform a keyboard interrupt when done capturing data

You can now process the data that was captured by running the csvFiller.py script via terminal

• Before running this script please ensure you have updated the following variables highlighted in yellow
  • 
  • Next update will fix the captureFile value to line back up with generated file name (it is this way because of testing I was doing)
  • Next update will also look to find the numGears and carOrdinalID value from the telem file and ask for user input if they are unable to be determined
• This update did introduce an input block to trigger the methods. For the time being it is best to delete all the data before running any of the methods. This will not affect your raw data capture but only files generated into their respective carID folder.

Known issues still in the works:

• Quadratic best fit lines for individual gear data appear to be falling very short of the maximum values in the raw data
  • This is causing issues with harnessing the Torque vs Speed chart to find optimal shift points
• Lighting condition scripts have not been updated to leverage the new data points in the new database file
  • More to come on this once curving issue has been solved

Please let me know if you have any issues with the new build!

[ltorrespr]

Thank you for the update! Will test it later today and provide feedback.

[ltorrespr]

I ran into this ERROR during telemetry data capture portion of program....

Traceback (most recent call last): File "/Documents/Forza-Telemetry/JeremyS/Forza_telemetry_project-main_jeremyS/telemetryCapture.py", line 3, in import data_packet ModuleNotFoundError: No module named 'data_packet'

Maybe the telemetryCapture.py needs to be in same directory as data_packet.py?? I moved the data_packet file into same Dir and did work, yet, not sure about the rest of the structure impact by doing so. I noticed you had ran and it had the logTelemetry2 file within the TelemDataFile directory, yet, since I moved the data_packet is did not write to that location.

[jeremySchriver]

Sorry I overlooked the fact that the locations for a lot of those files had changed as well. Updates made to csvFiller and telemetryCapture which should fix the dataPacket class issue and the location of the logTelemetry file. Please grab a new copy of the main branch and let me know if you encounter new issues.

[ltorrespr]

No worries Jeremy! I will download the latest and test later tonight, or, at the latest tomorrow morning. Thank you very much for all your efforts in this project. Will provide fdbk once I test.

[ltorrespr]

Hi Jeremy, could not resist the temptation of testing this tonight. The telemetry data capture works perfectly. I was surprised the file output *.csv was placed one directory up (meaning same location as the Main Directory) from the Main branch.

I tried running the csvFiller, yet, received several errors (could be one major error). The directory nor file (*.csv) were found. I tried renaming the file, then moved it to the main directory, as well as the TelemDataFiles directory, yet, could not get it to work. BTW, when I did so, I did edit the csvFiller with the correct filename I used (logTelemetry_Porsche-test.csv), and, also change the script to have the number of gears (6) and car ordinal #263.

I am probably missing somewhere else to edit to ensure I am directing the file/directory to proper location.

Let me know if you need any add'l details. Thank you!

[jeremySchriver]

If you switched the name of the telemetry file you will probably run into a large number of issues with the csvFiller as it is looking for direct locations for the files.

• telemetryCapture.py
  • should be creating the file "logTelemetry.csv" within the TelemDataFiles directory
    • Placement in the proper directory relies on the telemtryCapture.py file being in the root directory so that "os.getcwd()" can generate the file path directly prior to the folder/file defining
• csvFiller.py
  • Should be located in the TelemDataFiles directory so that the "os.path.join" command can work without needing the entire path name
  • If your logTelemetry file is in a different location, or is named different, this is the place where you would need to update it
  • 

I would suggest trying to get an entirely new copy of the repo and attempting to run things without adjusting locations/file names. The thought is that only one log file exists at any given time and that is the file that telemetryCapture.py is writing to. This file is then being ingested when csvFiller.py is run and new files (CarID_Gear_x_data.csv) are being generated under their own directory (CardID) under the TelemDataFiles directory. The methods within csvFiller.py rely on the additional csv files to be in the proper location or they won't run.

This does bring in a thought that I should ask if you want to store the previous log file before starting to write to it again. Additionally I need to add a method to delete the contents of the log file before starting a new capture if the user confirms they want this.

Just as a comparison this is what my directories look like after capture and filling. (Disregard the logTelemetry_copy and logTelemetry2 file)

[ltorrespr]

I will certainly do that (I kept the original zip file). However, I did notice that in your scripts, you use the following code...

Defines file path and name

file_name = "DBModule\carDb.csv"

And, this actually names the file to DBModule\carDb.csv, and does not actually place it within the Directory DBModule with the name "carDB.csv". I think the same is happening to the telemetryCapture file. WIll validate once I get use the code.

Thanks!

[ltorrespr]

As far as keeping a copy of previous log files = Yes. I actually rename it and place it in a different directory outside the Main code since I can refer to, and/or use it later for other stuff.

[jeremySchriver]

telemetryCapture.py actually has its own method for creating the CSV file on the fly if it isn't found, createHeaderCSV(fileName).

This method requires the input variable fileName which is set on line 37 of the file.

The command os.getcwd() should return the current working directory of the active file. In my case when printing this value from telemetryCapture.py I'm given E:\Code Projects\Forza_telemetry_project-develop\Forza_telemetry_project-develop

Adding "\TelemDataFiles\logTelemetry.csv" to the string that is fileName should produce E:\Code Projects\Forza_telemetry_project-develop\Forza_telemetry_project-develop\TelemDataFiles\logTelemetry.csv

When telemetryCapture.py is run in terminal it should call for the createHeaderCSV method on line 59 where it will only create the file if it is missing. Though there is an argument that I should add another os.path.exists before starting the main loop, just to confirm the file is present and to error out if not.

So there shouldn't be any issues with the pathing unless the directories don't already exist or if the active file is in a different directory. The method writing data to the csv file, within the main loop, is actually an append method and will only add to the file not delete/overwrite.

As far as the carDB file and methods there was just a major change to the module and new files should be used. You should see the placement of 2 new files, carDBmgmt.py and carDb2.csv, under the DBModule directory. db2 was necessary due to new index names that were added for additional shift point data. carDBmgmt.py is the new way to interact with this database file.

carDBmgmt uses a very similar way of ensuring the csv file is created, with headers, before allowing any additional methods to be run. The lines below actually control the pathing for the files being leverage

Defines file path and name

file_name = os.getcwd() + "\DBModule\carDb2.csv" telemPath = os.getcwd() + "\TelemDataFiles\logTelemetry2.csv"

The database file is then created using the createHeaderCSV(file_name) method near the top of the script. This again is a w method which only creates the file if missing and ignores any other condition.

Within carDBmgmt you will find the following methods

• Method for creating a new entry for a car that hasn't existed before by manual input

  • addNewCarByManualInputWithoutGearInfoInput(CarOrdinalID, file_name)

• Method for creating a new entry for a car that hasn't existed before by pulling data from logTelemtry

  • addNewCarByReadingTelemtry(file_name, telemPath)

• Method for updating all values on an existing car

  • def updateCarAllValuesManually(CarOrdinalID, file_name)

• Method for updating all values on an existing car by being passed the values in the method head

  • def updateCarAllValuesViaMethodInput(file_name,CarOrdinalID,CarFriendlyName,CarMake,CarModel,CarManYear,CarClass,CarPerformanceIndex,DriveTrainType,NumCylinders,EngineMaxRPM,EngineIdleRPM,Gears,FinalDriveRatio,Ratio1,Ratio2,Ratio3,Ratio4,Ratio5,Ratio6,Ratio7,Ratio8,Ratio9,Ratio10,mphShiftPoint1,torqueShiftPoint1,rpmShiftPoint1,mphShiftPoint2,torqueShiftPoint2,rpmShiftPoint2,mphShiftPoint3, torqueShiftPoint3,rpmShiftPoint3,mphShiftPoint4,torqueShiftPoint4,rpmShiftPoint4,mphShiftPoint5,torqueShiftPoint5,rpmShiftPoint5,mphShiftPoint6,torqueShiftPoint6,rpmShiftPoint6, mphShiftPoint7,torqueShiftPoint7,rpmShiftPoint7,mphShiftPoint8,torqueShiftPoint8,rpmShiftPoint8,mphShiftPoint9,torqueShiftPoint9,rpmShiftPoint9,mphShiftPoint10,torqueShiftPoint10,rpmShiftPoint10)

• Method for updating a specific value on an existing car via manual input

  • def updateCarValueViaManualInput(file_name,CarOrdinalID,field)

• Method for updating a specific value on an existing car via method pass

  • def updateCarValueViaMethodInput(file_name,CarOrdinalID,field,value)

• Method for manually updating gear ratios on an existing car

  • def updateCarGearRatiosViaManualInput(file_name,CarOrdinalID)

The changes here should give you much more granular control over editing the data within the database and also adds a way to extract data from the logTelemetry file so you don't have to add everything by hand.

I will look to add some user inputs and file archiving abilities into the telemetryCapture script so that I can meet your other request.

[ltorrespr]

Jeremy, thank you very much for such detailed explanation. I am certainly using this as a major opportunity to learn not what, but, the how & why. And, your explanations have certainly helped a ton.

I will start from "scratch" by running each of the scripts (telemetryCapture.py, then carDBAdd.py, and finally csvFiller.py). Will screen capture my current Dir and files that get created for reference.

BTW, are you using PC or Mac? And, should there be any differences on the file management targets?

Thank you for all your insights!

[ltorrespr]

Here are the pics of the file created. First, prior to running "python telemetryCapture.py", then, after it runs.

Look at the location and file name created!!!

[jeremySchriver]

hmm let me dig around for a better method to create that file. Can you share the whole name it made since that was chopped in the picture a bit?

[ltorrespr]

Forza_telemetry_project-main\TelemDataFiles\logTelemetry.csv

[ltorrespr]

That is the filename itself, it was created at the same directory level as the overall code (Directory = Forza_telemetry_project-main) location. Very weird!

They are both under the Directory = JeremyS which I created to host your work within my Documents area.

[jeremySchriver]

Looks like I was able to isolate the issue and will post a fix once I replace all the places that could be causing this.

[ltorrespr]

Great! I am glad you figured it out. Thanks for reviewing my "debugging".

[jeremySchriver]

@ltorrespr New merge to main completed. Please rebase and test again. I think I got it all fixed up for you now and graphing should work assuming you install any missing libraries from the requirments.txt file.

[ltorrespr]

Thank you very much! Will grab code right now. BTW, will it give me a Warning/Error for any missing Libraries?

Seems the overall process changed, so, will do more work/analysis tomorrow. Before when I ran the telemetryCapture.py, it would automatically capture the data from my XBox, now it goes through various questions prior to the capture/analysis. I tested with Y (and yes/Yes/YES), then, N (no/No/NO) and at the end it gave me various errors. I think I first have to manually enter the car data and probably deleting the existing files you have already captured days back prior to then running the new car.

The only car saved I saw was within the Directory 3655, yet, it mentioned 2352 for some reason, which is not the car I have either.

Info received after the question, "Do you want to rename the file?".... Note: New file name would already be named 2352_logTelemetry.csv and would be placed in its respective TelemDataFiles directory Traceback (most recent call last): File "/Documents/Forza-Telemetry/JeremyS/Forza_telemetry_project-main/telemetryCapture.py", line 90, in df.to_csv(path, index=False) File "/opt/anaconda3/lib/python3.9/site-packages/pandas/core/generic.py", line 3466, in to_csv return DataFrameRenderer(formatter).to_csv( File "/opt/anaconda3/lib/python3.9/site-packages/pandas/io/formats/format.py", line 1105, in to_csv csv_formatter.save() File "/opt/anaconda3/lib/python3.9/site-packages/pandas/io/formats/csvs.py", line 237, in save with get_handle( File "/opt/anaconda3/lib/python3.9/site-packages/pandas/io/common.py", line 702, in get_handle handle = open( FileNotFoundError: [Errno 2] No such file or directory: '/Documents/Forza-Telemetry/JeremyS/Forza_telemetry_project-main/TelemDataFiles/2352/2352_logTelemetry.csv' (base)

[jeremySchriver]

I think I have found the issue and have been working through some additional updates. More to come tonight when I have had some more time to do testing. Do me a favor and open a new bug for this so I can document the fix a bit for you.

[ltorrespr]

Cool! I started tracing the ERROR's thru the various files yet landed in nowhere land.

Will open New Bug

Thanks! Will check tonight.