Integration of images from Assetto Corsa

lupusorina commented 3 days ago

Hi,

First of all, great work!

I was wondering if you are planning to integrate other sensor outputs, such as cameras, or if you know if this task is straightforward/feasible? I am just getting started with Assetto Corsa so any advice is very appreciated on how to add image data.

Thanks!

dasGringuen commented 3 days ago

Hi,

Yes, images are actually in the plan. I have a small prototype, but I haven't had time to fully integrate and test it yet. If you'd like to volunteer, it would be greatly appreciated, and I will help you. Otherwise, you may need to wait a little longer.

The plan is basically to capture the image using OpenCV in the plugin, downsample it (and optionally save the raw image), and send it through the existing UDP interface. The only issue is that we would need to include OpenCV in the Python distribution of the plugin, which is not the same one being used for training the models and receiving the data. If screen capture works in Python 3.3, then the rest should be doable.

An easier option would be to capture the image directly from the client side and not through the plugin. The problem with this approach is that the images will have a slight delay, latency and potentially out of sync with the telemetry (some testing would be requiered to check how worse it gets). Additionally, it won't work in a distributed setup where the client is running on another computer / computers.

Also, if you decide to work on this in your repo, please let me know so that I can add it to this one.

Best, Adrian

lupusorina commented 3 days ago

Hi Adrian,

Thanks for letting me know. If you want to add here your preliminary work, I can definitely take over and try to help the best I can, including testing and further development.

I agree with you, capturing the images directly is not great, as the image data will be out of sync with telemetry.

Best wishes, Sorina

dasGringuen commented 3 days ago

Hi Sorina,

Here are my thoughts and code. Let me know what you think or if you face any issues! I plan to put some effort into this in the near future as well.

Image Grabber: I tested this up to 100Hz (our baselines were running at 25Hz).

import time
import cv2
import pygetwindow as gw
import mss
import numpy
import time
with mss.mss() as sct:
  title = "Assetto Corsa"

  # Find the window by title
  window = gw.getWindowsWithTitle(title)[0]  # Get the first window with the title
  window.activate()

  # Part of the screen to capture
  monitor = {"top": window.top, "left": window.left,
             "width": window.width, "height": window.height}
  while "Screen capturing":
      last_time = time.time()
      # Get raw pixels from the screen, save it to a Numpy array
      img = numpy.array(sct.grab(monitor))

      # Display the picture
      # cv2.imshow("OpenCV/Numpy normal", img)

      # Display the picture in grayscale
      cv2.imshow('OpenCV/Numpy grayscale',
                  cv2.cvtColor(img, cv2.COLOR_BGRA2GRAY))

      print("fps: {}".format(1 / ((time.time() - last_time)+ 1e-8) ))

      # Press "q" to quit
      if cv2.waitKey(25) & 0xFF == ord("q"):
          cv2.destroyAllWindows()
          break

Python Distribution: As far as I understand, if you just install the Python distribution, then AC will use that. Python 3.3.5. Then we just need to pip install from that distribution, and the packages will be visible in the AC plugin.
To send the captured images, you can simply add a new field in the update function. Here
Configuration flags should be placed here (by default, image capturing should be turned off for performance reasons).
The Python debug interface is very convenient to develop the plug-in part. See

Adrian

dasGringuen / assetto_corsa_gym

Integration of images from Assetto Corsa #6