Unable to Identify and Create Segments with non-standard resolutions and Modified HUD positions

[MarcoColomb0]

Steps to Reproduce:

Use a resolution different from 1920x1080 (16:9), for example, I'm using 1280x960 (4:3).

Expected Behavior:

Segments should still be created on different resolutions.

Observed Behavior:

The segments work fine on 1920x1080 resolution without any issues. However, when using a different resolution like 1280x960, the generated images in the "/sessions" folder are not positioned correctly, and the model is unable to identify game actions.

Environment:

Operating System: Windows 10 - 22H2 Python Version: 3.8

Attempts to Resolve:

I attempted to modify values in "/api/models/highlight.py" to correctly center the images, but the model still couldn't identify the segments. This could be due to the different aspect ratio between the resolutions.

Thank you.

Snapshot from one of my clips (HUD position showcase):

[Flowtter]

Hello Marco,

Crispy offers a fantastic feature for resizing videos that are not in 1080p. This feature is particularly valuable when dealing with 720p gameplay, for instance.

In your specific case, using this function may shift certain elements, including the overlay position.

Just to clarify, in CSGO2, is it not possible to adjust the overlay's position, and its placement appears higher due to your use of a stretched resolution, correct?

I've also noticed a couple of issues with the image you provided:

• It seems that the circular shape is slightly distorted here: .
• The circle appears slightly above the box I'm cropping:

The first issue could potentially disrupt the network. In such a scenario, you can find a tutorial here, but I can provide further guidance if necessary.

The second issue might be easily resolved by adjusting the position of the cropped box, which can be done here. It should align with the typical cropping section. You can experiment with a standard 1080p gameplay to find the correct position; a simple translation along the height axis should suffice.

I genuinely believe that modifying the position of the cropped box and reducing the confidence level should resolve the issue.

If these adjustments do not suffice, you may need to develop a new network.

Please let me know your thoughts, and feel free to respond to this thread.

Thank you for your support!

[MarcoColomb0]

Hello Brice,

CS2 allows you to edit the HUD position under Settings > Video > HUD Edge Positions, and it is not caused by the different resolution.

That's why there's more spacing on the HUD borders in my screenshot. It's a personal preference, and I would prefer not to change my in-game config as I've been using the same settings since CS:GO.

Another thing I would like to understand is that I'm using 1280x960 (stretched), but my clips on ShadowPlay are getting encoded in 1920x1080. I'm doing this so the clips are already stretched, and I don't have to manually stretch them in an editing program. Could this be a problem for the network to identify segments? I guess modifying parameters for this resizing feature is not necessary, as the clips are already 1920x1080.

Please let me know if I'm wrong, and thank you for the help :)

[Flowtter]

Hi Marco,

I was unaware of the possibility to edit the HUD. That is totally my bad. I'll deploy a fix this month.

In the meantime this should totally work, being in 4:3 or 16:9 will not make a difference

by adjusting the position of the cropped box, which can be done here. It should align with the typical cropping section. You can experiment with a standard 1080p gameplay to find the correct position; a simple translation along the height axis should suffice.

But because you're on a stretch resolution, don't forget to lower the confidence.

Overall, this should fix your issue:

I genuinely believe that modifying the position of the cropped box and reducing the confidence level should resolve the issue.

As for your question with ShadowPlay, I don't think it will create an issue.

[Flowtter]

Hi Marco,

I'm having some problems with all the customizations available on CS. I wasn't expecting you can change size, color and position.

That seems to be a lot for the current crispy.

I could fix the position, but that wouldn't satisfy the color and size.

I'll try to find a solution when I have the time.

[MarcoColomb0]

Hi Brice,

It's been a while, and I stopped playing CS2. I went back to Valorant (unfortunately 😂), and obviously, I wanted to try this project there too.

As I predicted, stretched resolution seems to be a problem here too, due to the kill indicator being completely off compared to 1920x1080. I tried tweaking the settings here, but I guess I'm doing something wrong. Basically, 90% of the kills aren't undetected, and the vast majority of the detected ones are just "false positives." I don't know if you still work on this, but I'm attaching a snapshot from one of the clips with the kill indicator.

In game resolution is: 1280x960

Maybe we can collaborate in some way to find a fix for it.

Thank you, as always, and I hope you had great holidays!

[Flowtter]

Hello Marco, I hope you're doing well!

I couldn't find a perfect solution for CS2. A few months ago, I was working on a LoL version, and I believe it could assist me in understanding the problem with CS2 better. However, currently, I'm not actively working on it.

Regarding your Valorant issue, the main problem with stretch resolution is that Crispy is converting it to 1080p.

This conversion is necessary to ensure the image is the right size for the AI. The crucial aspect is maintaining the same height and width for the kill-circle as in the footage it was trained on.

Now, I have an idea that might work.

If we calculate the height ratio (3:4) and apply it to the width (1920 * 0.75 = 1440), we get a resolution that the AI might be able to work with. This way, we'll have a kill circle of the perfect size, though not at the right place.

Your image would look like this: (resolution 1440x1080)

To achieve this, simply apply a Pillow filter here that resizes the image as done here.

Now, if you change the first value here from 899 to 660 (you can test between 658 to 662 to find the best fit), which should be the first coordinate of the box, you might have a solution.

I haven't tested it, so please let me know if it works for you!

If it does, please inform me, and we can consider making a pull request, either by you or me. This would be a great first contribution! ;)

[MarcoColomb0]

I tried doing what you suggested, but I think I made something wrong (I'm not a Python expert or anything 🤣). The images with the mask look like this. I honestly don't know.

[Flowtter]

Hey Marco, I made it work ! The fix I gave you earlier could not work, that was my bad. Here is the complete diff:

index c9b0a34..2e976b4 100644
--- a/crispy-api/api/models/highlight.py
+++ b/crispy-api/api/models/highlight.py
@@ -159,7 +159,7 @@ class Highlight(Thingy):
             return final

         return await self.extract_images(
-            post_process, (899, 801, 122, 62), framerate=framerate
+            post_process, (660, 801, 122, 62), framerate=framerate
         )

     async def extract_csgo2_images(self, framerate: int = 4) -> bool:
diff --git a/crispy-api/api/tools/setup.py b/crispy-api/api/tools/setup.py
index b45ed94..a2df5fc 100644
--- a/crispy-api/api/tools/setup.py
+++ b/crispy-api/api/tools/setup.py
@@ -93,8 +93,8 @@ async def handle_highlights(
             ).overwrite_output().run()
             shutil.move(tmp_path, highlight.path)

-        if Image.open(highlight.thumbnail_path_full_size).size != (1920, 1080):
-            await highlight.scale_video()
+        if Image.open(highlight.thumbnail_path_full_size).size != (1440, 1080):
+            await highlight.scale_video(1440, 1080)
             coroutines = [
                 highlight.extract_thumbnails(),
                 highlight.extract_snippet_in_lower_resolution(),

I basically moved the box to the left, and resized the video to the right size earlier ! I tried on the screenshot you provided with Reyna and it works.

[MarcoColomb0]

That's not working for me 😢 Some of my clips are recorded in this size, for example: I think I still have something to tweak, considering my in-game resolution of 1280x960

[Flowtter]

Could you send me the clip ? You can just upload it there or if it doesn't work, on youtube

[Flowtter]

Hi Marco,

I created a PR that should solve your issue. https://github.com/Flowtter/crispy/pull/43

Let me know if you have any issues, I'll merge it next week

[MarcoColomb0]

Hey Brice, great news – it's working perfectly!

I just had to adjust the confidence level a bit (0.3 seems like the sweet spot, but maybe I could tweak it a bit more). For now, I tested around 10/15 clips, and all the kills are being detected. You did a great job, and I want to thank you. I think the pull request could be merged, and if you need more testing, I'll definitely let you know in the next few days. You can close the issue whenever you want :)

[Flowtter]

Hi Marco, I'm glad to hear about that ! Yeah, I'm not surprised for the confidence! Merging the PR and closing this then :)