MShawon / YouTube-Viewer

A multithreaded view bot for YouTube
MIT License
1.5k stars 760 forks source link

Compiled list of anti-bot security measures by google #593

Open JijaProGamer opened 1 year ago

JijaProGamer commented 1 year ago

Is there an existing issue for this?

I'm submitting a ...

Description

I'm providing here a list of anti-bot POST network requests made by youtube.

I am using firefox since my bot is also based on firefox, but these should help you out no matter what browser you use (even though you should use firefox)

Environment

- OS : Windows 10
- Python: Not applicable
- Script version: not applicable
- Browser: Firefox v117

config.json

Not applicable
JijaProGamer commented 1 year ago

First network request when opening any youtube website. Used to track basic info about your fingerprint:

network request data:

    {
        "method": "POST",
        "headers": {
            "host": "www.youtube.com",
            "user-agent": "Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Firefox/102.0",
            "accept": "*/*",
            "accept-language": "en-US,en;q=0.5",
            "accept-encoding": "gzip, deflate, br",
            "content-type": "application/json",
            "content-length": "2078",
            "referer": "https://www.youtube.com/",
            "x-goog-eom-visitor-id": "CgttS0tnak81WWZkWSjh05uoBjIGCgJSTxIA",
            "x-youtube-bootstrap-logged-in": "false",
            "x-youtube-client-name": "1",
            "x-youtube-client-version": "2.20230914.04.00",
            "origin": "https://www.youtube.com",
            "sec-fetch-dest": "empty",
            "sec-fetch-mode": "same-origin",
            "sec-fetch-site": "same-origin",
            "connection": "keep-alive",
            "cookie": "YSC=zW3r7T6c4V4; __Secure-YEC=REDACTED; VISITOR_PRIVACY_METADATA=REDACTED; CONSENT=PENDING+373; PREF=tz=Europe.Bucharest"
        },
        "url": "https://www.youtube.com/youtubei/v1/guide?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8&prettyPrint=false"
    },

postData:

{
  "context": {
    "client": {
      "hl": "en",
      "gl": "RO",
      "remoteHost": "<redacted IP>",
      "deviceMake": "",
      "deviceModel": "",
      "visitorData": "CgttS0tnak81WWZkWSjh05uoBjIGCgJSTxIA",
      "userAgent": "Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Firefox/102.0,gzip(gfe)",
      "clientName": "WEB",
      "clientVersion": "2.20230914.04.00",
      "osName": "Windows",
      "osVersion": "10.0",
      "originalUrl": "https://www.youtube.com/",
      "platform": "DESKTOP",
      "clientFormFactor": "UNKNOWN_FORM_FACTOR",
      "configInfo": {
        "appInstallData": "COHTm6gGENShrwUQ26-vBRDZya8FEJfn_hIQ6sOvBRDuoq8FEN3rrwUQp-r-EhDd6P4SEKbs_hIQ1-mvBRDr6P4SENPhrwUQ1eWvBRDks_4SELzrrwUQ5OavBRDi1K4FENzjrwUQzK7-EhDE3a8FELzM_hIQ1eqvBRCG6q8FEMzfrgUQuIuuBRC9tq4FEOe6rwUQiOOvBRCst68FEKTerwUQjMuvBRD6vq8FELTJrwUQx-avBRD65P4SEInorgUQpcL-EhC1pq8FEPOorwUQtOavBQ%3D%3D"
      }
    },
    "browserName": "Firefox",
    "browserVersion": "102.0",
    "acceptHeader": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "deviceExperimentId": "ChxOekkzT1RjMk16QXdNVEEzTVRrek9EWTBOdz09EOHTm6gGGOHTm6gG",
    "screenWidthPoints": 1280,
    "screenHeightPoints": 720,
    "screenPixelDensity": 1,
    "screenDensityFloat": 1,
    "utcOffsetMinutes": 180,
    "userInterfaceTheme": "USER_INTERFACE_THEME_LIGHT",
    "mainAppWebInfo": {
      "graftUrl": "https://www.youtube.com/",
      "webDisplayMode": "WEB_DISPLAY_MODE_BROWSER",
      "isWebNativeShareAvailable": false
    },
    "timeZone": "Europe/Bucharest"
  },
  "user": {
    "lockedSafetyMode": false
  },
  "request": {
    "useSsl": true,
    "internalExperimentFlags": [
      {
        "key": "force_enter_once_in_webview",
        "value": "true"
      }
    ],
    "consistencyTokenJars": []
  },
  "adSignalsInfo": {
    "params": [
      {
        "key": "dt",
        "value": "1694951906097"
      },
      {
        "key": "flash",
        "value": "0"
      },
      {
        "key": "frm",
        "value": "0"
      },
      {
        "key": "u_tz",
        "value": "180"
      },
      {
        "key": "u_his",
        "value": "1"
      },
      {
        "key": "u_h",
        "value": "900"
      },
      {
        "key": "u_w",
        "value": "1440"
      },
      {
        "key": "u_ah",
        "value": "860"
      },
      {
        "key": "u_aw",
        "value": "1440"
      },
      {
        "key": "u_cd",
        "value": "24"
      },
      {
        "key": "bc",
        "value": "31"
      },
      {
        "key": "bih",
        "value": "720"
      },
      {
        "key": "biw",
        "value": "1263"
      },
      {
        "key": "brdim",
        "value": "4,4,4,4,1440,0,1292,811,1280,720"
      },
      {
        "key": "vis",
        "value": "1"
      },
      {
        "key": "wgl",
        "value": "true"
      },
      {
        "key": "ca_type",
        "value": "image"
      }
    ]
  },
  "fetchLiveState": true
}
JijaProGamer commented 1 year ago

Request made when visiting a video. Possibly to fingerprint as well. Many details are the same as the first important request.

network request data:

    {
        "method": "POST",
        "headers": {
            "host": "www.youtube.com",
            "user-agent": "Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Firefox/102.0",
            "accept": "*/*",
            "accept-language": "en-US,en;q=0.5",
            "accept-encoding": "gzip, deflate, br",
            "content-type": "application/json",
            "content-length": "2244",
            "referer": "https://www.youtube.com/",
            "x-goog-eom-visitor-id": "CgttS0tnak81WWZkWSjh05uoBjIGCgJSTxIA",
            "x-youtube-bootstrap-logged-in": "false",
            "x-youtube-client-name": "1",
            "x-youtube-client-version": "2.20230914.04.00",
            "origin": "https://www.youtube.com",
            "sec-fetch-dest": "empty",
            "sec-fetch-mode": "cors",
            "sec-fetch-site": "same-origin",
            "connection": "keep-alive",
            "cookie": "YSC=REDACTED; __Secure-YEC=REDACTED; VISITOR_PRIVACY_METADATA=REDACTED; CONSENT=PENDING+373; PREF=tz=Europe.Bucharest; SOCS=REDACTED"
        },
        "url": "https://www.youtube.com/youtubei/v1/att/get?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8&prettyPrint=false"
    }

postData:

{
  "context": {
    "client": {
      "hl": "en",
      "gl": "RO",
      "remoteHost": "<redacted IP>",
      "deviceMake": "",
      "deviceModel": "",
      "visitorData": "CgttS0tnak81WWZkWSjh05uoBjIGCgJSTxIA",
      "userAgent": "Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Firefox/102.0,gzip(gfe)",
      "clientName": "WEB",
      "clientVersion": "2.20230914.04.00",
      "osName": "Windows",
      "osVersion": "10.0",
      "originalUrl": "https://www.youtube.com/",
      "platform": "DESKTOP",
      "clientFormFactor": "UNKNOWN_FORM_FACTOR",
      "configInfo": {
        "appInstallData": "COHTm6gGENShrwUQ26-vBRDZya8FEJfn_hIQ6sOvBRDuoq8FEN3rrwUQp-r-EhDd6P4SEKbs_hIQ1-mvBRDr6P4SENPhrwUQ1eWvBRDks_4SELzrrwUQ5OavBRDi1K4FENzjrwUQzK7-EhDE3a8FELzM_hIQ1eqvBRCG6q8FEMzfrgUQuIuuBRC9tq4FEOe6rwUQiOOvBRCst68FEKTerwUQjMuvBRD6vq8FELTJrwUQx-avBRD65P4SEInorgUQpcL-EhC1pq8FEPOorwUQtOavBQ%3D%3D"
      },
      "playerType": "UNIPLAYER",
      "tvAppInfo": {
        "livingRoomAppMode": "LIVING_ROOM_APP_MODE_UNSPECIFIED"
      }
    },
    "browserName": "Firefox",
    "browserVersion": "102.0",
    "acceptHeader": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "deviceExperimentId": "ChxOekkzT1RjMk16QXdNVEEzTVRrek9EWTBOdz09EOHTm6gGGOHTm6gG",
    "screenWidthPoints": 1280,
    "screenHeightPoints": 720,
    "screenPixelDensity": 1,
    "screenDensityFloat": 1,
    "utcOffsetMinutes": 180,
    "userInterfaceTheme": "USER_INTERFACE_THEME_LIGHT",
    "mainAppWebInfo": {
      "graftUrl": "https://www.youtube.com/",
      "webDisplayMode": "WEB_DISPLAY_MODE_BROWSER",
      "isWebNativeShareAvailable": false
    },
    "timeZone": "Europe/Bucharest"
  },
  "user": {
    "lockedSafetyMode": false
  },
  "request": {
    "useSsl": true,
    "internalExperimentFlags": [
      {
        "key": "force_enter_once_in_webview",
        "value": "true"
      }
    ],
    "consistencyTokenJars": []
  },
  "adSignalsInfo": {
    "params": [
      {
        "key": "dt",
        "value": "1694951920095"
      },
      {
        "key": "flash",
        "value": "0"
      },
      {
        "key": "frm",
        "value": "0"
      },
      {
        "key": "u_tz",
        "value": "180"
      },
      {
        "key": "u_his",
        "value": "1"
      },
      {
        "key": "u_h",
        "value": "900"
      },
      {
        "key": "u_w",
        "value": "1440"
      },
      {
        "key": "u_ah",
        "value": "860"
      },
      {
        "key": "u_aw",
        "value": "1440"
      },
      {
        "key": "u_cd",
        "value": "24"
      },
      {
        "key": "bc",
        "value": "31"
      },
      {
        "key": "bih",
        "value": "720"
      },
      {
        "key": "biw",
        "value": "1263"
      },
      {
        "key": "brdim",
        "value": "4,4,4,4,1440,0,1292,811,1280,720"
      },
      {
        "key": "vis",
        "value": "1"
      },
      {
        "key": "wgl",
        "value": "true"
      },
      {
        "key": "ca_type",
        "value": "image"
      }
    ]
  },
  "engagementType": "ENGAGEMENT_TYPE_PLAYBACK",
  "ids": [
    {
      "playbackId": {
        "cpn": "EljjkHNU6y-SeKJt"
      }
    }
  ]
}
JijaProGamer commented 1 year ago

Request made when accessing any *.youtube.com page, made for initializing the GRPC connection

network request data:

    {
        "method": "POST",

        "headers": {
            "host": "jnn-pa.googleapis.com",
            "user-agent": "Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Firefox/102.0",
            "accept": "*/*",
            "accept-language": "en-US,en;q=0.5",
            "accept-encoding": "gzip, deflate, br",
            "content-type": "application/json+protobuf",
            "content-length": "24",
            "referer": "https://www.youtube.com/",
            "x-goog-api-key": "AIzaSyDyT5W0Jh49F30Pqqtyfdf7pDLFKLJoAnw",
            "x-user-agent": "grpc-web-javascript/0.1",
            "origin": "https://www.youtube.com",
            "sec-fetch-dest": "empty",
            "sec-fetch-mode": "cors",
            "sec-fetch-site": "cross-site",
            "connection": "keep-alive"
        },
        "url": "https://jnn-pa.googleapis.com/$rpc/google.internal.waa.v1.Waa/Create"
    }

postData:

[\"<REDACTED>\"]
JijaProGamer commented 1 year ago

Weird POST with encrypted data, internally named "LOG_EVENT", my best guess is that it's used to log stuff like hovering over suggestions, pausing, clicking, etc.

network request data:

    {
        "method": "POST",
        "headers": {
            "host": "www.youtube.com",
            "user-agent": "Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Firefox/102.0",
            "accept": "*/*",
            "accept-language": "en-US,en;q=0.5",
            "accept-encoding": "gzip, deflate, br",
            "content-type": "application/json",
            "content-length": "4505",
            "referer": "https://www.youtube.com/",
            "x-goog-request-time": "1694951924455",
            "x-goog-eom-visitor-id": "CgttS0tnak81WWZkWSjh05uoBjIGCgJSTxIA",
            "content-encoding": "gzip",
            "x-youtube-client-name": "1",
            "x-youtube-client-version": "2.20230914.04.00",
            "x-youtube-device": "cbr=Firefox&cbrver=102.0&ceng=Gecko&cengver=102.0&cos=Windows&cosver=10.0&cplatform=DESKTOP",
            "x-youtube-page-cl": "565485865",
            "x-youtube-page-label": "youtube.desktop.web_20230914_04_RC00",
            "x-youtube-utc-offset": "180",
            "x-youtube-time-zone": "Europe/Bucharest",
            "x-youtube-ad-signals": "dt=1694951906097&flash=0&frm&u_tz=180&u_his=1&u_h=900&u_w=1440&u_ah=860&u_aw=1440&u_cd=24&bc=31&bih=720&biw=1263&brdim=4%2C4%2C4%2C4%2C1440%2C0%2C1292%2C811%2C1280%2C720&vis=1&wgl=true&ca_type=image",
            "origin": "https://www.youtube.com",
            "sec-fetch-dest": "empty",
            "sec-fetch-mode": "cors",
            "sec-fetch-site": "same-origin",
            "connection": "keep-alive",
            "cookie": "YSC=zW3r7T6c4V4; VISITOR_PRIVACY_METADATA=CgJSTxIA; CONSENT=PENDING+373; PREF=tz=Europe.Bucharest; SOCS=CAISEwgDEgk1NjUxODgxOTQaAmVuIAEaBgiAg5moBg; VISITOR_INFO1_LIVE=FB1wVmopH_c"
        },
        "url": "https://www.youtube.com/youtubei/v1/log_event?alt=json&key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8"
    }

postData:

\u001f�\b\u0000\u0000\u0000\u0000\u0000\u0000\u0003�\\\u000bo�Ȗ�+\u0011ڻڽJ'.���\u001di�\u0018�\u0004\u001b����\u00112؀���m����{�LH&=\r=Yw��t[i�\u001cW��wN�2��V��Q��R���$\f��^��R��G��Ҍ\\�-���\u001a��/���Ǯ��A\u001c�\u0018��eX�Ȉ�a��!3�h\u001a̴h\u001a\u0013��j�Ei�a��\\�����0k��5O�N�\u0015>m�fm�w��z?�Fs�#�-B[�O@3pBƭ>%��y�\u0013ڜ�>�S2\u000e}Z�q\t�\u0019m�\u000f�=B[�#�5\u000ft.�r\t-@\u000f\u001c�;<\u0012��A$�TL�6\u000f:��?�8�!\u0010��a�̜�Z[��&gO0��\u000bdn�\"�)i&��\u0007�'�G}Md\b\u001b2�i�\u0013��\u0013�+�D?-�\t�դ\tr\u0015�\"�ڭ��ˈ~���\u001a�P\u000f��B���g톐t��f�O��G��$\u0004D�pG�8��>3\u0004\"_�D�C; 4�����ֲ��g�\u001fg\u0002����r?R_H\u0002�Qk�\t-\\S~�DB��-�1m��\u0006�\u0003�{\u0000��/�mw1��5�3�e+�q�4\u000e��êY��k��Ѹ.��24��Dݻ��r�s�\u0001�q=6���=�Wmv�\u0018���it\u0013׉9w��\u001cd̛�������Mǵ���_9^��zjX\u001d�M�o�\\��9�J�0u�aP���hҨo:}#�\"9�H3�j�\u000b�Ӗs�:=��P�g�W�ΡZ��+�X���Rb�}#\u0019,��I�ʺ\f����\u001d/P����c���t�6Zs����N��\u000e�a�7\u001e;�8���\u000ej\u001c��EXi���#?�K��Q5t���X���a\u000f,�衃&\r/��۽ۭ\u000bݻ������^����Io^��u\u0005R�jw\rcx�:C\\����a��)�Ϭ������\u001fB�s�a��3���k1���\u000f\u0005�qX�\u0017�\u001b��=��}��W��\u0012p�8���\u000e��5;!|\u001eg��ΰ;���mu{��\u0003F�\u0017\u001cА�{�3�:�\u0006�÷�փ��G�\u0016�\u0006�S�f���~�Т��'��ɔ�\u001e }!���\b�Gm�\u0001ڃH�=\u000euLh�����GhO-:w�\u0013�vX\u0011ڪ�\u0010\u001a\u0016\b�\u0019%��>��u\u0011R�����}\u0011�o\u0015\u0001\bG�rد\"]]m�ƀ5�\u001a��ad6����*\u001e��w\u0017\u000e\u000f��C\u001cn�}�i׼Ⱥ�1\u001d{�\u001d2uی4NW�$d�_�֩�Tft�(��!\bC���a���\u0017D^�M�\f�\n17��\\%�2b�\u001b濯\u001a�d\u0011߲\fb�\u0007]Ճğƻ[z\u0017`�t���Z����r��ǵ ]��^�=XrJ=�:�iV�Y\u0019��VM\u001dU�V�RM��8\u0001�~�/N�#�҉��J=ˋ��أʔ�z\u0018\u001d\u0005ҳi�,�RS�\u0007��&�\u0012�i��t�\f`�L\"7Tw+?\t���z���R����\u0016�\u001e���?�ɸQ\u001c�e\u0010���M�oA��\r�D�,�_���;p�708�@/m`��g$Ȝ�#��X`��_5I�(�\u000b��\\�&�o��;�,��,[�����v{����z��L��-�_l�n��V7g{�9z�E×�\u001d\\�ftD\\�2\u0014\u0002�嘩�[�f��Y��(�\b\u0017<#�\u0012G.8\u0016\u000b�5\u000f@�\u0004I�\u0014��xBa\u0004̳9E�\u0018J�\u000b���$��\u0012$^ cxV`�\u0002˲�P�\u0000W�|\u001c��t�\b,&�Y���� \u0011}X��9zK����\u0016�\u0019B�x,�\t\f���\u000f�yBA\u0012G5�\u0002+��\u0007K<H'\f9,c���xQ���`3��E�a�\u0002q\f���<�H�:\u0003CLg���\f��3Ce�X�\u0006J���-,�\u0000\u0013�`0\u0007n�\u0006�����3\"fs?c^�BY�\u001d�\u0000�r,\"&K���\u0010\b<�\u0006Btd�\u0010+\u0018����2��YH`e�\u0006��\u0006\u000eL\u0015\u0010�\u0007��\u0019ư�(I�\u0005�0\u000f�$PY\b!1w8f�<�\u0002�\t\u001f\u0001��6\u001ae���C\u0012\f��\u0018\u0016�\u0012�&\u000b\f� a8\u0010G(\u0002x��`e$Q���(w&D��5�����<�\u001cx��\u001b�\"�����1��hP\u0000\u0000��\u0003\u0012X�\u00100�H9�(�G\u001c�t0#������P�\n2˜�\u0011�\u001f��\"�g��/��c��\u0005�:,s\u0002G�L\u0010���\u0014�\\\r�\u0000\u0012܂y�r\ra6��RG�?�(�$\t�X\u0005U��\u0000�)��_��\f\u0011:� �2Xx�8\u0016�F�b��\u0014c\u00020Dy��P\u0007G\u0011��,\u0010G�\t��y,\u0000ct0\u0002\u001cѼ\u0010\u0011`�0\u0014��s�Ҵ�P\n\u001cu\u0014\u000f�@�`,��\u001c�\u0001\u0000�D���\u001cQH\u00121� \u0014\rȫ<O9�:p��\u0007�\u0011�j���)6\u0010�\u0011D\u001a\u001d�\u0011��X���\u0002���\n�?�P�X�З#�U��F�9+7q��\r�閟l��o\u0005�����ki�t����/\"��\"�g��d�'��N�0��:k�N�`���&�֒0��/��?��_����U\u0012{���JwZ45��A/��TϞ��\u00154N0<��)\t�+�\u000bBO�9��T����T�O��+\u0014�֨c�U\u0017�\tw��)�N��\u000f�\r��S�;��iA{�|�u�T\tNk�\tׯ��KI E�Wh `�=Y؉;�i�����u:q3�+�I�q�,2��z=��,��k]CzKZ2?�u�\u001e\u001a\u000b>�<\n��f3\u0005Xe�~4r�۱��7�i\u0019��eN�\u001c�3��\u001f\u0005�0WF,�\u001c�y\u0005rŌPf\u0011p��,n<h@�l2WI#V�KxY(c\t��L�B��2�>GY�\\YPV�]�IEr\u0016!�,�s�\u0007�r�eQ�\u001c��ݹ�\u0017�� ��u\u0018��HK��H��R\u0006\u0011���h;+���,�\u0011�c\u0017�������2t�9���<a[(S\u0011��s��o`���l�\u0002V�\u000fU�� J�T�\u0011�\u001e��s�\u0014� Q\u0014P\u0019V�ϑV$R\u0010Ǘ9�ߣ��f� ���]�5\u000bqBY�ڲp�-㰅)�m�LٲD�J�P�P��/�Q44���\u000bb��%\t�\u0005O\u001b\u0005��2��<�7��E�l\u0014�`\u00114�\u0010�V�(�\u0018\u001a���_\u0012d�bA\u0006``�J��%)�7�$د\u0000o�H�9Y\u0014���\u001cU�\u0005�q�\u0019�\f��Yg�z~���խ7��\u0013�an\u001f�q\u0010ݦ���C�O�y�\u0010\bk\\F��n�|\u0001h��&�k\u0017�2_� ����5��\u00157\fǰ����\u0000��/A�}F���Q��[\"�#\u0015\u001f`�,\u0001Է�;����*Q�t� �ꉻ��b�+B������%�D��0���e\u0010=�qv\fIq\u0001�G:q���k�\u001cʿ�S���Ŵz~/p���9J�դwq�r\u0019\t������;B}Amؼ2\u00000�A��\u0002��,�\u0016n\t\b\u0013A���Uw�z�}<�뒎��*\u000e�K��~�C�\r�Q0\u0001\u0000<+pq\u001c]\u0002%�1&��\u001b՚�\u0004q\u0012d��AK\u0016\\�9\u001a��\u0011�B5�e��`=\u001b��wM?F\u0017L\u001fP�x�0�'�v�\u0006�\u0018�>ZC@\u0017�a͛�\u001f�-��e\u0001�|��7�\u001b7<%�2�h��Q\u0002OB��\u0011\u0013�\r~�T��\u0005V�􃫏ċlY\u0000M&�%���\u0011��E�\u001e�^�}������\u00027�^��>�\\h���,\u000f�LV\u0016�_��\u0014�tHe�D[)��\u0000\u001bCX�V���1�)W�߀��\u0001;\f(�\u001fʇ���\fO�\u0015,�WN\u000b� �h�$^�K&����\r\u0019�1�8�o|`�#T\"��*�����\u0007��#�Tl{��؏P�!��>8,�`�Or�'�$�yS\u0018�č�f��+\u0010[�f��}$��Ҥ\u0010�\fz���`%\t\u000be�Aj��K� �����W\u000f��c�a<��4ug�\u0014L�o|r\u0016V5͖9j�]�9�UL�Ќ\u00069}�<�d�&WS\u0017dxW�ڿ���<��'�n�_W����W�u��|u�O�W�A?)��>�,���^�Rw�\n��\u001f��Y���r�޹�\tYe�\u0005� ۓc�%�'��9��^�6A�vC5���)k\u001eo#��NJ�.�\n��\u001f5N�;\u0007�6�\u0019�\u000eg\u001c4^?tn�),�\u000fȸ�\"�fmz����yp\u0014��n���%�=�&�ݬ�w��N��x�l�~��t�\rIK,�`�x�\u0010N@�ɐj���K���w��^*�\u0012��\u0018K\u0016?�\u0012^�>ir�\u0017m\t>o����b��\u0014m�Ĝ3�ce�Ɛ���\u0013���ۖ\n�a���%[f��ܚ�fتYQl�e���F\u0003V�QC�l�TG����\u0007C55e�45�t\t�gm'g�Ki\u0018��hUr��2�g3�S_����aXrZ6%�_^���qZs?\f��\u000b���w�Q�u�\t6nh�A��G��6J\u000f�uFBƼ\u001a��ٚ�j9fW�V�#ˮ؎�\u001b[ij]�PL��J��\u00185��\u0019jm�XF���$N�\\��\"l5��Q��Ѡ��PzG�5\u001dK�,��6�ӈ�K�Xj�<�HVG�Q;�s��7@%�Q\u0014`Aވ���n����\u001c��W4cTi��V�J�\u0018��͡\u0002\f%����m����;i�5�4\u0017}�\u0016�����9�\u0003{�\u0019�8\bR�bWFmS�T��&!����-wv\u0017����kA8\u0013�/�zG�NA8��\u0005|v\u001d��TAOsd�U�Q�PG�f8��M\u001e\\�U��֡yу_J�U�(��\u0010ѻ��A��\\\u0016�B�*p�2\u001f@��64�6+_ͭ����:�\r7?.�����Tx?f_��g�,X�p����ӽ!*.�p�\u0002\u0010�~g���\u0012�j\u0013� ���Z�M\u0016Ngf����t�\"�ߚX�hMX���B��������~��9\n��~�/0���:���4O�^��]�Р;\rA�h�'\u0013��\u000e&��7��긧w��\u001f��G\u0019�^�A��\u0012��>-lI�W\u0006��~�g/��]�w��Y\u0012�#�·�n컔�4H�rC\u001c:)V�'k�^?�����튝�~����\u0003\u0018\u0018D���]�5U���懙���\u0018��ʒ\u0000���7�[P/�\u0016�I�;\u0015ҳ\n�\u001e٦�h\u0000ȡѫV����e��\u0005Å7ۄ��������\u001c�+,�ܛm�pQó;�\\ß��WTÂ�Պ_�\u000bl�ߗ�g\u001f�Ahd\"7���A�&\t�O�ĩ�\u0004n\u0018\u001c|�t�\u0018���(m\u0006�n5��L\u001f\u001b�dٝ������Ȼ�\u0006CK^\r\u001bݵ�\b\u0019ߒc���}v�\u001f��\b�<�]�q�ʉ��N'K�\u0019��՘��jd�k4�k�2K\u001f\u0014'l9Ȭ��nV5E��Am\u001b�+3��u��vUu�i��/�\\N��Ϥ8�g���%O�aOۀ7\u0002�7�\f�����p\u0012��\u0000�e�S��.n\rXy�a�g��A�>\u001d�:�R��FE�*�Y����\u0005+\u0011y\"\u001c�a�݇4���J|�X\u0000苫���-l�˪��X��Ec8\b�J�‚��\u000fڃ�U\u0016�ʬ@)�h=ey�#2�\u000f\u0005��9I\u0000\u0000
JijaProGamer commented 1 year ago

Request sent every few seconds while watching a video, to show youtube your progress on the video:

network request data:

    {
        "method": "POST",
        "headers": {
            "host": "www.youtube.com",
            "user-agent": "Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Firefox/102.0",
            "accept": "*/*",
            "accept-language": "en-US,en;q=0.5",
            "accept-encoding": "gzip, deflate, br",
            "content-type": "application/x-www-form-urlencoded",
            "content-length": "226",
            "referer": "https://www.youtube.com/",
            "x-goog-visitor-id": "CgtGQjF3Vm1vcEhfYyj105uoBjIGCgJSTxIA",
            "x-youtube-client-name": "1",
            "x-youtube-client-version": "2.20230914.04.00",
            "x-youtube-device": "cbr=Firefox&cbrver=102.0&ceng=Gecko&cengver=102.0&cos=Windows&cosver=10.0&cplatform=DESKTOP",
            "x-youtube-page-cl": "565485865",
            "x-youtube-page-label": "youtube.desktop.web_20230914_04_RC00",
            "x-youtube-utc-offset": "180",
            "x-youtube-time-zone": "Europe/Bucharest",
            "x-youtube-ad-signals": "dt=1694951960794&flash=0&frm&u_tz=180&u_his=1&u_h=900&u_w=1440&u_ah=860&u_aw=1440&u_cd=24&bc=31&bih=720&biw=1263&brdim=4%2C4%2C4%2C4%2C1440%2C0%2C1292%2C811%2C1280%2C720&vis=1&wgl=true&ca_type=image",
            "origin": "https://www.youtube.com",
            "sec-fetch-dest": "empty",
            "sec-fetch-mode": "cors",
            "sec-fetch-site": "same-origin",
            "connection": "keep-alive",
            "cookie": "YSC=zW3r7T6c4V4; VISITOR_PRIVACY_METADATA=CgJSTxIA; CONSENT=PENDING+373; PREF=tz=Europe.Bucharest; SOCS=CAISEwgDEgk1NjUxODgxOTQaAmVuIAEaBgiAg5moBg; VISITOR_INFO1_LIVE=FB1wVmopH_c; GPS=1"
        },
        "url": "https://www.youtube.com/api/stats/qoe?fmt=243&cpn=h4fYjUTUAKyV87Qy&el=detailpage&ns=yt&fexp=v1%2C23983296%2C2730%2C18618%2C2602%2C73492%2C54572%2C233521%2C2567%2C381%2C12995%2C54587%2C84737%2C25387%2C973%2C8870%2C1087%2C3781%2C1193%2C976%2C322%2C3200%2C125880%2C4132%2C26306256%2C26%2C171%2C136%2C840%2C146%2C2225%2C536%2C1253%2C677%2C612%2C243%2C5869%2C5694&cl=564805759&seq=1&docid=XaK1ht8poRc&ei=Q-sGZevOKNjigAeziomIDA&event=streamingstats&feature=g-high-rec&plid=AAYFjNNZaVEN45oR&referrer=https%3A%2F%2Fwww.youtube.com%2F&sdetail=p%3A%2F&sourceid=y&qclc=ChBoNGZZalVUVUFLeVY4N1F5EAE&embargoed=0&cbr=Firefox&cbrver=102.0&c=WEB&cver=2.20230914.04.00&cplayer=UNIPLAYER&cos=Windows&cosver=10.0&cplatform=DESKTOP&vps=0.000:N,0.006:N&cat=streaming&vfs=0.006:243:243::r&view=0.006:371:209&bwe=0.006:130000&vis=0.006:0&cmt=0.006:0.000&bh=0.006:0.000"
    }

postData:

session_token=REDACTED
JijaProGamer commented 1 year ago

Event POST message. This affirms my theory that youtube sends your key strokes and mouse movements to their server to detect bot activity:

    {
        "method": "POST",
        "headers": {
            "host": "www.youtube.com",
            "user-agent": "Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Firefox/102.0",
            "accept": "*/*",
            "accept-language": "en-US,en;q=0.5",
            "accept-encoding": "gzip, deflate, br",
            "content-type": "application/json",
            "content-length": "3643",
            "referer": "https://www.youtube.com/watch?v=XaK1ht8poRc",
            "x-goog-request-time": "1694952268990",
            "x-goog-visitor-id": "CgtGQjF3Vm1vcEhfYyj105uoBjIGCgJSTxIA",
            "x-youtube-client-name": "1",
            "x-youtube-client-version": "2.20230914.04.00",
            "x-youtube-device": "cbr=Firefox&cbrver=102.0&ceng=Gecko&cengver=102.0&cos=Windows&cosver=10.0&cplatform=DESKTOP",
            "x-youtube-page-cl": "565485865",
            "x-youtube-page-label": "youtube.desktop.web_20230914_04_RC00",
            "x-youtube-utc-offset": "180",
            "x-youtube-time-zone": "Europe/Bucharest",
            "x-youtube-ad-signals": "dt=1694951925740&flash=0&frm&u_tz=180&u_his=2&u_h=900&u_w=1440&u_ah=860&u_aw=1440&u_cd=24&bc=31&bih=720&biw=1263&brdim=4%2C4%2C4%2C4%2C1440%2C0%2C1292%2C811%2C1280%2C720&vis=1&wgl=true&ca_type=image",
            "origin": "https://www.youtube.com",
            "sec-fetch-dest": "empty",
            "sec-fetch-mode": "cors",
            "sec-fetch-site": "same-origin",
            "connection": "keep-alive",
            "cookie": "YSC=zW3r7T6c4V4; VISITOR_PRIVACY_METADATA=CgJSTxIA; CONSENT=PENDING+373; PREF=tz=Europe.Bucharest; SOCS=CAISEwgDEgk1NjUxODgxOTQaAmVuIAEaBgiAg5moBg; VISITOR_INFO1_LIVE=FB1wVmopH_c; GPS=1"
        },
        "url": "https://www.youtube.com/youtubei/v1/log_event?alt=json&key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8"
    },

postData:

{
   "context":{
      "client":{
         "hl":"en",
         "gl":"RO",
         "clientName":1,
         "clientVersion":"2.20230914.04.00",
         "configInfo":{
            "appInstallData":"CPXTm6gGELWmrwUQ1uqvBRDM364FEIjjrwUQteavBRCX5_4SEMTdrwUQzK7-EhDuoq8FENXlrwUQjMuvBRDk5q8FEPq-rwUQ1KGvBRDqw68FEOLUrgUQp-r-EhDr6P4SEPrk_hIQ2cmvBRC0ya8FENPhrwUQ57qvBRDbr68FEKPerwUQhuqvBRCJ6K4FEKy3rwUQu-uvBRClwv4SEKXs_hIQ8qivBRDks_4SELzM_hIQ3ej-EhC4i64FEL22rgUQx-avBRDd668FEPTnrwUQ3fWvBQ%3D%3D",
            "coldConfigData":"CPXTm6gGEO-6rQUQvbauBRDi1K4FEPKorwUQyLmvBRCfx68FEObLrwUQg8yvBRCj3q8FEMLerwUQxd-vBRCj4q8FELXmrwUQ5OavBRD0568FEOHprwUQhuqvBRDW6q8FELvrrwUQgu6vBRD9768FEM7wrwUQ0PCvBRDY8K8FEN31rwUaMkFPakZveDNzTk1udld1eDloZUxyUy1wSlcwRV8tbjdzeURUdTJwWkdERjFoVVZKZmRRIjJBT2pGb3gzc05NbnZXdXg5aGVMclMtcEpXMEVfLW43c3lEVHUycFpHREYxaFVWSmZkUSpEQ0FNU0xnMFZncGFvQXNnV19nV21INTBLdlFfOUJ1QWVGUi1TZ3RBTXMwZWd0Z2FPSHJKRzNtS2RMNURaQlBPcHpBcz0%3D",
            "coldHashData":"CPXTm6gGEhMxNzg2MTM0NzkyODA4NzE3NTM0GPXTm6gGMjJBT2pGb3gzc05NbnZXdXg5aGVMclMtcEpXMEVfLW43c3lEVHUycFpHREYxaFVWSmZkUToyQU9qRm94M3NOTW52V3V4OWhlTHJTLXBKVzBFXy1uN3N5RFR1MnBaR0RGMWhVVkpmZFFCRENBTVNMZzBWZ3Bhb0FzZ1dfZ1dtSDUwK3c4cXVPUmdNQlhOTFJiS3k0cTRKcWc4QUFRNkFJVVNvTHVaZjFMc1lpLTV3c0ZFUkx4akozNEdvYmp2N0hrcnlUbWVqdElkYVRuMkdoeFhxR0RSTVhoVlZLaW1sZ0YycklqSkJUQW5oT0kxN3lOSXlTcjdtbElkakg1bWFzdndydVVRdVR2dW80V2hsVEhKdk00SjRxN2R3TWFJSkN3aVZNSjNtc2NzWG1rbk1lNUlqa1dvVzFSZ2xvVGhBQT09",
            "hotHashData":"CPXTm6gGEhQxNTQ0MjIwMTg5MjY0MTk4NTQ1Mxj105uoBiiU5PwSKNuT_RIoxrL9EiiqtP0SKKXQ_RIonpH-Eiiarf4SKIuu_hIoyMr-EiiUzf4SKN3O_hIowd3-Eiio4f4SKJnm_hIojOn-Eij26f4SKKTq_hIopez-EijE7P4SMjJBT2pGb3gzc05NbnZXdXg5aGVMclMtcEpXMEVfLW43c3lEVHUycFpHREYxaFVWSmZkUToyQU9qRm94M3NOTW52V3V4OWhlTHJTLXBKVzBFXy1uN3N5RFR1MnBaR0RGMWhVVkpmZFFCMENBTVNJQTBMb3RmNkZiNF95Z0NvT2NBUkZSZmR6OElNcUpzQ2ktNEI4ZUFPdFlBRQ%3D%3D"
         },
         "userAgent":"Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Firefox/102.0",
         "mainAppWebInfo":{
            "webDisplayMode":"WEB_DISPLAY_MODE_BROWSER"
         },
         "browserName":"Firefox",
         "browserVersion":"102.0",
         "osName":"Windows",
         "osVersion":"10.0",
         "platform":"DESKTOP"
      },
      "request":{
         "internalExperimentFlags":[
            {
               "key":"force_enter_once_in_webview",
               "value":"true"
            }
         ]
      }
   },
   "events":[
      {
         "eventTimeMs":1694952267942,
         "latencyActionTicked":{
            "tickName":"att_s",
            "clientActionNonce":"2LRg0P--wguss09S"
         },
         "context":{
            "lastActivityMs":"124"
         }
      },
      {
         "eventTimeMs":1694952267942,
         "latencyActionTicked":{
            "tickName":"bg_v",
            "clientActionNonce":"2LRg0P--wguss09S"
         },
         "context":{
            "lastActivityMs":"125"
         }
      },
      {
         "eventTimeMs":1694952267984,
         "latencyActionTicked":{
            "tickName":"bg_s",
            "clientActionNonce":"2LRg0P--wguss09S"
         },
         "context":{
            "lastActivityMs":"167"
         }
      },
      {
         "eventTimeMs":1694952267984,
         "latencyActionTicked":{
            "tickName":"att_f",
            "clientActionNonce":"2LRg0P--wguss09S"
         },
         "context":{
            "lastActivityMs":"168"
         }
      },
      {
         "eventTimeMs":1694952268374,
         "visualElementShown":{
            "csn":"MC43OTc4MDk3NDUwMjU2MTU0",
            "ve":{
               "trackingParams":"CEQQkKwCIhMIjbeSnM2xgQMV0lrgCh2t0AQ-"
            },
            "eventType":1,
            "clientData":{
               "adsClientData":{
                  "adClientDataEntry":{
                     "slotData":{
                        "type":"SLOT_TYPE_IN_PLAYER",
                        "controlFlowManagerLayer":"CONTROL_FLOW_MANAGER_LAYER_CORE",
                        "entryTriggerType":"TRIGGER_TYPE_LAYOUT_ID_ENTERED",
                        "debugData":{
                           "slotId":"RviyELkxejtZQxJX",
                           "slotEntryTriggerData":{
                              "type":"TRIGGER_TYPE_LAYOUT_ID_ENTERED",
                              "triggerSourceData":{
                                 "associatedLayoutId":"Dfplaq7bsu7NA5qN"
                              }
                           },
                           "fulfillmentTriggerData":[
                              {
                                 "type":"TRIGGER_TYPE_SLOT_ID_ENTERED",
                                 "triggerSourceData":{
                                    "associatedSlotId":"RviyELkxejtZQxJX"
                                 }
                              }
                           ],
                           "expirationTriggerData":[
                              {
                                 "type":"TRIGGER_TYPE_ON_NEW_PLAYBACK_AFTER_CONTENT_VIDEO_ID"
                              },
                              {
                                 "type":"TRIGGER_TYPE_SLOT_ID_EXITED",
                                 "triggerSourceData":{
                                    "associatedSlotId":"RviyELkxejtZQxJX"
                                 }
                              }
                           ]
                        }
                     },
                     "layoutData":{
                        "type":"LAYOUT_TYPE_MEDIA_LAYOUT_PLAYER_OVERLAY",
                        "controlFlowManagerLayer":"CONTROL_FLOW_MANAGER_LAYER_CORE",
                        "debugData":{
                           "layoutId":"ywBBjxutchYq5Xz1"
                        }
                     }
                  }
               }
            }
         },
         "context":{
            "lastActivityMs":"557"
         }
      }
   ],
   "serializedClientEventId":{
      "serializedEventId":"9ekGZdzSCcux6dsP-e6PqAk",
      "clientCounter":"19839"
   }
}
kattstof commented 1 year ago

I could be wrong but isn't this the automation detection that was bypassed in undetected_chromedriver v3.4.5? it could just be that this uses a deprecated and hence detectable version (3.2 vs the current 3.5) my recommendation of switching user agents with chrome properly causes it to keep some views but delete others, but firefox geckodriver seems to keep all of them so far. Maybe its a case of needing to update undetected_chromedriver? i'll try in the next few days when i get time and update with anything i find. cheers!

JijaProGamer commented 1 year ago

I could be wrong but isn't this the automation detection that was bypassed in undetected_chromedriver v3.4.5? it could just be that this uses a deprecated and hence detectable version (3.2 vs the current 3.5) my recommendation of switching user agents with chrome properly causes it to keep some views but delete others, but firefox geckodriver seems to keep all of them so far. Maybe its a case of needing to update undetected_chromedriver? i'll try in the next few days when i get time and update with anything i find. cheers!

No...?

This isn't even using chrome.

This is standard Firefox. These are just the network requests google sends to fingerprint the user.

kattstof commented 1 year ago

No, i know it's using firefox But these mitigation techniques are what borked undetected_chromedriver a few months ago and caused the need for a complete overhall i'm pretty sure.

JijaProGamer commented 1 year ago

No, i know it's using firefox But these mitigation techniques are what borked undetected_chromedriver a few months ago and caused the need for a complete overhall i'm pretty sure.

But undetected_chromedriver has nothing to do with this.

This is just a compiled list of every POST request made by YouTube.

undetected_chromedriver does nothing to change the user agent, the window size, the CPUs and memory shown by the user agent, etc.

Even IF undetected_chromedriver was randomising the fingerprint, it would still have nothing to do with my list of POST requests

kattstof commented 1 year ago

most of my work is based on selenium (geckodriver) and requests ( shout out to everyone else helping with v3n0m ;) )

the reason i'm bring up undetected_chromedriver is not because of your POST request LOL I'm pointing out that the automation detection that these allude to borked undetected_chromedriver 3.2 firefox doesn't have this problem, undetected_chromedriver does.

JijaProGamer commented 1 year ago

most of my work is based on selenium (geckodriver) and requests ( shout out to everyone else helping with v3n0m ;) )

the reason i'm bring up undetected_chromedriver is not because of your POST request LOL I'm pointing out that the automation detection that these allude to borked undetected_chromedriver 3.2 firefox doesn't have this problem, undetected_chromedriver does.

So if your project has nothing to do with my post, why did you even comment?

kattstof commented 1 year ago

Are you dense? I'm stating that your post requests allude to mitigation techniques that have been bypassed in a newer version of undetected_chromedriver so my comment has everything to do with this thread. LOL

JijaProGamer commented 1 year ago

Are you dense? I'm stating that your post requests allude to mitigation techniques that have been bypassed in a newer version of undetected_chromedriver so my comment has everything to do with this thread. LOL

Please keep it profesionional.

And it seems like your project has nothing about this, which anti fingerprinting depends on: "undetected_chromedriver does nothing to change the user agent, the window size, the CPUs and memory shown by the user agent, etc."

The bot isn't detected because it's using a old chromedriver, it's because it doesn't even try to anti fingerprint.

kattstof commented 1 year ago

yes because you not understanding me is unprofessional lol let me dumb this down the use of the undetected_chromedriver package used in this script is used for bypassing known anti-bot detection mechanisms. catch my drift? so it is part of the solution if we insist on using chrome

JijaProGamer commented 1 year ago

yes because you not understanding me is unprofessional lol let me dumb this down the use of the undetected_chromedriver package used in this script is used for bypassing known anti-bot detection mechanisms. catch my drift? so it is part of the solution if we insist on using chrome

That part of the solution is useless. Undetected_chromedriver doesn't live up to it's name, considering it gets detected every 2 weeks, and you still have to fix a lot of stuff by yourself.

Firefox has everything done for you, using less resources, with a neater API. You just have to do the anti fingerprinter.

kattstof commented 1 year ago

Trust me i know, i've been repeatedly saying to use geckodriver on deaf ears to the point where i'm probably just going to port it to firefox myself and submit a PR. But, The use of chrome seems to be what mshawon would prefer -- so im trying to keep that in mind when discussing it.

"firefox would be better, but if chrome is insisted upon..." is my line of thinking currently