projectdiscovery / proxify

A versatile and portable proxy for capturing, manipulating, and replaying HTTP/HTTPS traffic on the go.
https://projectdiscovery.io
MIT License
2.71k stars 231 forks source link

proxify is missing some encodings #519

Open ehsandeep opened 7 months ago

ehsandeep commented 7 months ago

seems like proxify is missing some encodings ( + those are shown in jsonl but not yaml multidoc ) Screenshot 2024-03-12 at 9 58 34 PM

Steps to reproduce

Implementation Notes

Originally posted by @tarunKoyalwar in https://github.com/projectdiscovery/proxify/issues/438#issuecomment-1992664616

dogancanbakir commented 7 months ago

I'm unable to reproduce the issue. Here are the steps I followed:

$ go run . -of jsonl -sr

$ python3 -c 'import requests; proxies = {"http": "http://127.0.0.1:8888", "https": "http://127.0.0.1:8888"}; response = requests.get("https://github.com/projectdiscovery/proxify", proxies=proxies, verify=False); print(response.text)'

I've also tried browser but couldn't.

@tarunKoyalwar Could you please provide me with the repro steps?

tarunKoyalwar commented 7 months ago

looks like there is another issue related to max response read size

$  proxify -v

                       _ ___    
   ___  _______ __ __ (_) _/_ __
  / _ \/ __/ _ \\ \ // / _/ // /
 / .__/_/  \___/_\_\/_/_/ \_, / 
/_/                      /___/

        projectdiscovery.io

[INF] Current proxify version v0.0.15 (latest)
[INF] HTTP Proxy Listening on 127.0.0.1:8888
[INF] Saving proxify logs to proxify_logs.jsonl
[WRN] responseChain: Error while dumping response: error reading response body: could not read response body: http: request body too large
$ curl -v -x http://127.0.0.1:8888 https://github.githubassets.com/asses/chunk-ui_packages_cookie-consent-link-element_cookie-consent-link-element_ts-80fd8ee00406.js

cc: @dogancanbakir

tarunKoyalwar commented 7 months ago

also looks like some servers do send br encoded response body and this seems to be advertisied by firefox browser

      "Accept-Encoding": "gzip, deflate, br",

^ this seems to be the reason why this is reproducable when proxiying browser traffic and not when using curl

Screenshot 2024-03-13 at 5 35 38 PM

tarunKoyalwar commented 7 months ago
"response": {
    "header": {
      "Accept-Ranges": "bytes",
      "Access-Control-Allow-Origin": "*",
      "Age": "1196786",
      "Cache-Control": "public, max-age=31536000, immutable",
      "Content-Encoding": "br",
      "Content-Type": "application/javascript",
      "Cross-Origin-Resource-Policy": "cross-origin",
      "Date": "Wed, 13 Mar 2024 12:08:32 GMT",
      "Etag": "0x8DC348B9980FBA0",
      "Last-Modified": "Fri, 23 Feb 2024 16:22:16 GMT",
      "Server": "Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0",
      "Strict-Transport-Security": "max-age=31536000",
      "Vary": "Accept-Encoding",
      "Via": "1.1 varnish, 1.1 varnish",
      "X-Cache": "HIT, HIT",
      "X-Cache-Hits": "88, 15269",
      "X-Fastly-Request-Id": "3c72b7382202b93fec00c5c1e1f06972ff1eb20c",
      "X-Served-By": "cache-iad-kiad7000175-IAD, cache-hyd1100027-HYD"
    },
    "body": " j����\r!�s1�]\u0015C&�B�9\u000f��cl�\u0010��a�D�ϩ�\u0018\u0002�D?(\u0010VK)3�;e�R�\u000b�:fZ\u0019Cz&�s?�Xs��[��bB��٘J�\u001d¦���|G��ݩ\u0012�ѩ\u001f�?�������\u0013fl/�\u001fE�ƣ,��ށ3\r�/f^�\u0015��V�\u0004�SϬ��\b�d��n\u000511H�\u0003\t2w\u000b�\u0016Iw=�Q�\\�D\u0010�*?r�쎲�\u0007���R\u001e\u0015\u0019\u001b�]�\u0005ƨm\u0019ļ��Ggq\u0017�q��6�\u0007wA|�_7��{��6fߒX�u/k\b3�|a��\u0012r�\u001b��9h#ю�\u0006���H}R��dIo��,\u001a��\\��\u001f�\u0003�\u0001��)�D17\u0000(�\u0017fɚ����Y\u0011*��/�[�\u007f�D�e�\u001a����[eY\u001a�\f��d��!�8�:���\u0005\u007f���>b\rHz�h�J����t�jcN\u0019p��xA�ns\u001c_pr\\�\"��\"oR\u0010O���)f��[�b\u001a%�M��^z��t\u0018�\u001e�\u001e��\u0018�IO\u0013%ԧZ�V�x��K��:L�.K �p\u0013��-�Km�۵Ph\u0013�/q\u001b�\u0006������1\u0007-4E\u0016�\u001b�lі��Q�b�<�\b�\u0000��Ҥ���z\u0006��$�EVa��9\u0011(ʭ\u001a\u0017�\r\u0018�\b���\u0002Ơ� \u0011\u0019\b\u0004�$d���9�\f�[���*7�OT|��F�ꈵW�v\u00048��Չ�⢾�3�\u0019<n\u0014cb�3+�B���\u0001���8��r�&(�*\u0018�?�\f\u001a���A�/\u0018\u001a���\u001a�\u0006ђ_c���IR\u0019]������N����\u001dc�V\u0007�\u0019������wo�)r'�\u0018�1[��b\bI�����\u0013h���\n�a���+�G���A�\b�l\u000e)%|\u000f>�\u000b�i��\"t�#�6\u0006����\u0002O��i\u001b(t�}�Z�\u0017D��\u001e��Wя�\nB\u0000\u0011s]|���v�Ui�Ԫ�\u001a�r�]�^\u0010�ҼiG����v\u001e� \u0010�w�{7y\u0012\u0011��5{(1�y�\n�m�X�Z\n����\u001e��ď�ӟz�\u0004Qmt\u0002�aa—y�1Fo���\u001ds�1\u0010Y॔\u0019�A�<K\u0006C\u0018�:\u0003\u00057�u\u0016���������+1N\u000fY�C0�?\u007f�����\u000ejR�|<\u0002�\u001au�j\u000e�+,`�[��e`I���JS T4\u00027�5}��m+ �mkLg�5ᚕI<\u0006~�7����ӓ���<\u0016׿�Hh����\u0015��M���H���R��{��\r6\u0012���KU���`{�\u000f�vncы\b\u001e\u001b'<(�\u0003��\u0018\u0003�\u0019 �\u001ct�������x\u0006�I&\u0005�n��IQ?� \u0002O��cb\u001e�D�/\u0017s\u0002Jb�\u007f\u0000�Ě{��K�yx\u0018\u000f\u000f���1jI���\u0000�y\u0000f���d\u0018\u0014W\u0013E\u0010��i���v�hQa�7p.�B`�[Z?�Y�\u0010\u001dJ�<��I��\u001b����Z\u0012)c\u0010]T�ഖ�\u001d\u000bpE\u0004�w���L�y�\u0012\u001a��� \u0002�X\b��Y�ʷ����X8�T�-8;g!�7K.\n̈m�\r\u000b��2�\u0006�Q\rd�\u0005����\u0004�>�9��b\rY\u001b��=��}�:�\u0010����~�l �RF��cN}�����\u0013��р˄0�]�\u0005��o/��n���]\f�\fV�psq%R\u0006�kx���\u000f���������_�J�\u0018|��Q��Ao���\u0000\u0019\u0005.�1�ڄNZ���\b�$������\u001d2���E玝�\u00057tH��\u007fh%��\u00064c}��u\"�N%\u007fd�8܈\u001c�i���\u0018�7ֈ�$��X�F\u0007\u00181dZ\u0014�œ\u0019���(c]�\\q���@J{x�.K\u000e�=/�؛P7�r�!��\u0016�_dp\u0015L�Ę��x�W��0�Gz��\u0004\u0004\u0003�F�g����ZT��\n��zC\u001e<\u001b�L�PA�\u0016��\u001d�\u0005\f=����\u0001LA3�$͂\u0000(�BG��4�\u001f��u�u�g�U:Q�\u0013�\n�H�cN�A%?��jI?�\u0014��\u0012y�z�8?<�ߗ\u0002�<٘�f{=l��ݦq����c!s�\u0012�˘�ǜ\"Ϩ\u001f���ja\n�`\u000b9�0��?1�6��.���s�\u0001��u�S��7t04\tv�\u001e5�ن�<?�\u000f0�������\u0011�Ay��QdPd����xʷ��IY�H\u0007�30�>�\u0001K�!��f̍�\r?�\u000eo��c��j\u000f�b\u0000�\u000b\u001e\u0018��\"�\u001e���7�\rT�3[œ����3����hI�ˏ<��?�y��)8{�h2�Zt\u000f�WZ��\u0003�j�_x\r����nLK�\u001bW@�&KU�P�j[V��+�!`\\\\���f��\u0007�1N�^�=�̱\u0010o���\u0011,9��LM�4�,����:`�&����\u0000HT��7���\\H*)������VHu�\u001f��;\u0005P��b���Op�'GC�s\\T�M�\t0'HY7E�V\u0006��}6\u001a?-|���K\t�ֲ�/0���ÅN��ˆ<��ҡj�\no\u001a7��|̩4\u0010�{ֽ)�<<�\b�����\u0016���q��^�����\u000b�B�67�MW?�\u0006���Ww�\\郯��ѿw1�Z�a�M�3:'pQ��\u001f�����]G�\u0004�u\u0011ܛ8\u000b�x\u0019��4k1�f+FY$9\u0013��\fmDO�ۄќ?B�� �)I�\f\u00181�B\u0005�\t!.ԣ\u000fr2\u001e�{����jvx�p%'���1Y<\u000b�\u0015a������@����׏�V\u0005MJ�ֲ�\u0003\b}-� ���S���\u0005�\u0012.j�v��e���e���V\u0014�xL�m\u0013Qx��\u001dt�\u0007��H�%]Vk�\"\u001e\u000b1'(���M\u0014#h�bvx�J)�\u0003�r�Th�V�\u001bJ��؄�ul��h\r��v��\"6Қ��mWo��\b��V��\b�*�ꍨH�u\u0016I�\u0017�D��$)��~��WK)��k�O3��3\u0011L��$|,J\u0007L���g?�g�\u0007�7S\u0010��$��.c1��\u0010���\u000b)e9��\u0010\u0010�a7����ܐ}.\u0018\u0014��[��,��\u0001:���\fL~n\t_Q�Uj��\u001c_pТ��Y+_j�<\t�\u001c�*{-�km�Y\f�\u0014�ۙ�A���D���c�o\u0014�\u0000�Fm&��� ���l��a6\u001eͷd\u0010�ƾB�����\u0012r\u001b�l\u000f��\u0012x%�M3���M8��������S\u0006[g<�����V�S���� �ߩ(�\u0011\u0010���\tL*�������F{+s݂?BgJ\u0013Qß��Ȝ��K�Ā�z�QcT��39�b�����j�'*����\u0013\u0011[{�?zf(��[���l�*��m�F-�ϭl;*_b����!*��+��姇���y2W�ᡧ��\\ch��$�'����V�O&&�h2�8\u0015h�Q��\u001d� ;�\u0011\u0004�b�<g�29�p\"��\u0006\u0007\u00039[��@z���2�����u\u0010�0\u007f<w�|v���\u0010]\u0012*�!C��r���Hjr�7�@\t�%y&�AJ�W\u0011�\u001aC/W3���U�4�Z\u0019z6�q�M���h�H�5��V���M��I�7 ~N��\u0002�9H�d\u001fn�5��\fp�� /I�Yc\u0002Y��Bع��\u0006Y)Oq.\u0005��,^���ȸ�:\u0016K\u0017��\u001c�yx��\u0007Z�\u001f%\u0010�,M]dž���f'H��sL�͙j�\u0007�_��V��w��:B���w�AQa�~O�\u0002�\u0010@��ú\u0006�ϔ���4m�Q� �\u0011���X,q\u001bM�����|�/凎�\u001c��I�\b�\u0018�)�L�@��\n��]���M�J��\u007f�ؒs,B�A16�\u001b�\u001b\u0015�\u0019�X�\u001fj0&��\u0015Tќ���\\��W�\u0016�U�\u0003(�\u0006\u007fb��#>\u0015\u000bz�\u0015O���i\u0011ι���\u0001�9\u0016I�5H4ر�\u00114!\u0018s4���(��ss���3g�w&�{�\u0018H���ó�\u0005���\u0018\u0014f�BȺ��.�8W�n�H]��\u0003\u0016\u0017%sC\u0011�&;Dz�O\u0019U�\u0007�8�\u0002��(�!������E\u000e�����y�D��\u0002\u007f<�.�q\u0012+2q��\u0013�bb��\u0013\u0002\u001b�a09\"�~��/���~����.��\u0006�j\u001dH\u001f\t\u001f�?C�ًz�T��\u0013�������$��g�I5������l�o2��h�����Eq��c�;�Z5"
dogancanbakir commented 7 months ago

depends on https://github.com/projectdiscovery/utils/issues/370