lexiforest / curl_cffi

Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
https://curl-cffi.readthedocs.io/
MIT License
2.54k stars 269 forks source link

Add Cookie to dict #231

Closed Ausli closed 8 months ago

perklet commented 10 months ago

Actually, you can do this:

c = dict(session.cookies)
Ausli commented 10 months ago

Yes, it's optional, and I thought it would be more convenient to add it, since the requests library has this function as well

perklet commented 10 months ago

OK, it makes sense to me.

Ausli commented 10 months ago

Is there a unit test template for the project, lend me for reference.

perklet commented 9 months ago

Sorry, there is no such templates, you can take this as a reference.

Ausli commented 9 months ago

I get the following error when I run the test_cookies.py file ModuleNotFoundError: No module named 'curl_cffi._wrapper' How should I get _wrapper.so

perklet commented 9 months ago

I get the following error when I run the test_cookies.py file ModuleNotFoundError: No module named 'curl_cffi._wrapper' How should I get _wrapper.so

pip install -e .
Ausli commented 9 months ago

G:\workarea\MyProject\pythonProject\curl_cffi\ffi\shim.h(5): fatal error C1083: �޷��򿪰����ļ�: ��curl/curl.h��: No such file or directory The compile path is messed up, see what's wrong with it I suspect the reason is that the program reads the file using GBK encoding, but the file is UTF8 encoded causing the problem

Ausli commented 9 months ago
(pythonProject) PS G:\workarea\MyProject\pythonProject\curl_cffi> pip install -e .
Obtaining file:///G:/workarea/MyProject/pythonProject/curl_cffi
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... error
  error: subprocess-exited-with-error

  × Getting requirements to build editable did not run successfully.
  │ exit code: 1
  ╰─> [42 lines of output]
      Traceback (most recent call last):
        File "G:\Env\installer\miniconda\envs\pythonProject\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
          main()
        File "G:\Env\installer\miniconda\envs\pythonProject\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "G:\Env\installer\miniconda\envs\pythonProject\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 132, in get_requires_for_build_editable
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\a5320\AppData\Local\Temp\pip-build-env-gdxz1ugr\overlay\Lib\site-packages\setuptools\build_meta.py", line 441, in get_requires_for_build_editable
          return self.get_requires_for_build_wheel(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\a5320\AppData\Local\Temp\pip-build-env-gdxz1ugr\overlay\Lib\site-packages\setuptools\build_meta.py", line 325, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\a5320\AppData\Local\Temp\pip-build-env-gdxz1ugr\overlay\Lib\site-packages\setuptools\build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "C:\Users\a5320\AppData\Local\Temp\pip-build-env-gdxz1ugr\overlay\Lib\site-packages\setuptools\build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 16, in <module>
        File "C:\Users\a5320\AppData\Local\Temp\pip-build-env-gdxz1ugr\overlay\Lib\site-packages\setuptools\__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\a5320\AppData\Local\Temp\pip-build-env-gdxz1ugr\overlay\Lib\site-packages\setuptools\_distutils\core.py", line 147, in setup
          _setup_distribution = dist = klass(attrs)
                                       ^^^^^^^^^^^^
        File "C:\Users\a5320\AppData\Local\Temp\pip-build-env-gdxz1ugr\overlay\Lib\site-packages\setuptools\dist.py", line 303, in __init__
          _Distribution.__init__(self, dist_attrs)
        File "C:\Users\a5320\AppData\Local\Temp\pip-build-env-gdxz1ugr\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 283, in __init__
          self.finalize_options()
        File "C:\Users\a5320\AppData\Local\Temp\pip-build-env-gdxz1ugr\overlay\Lib\site-packages\setuptools\dist.py", line 654, in finalize_options
          ep(self)
        File "C:\Users\a5320\AppData\Local\Temp\pip-build-env-gdxz1ugr\overlay\Lib\site-packages\setuptools\dist.py", line 674, in _finalize_setup_keywords
          ep.load()(self, ep.name, value)
        File "C:\Users\a5320\AppData\Local\Temp\pip-build-env-gdxz1ugr\overlay\Lib\site-packages\cffi\setuptools_ext.py", line 216, in cffi_modules
          add_cffi_module(dist, cffi_module)
        File "C:\Users\a5320\AppData\Local\Temp\pip-build-env-gdxz1ugr\overlay\Lib\site-packages\cffi\setuptools_ext.py", line 49, in add_cffi_module
          execfile(build_file_name, mod_vars)
        File "C:\Users\a5320\AppData\Local\Temp\pip-build-env-gdxz1ugr\overlay\Lib\site-packages\cffi\setuptools_ext.py", line 22, in execfile
          src = f.read()
                ^^^^^^^^
      UnicodeDecodeError: 'gbk' codec can't decode byte 0xa3 in position 1488: illegal multibyte sequence
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build editable did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

(curl) PS G:\workarea\MyProject\curl_cffi> pip install  .  
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Processing g:\workarea\myproject\curl_cffi
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: cffi>=1.12.0 in g:\env\installer\miniconda\envs\curl\lib\site-packages (from curl_cffi==0.6.0b9) (1.16.0)
Requirement already satisfied: certifi in g:\env\installer\miniconda\envs\curl\lib\site-packages (from curl_cffi==0.6.0b9) (2024.2.2)
Requirement already satisfied: pycparser in g:\env\installer\miniconda\envs\curl\lib\site-packages (from cffi>=1.12.0->curl_cffi==0.6.0b9) (2.21)
Building wheels for collected packages: curl_cffi
  Building wheel for curl_cffi (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for curl_cffi (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [46 lines of output]
      .so files alreay downloaded.
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build\lib.win-amd64-cpython-38
      creating build\lib.win-amd64-cpython-38\curl_cffi
      copying curl_cffi\aio.py -> build\lib.win-amd64-cpython-38\curl_cffi
      copying curl_cffi\const.py -> build\lib.win-amd64-cpython-38\curl_cffi
      copying curl_cffi\curl.py -> build\lib.win-amd64-cpython-38\curl_cffi
      copying curl_cffi\_asyncio_selector.py -> build\lib.win-amd64-cpython-38\curl_cffi
      copying curl_cffi\__init__.py -> build\lib.win-amd64-cpython-38\curl_cffi
      copying curl_cffi\__version__.py -> build\lib.win-amd64-cpython-38\curl_cffi
      creating build\lib.win-amd64-cpython-38\curl_cffi\requests
      copying curl_cffi\requests\cookies.py -> build\lib.win-amd64-cpython-38\curl_cffi\requests
      copying curl_cffi\requests\errors.py -> build\lib.win-amd64-cpython-38\curl_cffi\requests
      copying curl_cffi\requests\exceptions.py -> build\lib.win-amd64-cpython-38\curl_cffi\requests
      copying curl_cffi\requests\headers.py -> build\lib.win-amd64-cpython-38\curl_cffi\requests
      copying curl_cffi\requests\models.py -> build\lib.win-amd64-cpython-38\curl_cffi\requests
      copying curl_cffi\requests\session.py -> build\lib.win-amd64-cpython-38\curl_cffi\requests
      copying curl_cffi\requests\websockets.py -> build\lib.win-amd64-cpython-38\curl_cffi\requests
      copying curl_cffi\requests\__init__.py -> build\lib.win-amd64-cpython-38\curl_cffi\requests
      running egg_info
      writing curl_cffi.egg-info\PKG-INFO
      writing dependency_links to curl_cffi.egg-info\dependency_links.txt
      writing requirements to curl_cffi.egg-info\requires.txt
      writing top-level names to curl_cffi.egg-info\top_level.txt
      reading manifest file 'curl_cffi.egg-info\SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no files found matching 'include\curl\*'
      adding license file 'LICENSE'
      writing manifest file 'curl_cffi.egg-info\SOURCES.txt'
      copying curl_cffi\libcurl.dll -> build\lib.win-amd64-cpython-38\curl_cffi
      running build_ext
      generating cffi module 'build\\temp.win-amd64-cpython-38\\Release\\curl_cffi._wrapper.c'
      creating build\temp.win-amd64-cpython-38
      creating build\temp.win-amd64-cpython-38\Release
      building 'curl_cffi._wrapper' extension
      creating build\temp.win-amd64-cpython-38\Release\build
      creating build\temp.win-amd64-cpython-38\Release\build\temp.win-amd64-cpython-38
      creating build\temp.win-amd64-cpython-38\Release\build\temp.win-amd64-cpython-38\Release
      creating build\temp.win-amd64-cpython-38\Release\ffi
      D:\Software\IDE\Visual_Studio\IDE\Enterprise\VC\Tools\MSVC\14.38.33130\bin\HostX86\x64\cl.exe /c /nologo /O2 /W3 /GL /DNDEBUG /MD -Iinclude -Iffi -IG:\Env\installer\miniconda
\envs\curl\include -IG:\Env\installer\miniconda\envs\curl\Include -ID:\Software\IDE\Visual_Studio\IDE\Enterprise\VC\Tools\MSVC\14.38.33130\include -ID:\Software\IDE\Visual_Studio\I
DE\Enterprise\VC\Tools\MSVC\14.38.33130\ATLMFC\include -ID:\Software\IDE\Visual_Studio\IDE\Enterprise\VC\Auxiliary\VS\include "-ID:\Windows Kits\10\include\10.0.22621.0\ucrt" "-ID:
\Windows Kits\10\\include\10.0.22621.0\\um" "-ID:\Windows Kits\10\\include\10.0.22621.0\\shared" "-ID:\Windows Kits\10\\include\10.0.22621.0\\winrt" "-ID:\Windows Kits\10\\include\
10.0.22621.0\\cppwinrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" /Tcbuild\temp.win-amd64-cpython-38\Release\curl_cffi._wrapper.c /Fobuild\temp.win-amd64-cpython-38\Release\build\temp.win-amd64-cpython-38\Release\curl_cffi._wrapper.obj
      curl_cffi._wrapper.c
      ffi\shim.h(5): fatal error C1083: 无法打开包括文件: “curl/curl.h”: No such file or directory
      error: command 'D:\\Software\\IDE\\Visual_Studio\\IDE\\Enterprise\\VC\\Tools\\MSVC\\14.38.33130\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for curl_cffi
Failed to build curl_cffi
ERROR: Could not build wheels for curl_cffi, which is required to install pyproject.toml-based projects
Ausli commented 9 months ago

I can't compile the project, can't find curl/curl.h, switching to v0.6.0b9 version the problem still exists, can you compile the project properly?

perklet commented 9 months ago

You need to check out the Makefile and .github/workflows to see how the build process works.