lexiforest / curl_cffi

Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
https://curl-cffi.readthedocs.io/
MIT License
2.44k stars 263 forks source link

想问一下现在支持sse吗 #155

Closed bao-io closed 1 year ago

bao-io commented 1 year ago

我想以流的形式读区返回的请求,发现行不通

perklet commented 1 year ago

你是说 ServerSentEvents 么?用 stream 模式应该可以读的,试试测试版 0.5.10b4

bao-io commented 1 year ago

对,现在只有测试版支持吗,什么时候上正式呀

perklet commented 1 year ago

快了,你可以先试下

bao-io commented 1 year ago

好滴,谢谢大佬

bao-io commented 1 year ago

大佬,安装报错了,这个测试版本是需要什么环境版本吗

Collecting curl-cffi==0.5.10b4
  Using cached curl_cffi-0.5.10b4.tar.gz (34 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: cffi>=1.12.0 in ./venv/lib/python3.9/site-packages (from curl-cffi==0.5.10b4) (1.16.0)
Requirement already satisfied: pycparser in ./venv/lib/python3.9/site-packages (from cffi>=1.12.0->curl-cffi==0.5.10b4) (2.21)
Building wheels for collected packages: curl-cffi
  Building wheel for curl-cffi (pyproject.toml): started
  Building wheel for curl-cffi (pyproject.toml): finished with status 'error'
Failed to build curl-cffi

  error: subprocess-exited-with-error

  × Building wheel for curl-cffi (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [97 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-10.9-universal2-cpython-39
      creating build/lib.macosx-10.9-universal2-cpython-39/curl_cffi
      copying curl_cffi/build.py -> build/lib.macosx-10.9-universal2-cpython-39/curl_cffi
      copying curl_cffi/curl.py -> build/lib.macosx-10.9-universal2-cpython-39/curl_cffi
      copying curl_cffi/__init__.py -> build/lib.macosx-10.9-universal2-cpython-39/curl_cffi
      copying curl_cffi/aio.py -> build/lib.macosx-10.9-universal2-cpython-39/curl_cffi
      copying curl_cffi/const.py -> build/lib.macosx-10.9-universal2-cpython-39/curl_cffi
      creating build/lib.macosx-10.9-universal2-cpython-39/curl_cffi/requests
      copying curl_cffi/requests/cookies.py -> build/lib.macosx-10.9-universal2-cpython-39/curl_cffi/requests
      copying curl_cffi/requests/models.py -> build/lib.macosx-10.9-universal2-cpython-39/curl_cffi/requests
      copying curl_cffi/requests/session.py -> build/lib.macosx-10.9-universal2-cpython-39/curl_cffi/requests
      copying curl_cffi/requests/__init__.py -> build/lib.macosx-10.9-universal2-cpython-39/curl_cffi/requests
      copying curl_cffi/requests/errors.py -> build/lib.macosx-10.9-universal2-cpython-39/curl_cffi/requests
      copying curl_cffi/requests/headers.py -> build/lib.macosx-10.9-universal2-cpython-39/curl_cffi/requests
      running egg_info
      writing curl_cffi.egg-info/PKG-INFO
      writing dependency_links to curl_cffi.egg-info/dependency_links.txt
      writing requirements to curl_cffi.egg-info/requires.txt
      writing top-level names to curl_cffi.egg-info/top_level.txt
      reading manifest file 'curl_cffi.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no files found matching 'curl_cffi/cacert.pem'
      warning: no files found matching 'curl_cffi/_wrapper.*'
      warning: no files found matching 'curl_cffi/include/curl/*'
      adding license file 'LICENSE'
      writing manifest file 'curl_cffi.egg-info/SOURCES.txt'
      /private/var/folders/f9/gxv59n954t984qbz21w757dh0000gn/T/pip-build-env-sezcaafp/overlay/lib/python3.9/site-packages/setuptools/command/build_py.py:204: _Warning: Package 'curl_cffi.ffi' is absent from the `packages` configuration.
      !!

              ********************************************************************************
              ############################
              # Package would be ignored #
              ############################
              Python recognizes 'curl_cffi.ffi' as an importable package[^1],
              but it is absent from setuptools' `packages` configuration.

              This leads to an ambiguous overall configuration. If you want to distribute this
              package, please make sure that 'curl_cffi.ffi' is explicitly added
              to the `packages` configuration field.

              Alternatively, you can also rely on setuptools' discovery methods
              (for example by using `find_namespace_packages(...)`/`find_namespace:`
              instead of `find_packages(...)`/`find:`).

              You can read more about "package discovery" on setuptools documentation page:

              - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html

              If you don't want 'curl_cffi.ffi' to be distributed and are
              already explicitly excluding 'curl_cffi.ffi' via
              `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
              you can try to use `exclude_package_data`, or `include-package-data=False` in
              combination with a more fine grained `package-data` configuration.

              You can read more about "package data files" on setuptools documentation page:

              - https://setuptools.pypa.io/en/latest/userguide/datafiles.html

              [^1]: For Python, any directory (with suitable naming) can be imported,
                    even if it does not contain any `.py` files.
                    On the other hand, currently there is no concept of package data
                    directory, all directories are treated like packages.
              ********************************************************************************

      !!
        check.warn(importable)
      creating build/lib.macosx-10.9-universal2-cpython-39/curl_cffi/ffi
      copying curl_cffi/ffi/cdef.c -> build/lib.macosx-10.9-universal2-cpython-39/curl_cffi/ffi
      copying curl_cffi/ffi/shim.c -> build/lib.macosx-10.9-universal2-cpython-39/curl_cffi/ffi
      copying curl_cffi/ffi/shim.h -> build/lib.macosx-10.9-universal2-cpython-39/curl_cffi/ffi
      running build_ext
      generating cffi module 'build/temp.macosx-10.9-universal2-cpython-39/curl_cffi._wrapper.c'
      creating build/temp.macosx-10.9-universal2-cpython-39
      building 'curl_cffi._wrapper' extension
      creating build/temp.macosx-10.9-universal2-cpython-39/build
      creating build/temp.macosx-10.9-universal2-cpython-39/build/temp.macosx-10.9-universal2-cpython-39
      creating build/temp.macosx-10.9-universal2-cpython-39/curl_cffi
      creating build/temp.macosx-10.9-universal2-cpython-39/curl_cffi/ffi
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Headers -arch arm64 -arch x86_64 -Werror=implicit-function-declaration -Wno-error=unreachable-code -Icurl_cffi/include -Icurl_cffi/ffi -I/Users/baobao/PycharmProjects/flaskProject/venv/include -I/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/include/python3.9 -c build/temp.macosx-10.9-universal2-cpython-39/curl_cffi._wrapper.c -o build/temp.macosx-10.9-universal2-cpython-39/build/temp.macosx-10.9-universal2-cpython-39/curl_cffi._wrapper.o -Wno-implicit-function-declaration
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Headers -arch arm64 -arch x86_64 -Werror=implicit-function-declaration -Wno-error=unreachable-code -Icurl_cffi/include -Icurl_cffi/ffi -I/Users/baobao/PycharmProjects/flaskProject/venv/include -I/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/include/python3.9 -c curl_cffi/ffi/shim.c -o build/temp.macosx-10.9-universal2-cpython-39/curl_cffi/ffi/shim.o -Wno-implicit-function-declaration
      curl_cffi/ffi/shim.c:9:16: warning: unused variable 'opt_value' [-Wunused-variable]
          CURLoption opt_value = (CURLoption) option;
                     ^
      1 warning generated.
      curl_cffi/ffi/shim.c:9:16: warning: unused variable 'opt_value' [-Wunused-variable]
          CURLoption opt_value = (CURLoption) option;
                     ^
      1 warning generated.
      clang -bundle -undefined dynamic_lookup -arch arm64 -arch x86_64 -Wl,-headerpad,0x1000 build/temp.macosx-10.9-universal2-cpython-39/build/temp.macosx-10.9-universal2-cpython-39/curl_cffi._wrapper.o build/temp.macosx-10.9-universal2-cpython-39/curl_cffi/ffi/shim.o -L/usr/local/lib -lcurl-impersonate-chrome -o build/lib.macosx-10.9-universal2-cpython-39/curl_cffi/_wrapper.abi3.so
      ld: library 'curl-impersonate-chrome' not found
      clang: error: linker command failed with exit code 1 (use -v to see invocation)
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for curl-cffi
ERROR: Could not build wheels for curl-cffi, which is required to install pyproject.toml-based projects

[notice] A new release of pip is available: 23.2.1 -> 23.3.1
[notice] To update, run: pip install --upgrade pip
image
perklet commented 1 year ago

不好意思,M1 的包我没上传,你看看 0.5.10b3 行不行吧,或者换个 Intel 或者 Linux 的机器

bao-io commented 1 year ago

b3可以,谢谢大佬

bao-io commented 1 year ago

不知道是不是bug,如果我用content_callback的话,是可以正常接受sse返回的内容,但是由于content_callback是同步的,我不能使用这种方式,所以想给你看看这个报错是不是代码的问题。

image image image
perklet commented 1 year ago

你得 stream=True 才能 iter 啊

bao-io commented 1 year ago
image

但是我看你这里没有这个参数呀,难道我下的版本不对吗

bao-io commented 1 year ago

解决好啦,真的太感谢大佬,帮了大忙,还有最后一个问题,就是sse是一个长连接,这个timeout默认给30,为啥已经连接成功,到了30秒的时候还是会自动断开呀,不应该创建成功之后,这个timeout计时器就应该关掉的吗?

perklet commented 1 year ago

嗯,连上之后还断可能确实欠妥,你现在就先设置成 2^32 - 1 得了……

bao-io commented 1 year ago

哈哈哈哈,这个后期会改吗,你应该只是设置了一个定时器吧??,应该关掉定时器就可以了吧

perklet commented 1 year ago

会改的

bao-io commented 1 year ago

okk,期待,大佬加油

bao-io commented 1 year ago
image image image

想问问大佬,docker容器里面是下载不到这个测试版本吗

perklet commented 1 year ago

不会下载不到的,应该是你容器的网络问题

bao-io commented 1 year ago

嗷嗷好的