Ge0rg3 / requests-ip-rotator

A Python library to utilize AWS API Gateway's large IP pool as a proxy to generate pseudo-infinite IPs for web scraping and brute forcing.
https://pypi.org/project/requests-ip-rotator/
GNU General Public License v3.0
1.36k stars 140 forks source link

Unable to download video #37

Closed mittster closed 2 years ago

mittster commented 2 years ago

If you try to view a video, the response has status code 500 with message {"message": "Internal server error"}

You can try it out with this sample:

if __name__ == '__main__':
    gateway = ApiGateway("https://video.xx.fbcdn.net", regions=["regionhere"], access_key_id=ACCESS_KEY_ID, access_key_secret=ACCESS_KEY_SECRET)
    gateway.start()

    session = requests.Session()
    session.mount("https://video.xx.fbcdn.net", gateway)

    response = session.get("https://video.xx.fbcdn.net/v/t66.36240-2/10000000_5162062970516874_3952172667405771162_n.mp4?_nc_cat=110&_nc_ht=video.fmnl8-2.fna&_nc_ohc=RuGOgev5TvEAX8FI4Hk&_nc_sid=985c63&ccb=1-7&efg=eyJybHIiOjIzNzQsInJsYSI6MTc5MSwidmVuY29kZV90YWciOiJvZXBfaGQifQ%3D%3D&oe=628E6A4A&oh=00_AT__3AG9wy4Vz9QqimNjwzLTspBsfs0E5-VyVRaLCqGAUw&rl=2374&vabr=1583")
    print(response.status_code) // this prints 500

    # Delete gateways
    gateway.shutdown()

Also tried with: -https://previews.customer.envatousercontent.com/h264-video-previews/34bc5a70-9db3-4ee9-821d-b4304d0c5dc5/33244341.mp4

Any idea whats going on? It seems that default aws gateway configuration does not support video.

Ge0rg3 commented 2 years ago

Hi! Thank you for raising this issue. I'll look into this shortly 👍

Ge0rg3 commented 2 years ago

Hi @mittseter, after some investigation it looks like this is the result of API Gateway having a hard response size limit of 10MB. However, in your case, this can be mitigated via the HTTP Range header. For example, you could iterate through requests starting with Range: bytes=0-9000000, then bytes=9000000-18000000 etc. Please let me know if this works for you? 😄

mittster commented 2 years ago

Should have came back to the post sooner :D I came to the same conclusion. Thanks