Not requesting endpoints that are too large

[percepo]

Any URL could contain an arbitrary response size. While the limit is fuzzy, a 3 gb response should not be requested by wenum, as at some points risks of running out of ram become relevant. Its unlikely parsing such a big response makes enough sense to warrant that. A HEAD request should instead be logged for the user to then inspect that endpoint manually. But knowing an endpoint's response size can be tricky, and a method needs to be devised for that

[percepo]

Pycurl can throw error 28: Operation timed out after 40000 milliseconds with 409600 out of 1139893 bytes received. Maybe this is a hint towards the fact that pycurl knows how many total bytes are supposed to arrive. If its possible to utilize this in code before the response has arrived, it would be very easy to deal with.

[percepo]

https://curl.se/libcurl/c/CURLOPT_MAXFILESIZE.html can help with that. For a maximum, 200MB should be fine. Due to plugins, even big endpoints can be of interest. But an arbitrary limit has to be set, and most realistic, non-downloaded files should jump around that area. Potentially allow for CLI option to change this value. If a response is bigger than that, create a new HEAD request for this endpoint with the information that it was too big.

[percepo]

Turns out curl does nothing more but to inspect the Content-Length header. Also turns out that it looks like web servers, when sending a HEAD request, will put the Content-Length of a GET into its response. This means that curl, when it thinks the response is too big, will fail on both GET and HEAD, and cause somewhat buggy behavior when the user uses POSTs, I assume. The safest option seems to be to simply let responses that were too big display an errored response due to being too big. It looks less elegant to the user, but the user will still see the relevant information: That the response has not been processed due to the size having been too big. They then can manually request it for further inspection.