I have a few other projects (eg, que) that pull down webpages from the command line and then do something else. Que's requests based stuff is pretty good in that it caches the page even if caching headers aren't set and things like that, but it would be really handy if I could use that caching and request stuff in other libs.
So I'm proposing merging que's code into a special command line class here that I can then use to make requests and cache their result in order to further manipulate it. The class should be able to take a string like firefox or chrome to switch from using requests to using the actual browser since I've run into issues where I tried to parse a page with que and wasn't able to because js didn't load the part I wanted, so it would be nice to have the option to switch to firefox or chrome and load the page that way
I have a few other projects (eg, que) that pull down webpages from the command line and then do something else. Que's requests based stuff is pretty good in that it caches the page even if caching headers aren't set and things like that, but it would be really handy if I could use that caching and request stuff in other libs.
So I'm proposing merging que's code into a special command line class here that I can then use to make requests and cache their result in order to further manipulate it. The class should be able to take a string like
firefoxorchrometo switch from using requests to using the actual browser since I've run into issues where I tried to parse a page with que and wasn't able to because js didn't load the part I wanted, so it would be nice to have the option to switch to firefox or chrome and load the page that way