search

jakopako / goskyr

A configurable command-line web scraper written in go with auto configuration capability
GNU General Public License v3.0
32 stars 4 forks source link

Add debug flag #218

Closed jakopako closed 2 months ago

jakopako commented 1 year ago

debug output should among others show the http request and response (only headers?)

MarkJaroski commented 2 months ago

I'd like to be able to output the DOM, as seen by goquery. I've seen a couple of cases where the post js redered DOM appears to be different from what I see in the browser.

jakopako commented 2 months ago

Good point! I've had such cases too. Modified the source code temporarily to print the DOM which is of course ugly 😅 We'll have to think about how to do this exactly. In my experience the DOM often is way too big to print it to logs in an appealing way. Maybe write it to a file instead..

jakopako commented 2 months ago

Side note: I didn't have much time lately to work on this project. I'm hoping it'll get better but not sure 🤞

MarkJaroski commented 2 months ago

Good point! I've had such cases too. Modified the source code temporarily to print the DOM which is of course ugly 😅 We'll have to think about how to do this exactly. In my experience the DOM often is way too big to print it to logs in an appealing way. Maybe write it to a file instead..

Dumping it to an html file is probably the best.

MarkJaroski commented 2 months ago

Side note: I didn't have much time lately to work on this project. I'm hoping it'll get better but not sure 🤞

Don't worry about it at all. I completely understand. It's going to be like that for me the rest of the month and most of May.

jakopako commented 2 months ago

Added a few basic debugging logs. If the need arises, I'll add more. Moreover, if the -debug flag is set all html docs are written to files for inspection.