madwort
commented
8 years ago Ok, I've been doing some digging into this, what's happening is there are errors in your HTML & this is tripping up both of the open-source HTML parsers written in PHP that I've tried. This is a very fiddly area - browsers tend to do a "best-guess" when confronted with malformed HTML & therefore often render something that looks ok, patching over any confusion under the hood, but programmatic parsers aren't often as highly developed.
I've been using the source for http://www.rodrigoconstanzo.com/2015/04/making-decisions-in-time/ as a test case, the problem that trips it up is on line 260 (when downloaded as a complete HTML file):
In the time between coming up with the system and creating the initial analyses and visualization
tools I found that my ability to improvise had improved. Outlining these <em>decision streams</em>
created a mental framework, outside of data analysis, that allowed me to critically think and talk about
my improvisation. This would materialize in a performative setting by allowing me to focus on the types
of decisions I was making, and when they should happen, in a kind of meta-formal capacity/way.</p>The </p> tag at the end is an erroneous extra tag that does not correspond to a <p> & causes the parsers I've tried to barf here.
The best thing would be if you were able to go through all the pages that need to be scraped & fix the errors - you can find where they are by using the W3C validator, eg. https://validator.w3.org/check?uri=http%3A%2F%2Fwww.rodrigoconstanzo.com%2F2015%2F04%2Fmaking-decisions-in-time%2F&charset=%28detect+automatically%29&doctype=Inline&group=0
You can ignore some of these errors (there's lots of errors on line 46 of that file caused by unescaped JavaScript, but fixing that looks like it'll mean editing one of your plugins, which is probably going to get annoying). The ones I'm interested firstly are the extra end tags, eg.
Line 186, Column 2172: end tag for element "p" which is not openDo you think that's possible...?
rconstanzo
most evident on the long blog posts by checking links/vids towards the bottom of page.
for example : http://johnlely.co.uk/ near the bottom of http://www.rodrigoconstanzo.com/2015/04/making-decisions-in-time/
or http://www.lcasserley.co.uk/ at the bottom of this : http://www.rodrigoconstanzo.com/2015/06/cut-glove/