Closed ZoeLeBlanc closed 4 years ago
@ZoeLeBlanc Are you using python 2 or 3 ? Asking because I'm not seeing the same error you are reporting, although I did get an error for hitting the wayback machine api too many times in a row. 😆 (I'm using python3.6)
I think we should clean up the archive and rename the files with those weird characters. I've corrected a couple of variants with question marks where there was duplicate content but I think there are still a few more.
You were totally right @rlskoeser about my python version (Shakes fist at pipenv) and now everything is working. The only unicode error I got before was that topic with the apostrophe, so I can change that one in this PR (or make a new one). Thanks for helping me 👏
@ZoeLeBlanc would you mind making an issue on this repo with a list of the topics with punctuation in the url? I hope to clean them up at some point. I know we have the one table and I think a couple with question marks, have you run into any others?
nope! merging in now 👍
This PR was primarily to add code for extracting the text from posts so that we could do some analysis on them.
I updated the
content
field to behtml_content
, and then added code that striped the paragraph tags from that content and stored in thecontent
field.However, I was never ever able to successfully run the script because I kept getting encoding errors. It seems like this lesson
tei-in-oxygen-author-problem-with-tables’-labels-in-xsl-fo-1
has an apostrophe in it that prevents it from being handled by string formatting in python3. I tried removing it with an if statement (it's still in the script commented out) but even then I couldn't write to file and instead got this error message:@rlskoeser would definitely appreciate your help fixing this. When I rewrote the script to work with pandas I used this
encoding='utf-8-sig'
to get rid of the error, but now I'm wondering why the script isn't working on my computer. Thoughts?