Fix content bugs in the medium importer

dglazkov / polymath

MIT License

132 stars 9 forks source link

Fix content bugs in the medium importer #66

Open jkomoros opened 1 year ago

jkomoros commented 1 year ago

The medium importer has a few weird things, like escape codes for quotes and ellipses because medium does fancy things with formatting. Those should be normalized.

This should likely run in the strip_emojis and run on all input.

jkomoros commented 1 year ago

[ ] em-dashes are represented by U+200a, then U+2014, then U+200a
[ ] apostrophes are represented by U+2019
[ ] U+00a0 shows up randomly sometimes?
[ ] Start and end double quotes are U+201c and U+201d respectively
[ ] Do similar deep links based on headings that are done for the googledoc importer

jkomoros commented 1 year ago