search

openlibhums / pandoc_plugin

Plugin for janeway for automatic galley generation
GNU Affero General Public License v3.0
4 stars 1 forks source link

explicit encoding utf8 with open #11

Closed hachacha closed 5 years ago

hachacha commented 5 years ago

I was consistently having a problem with bs4 trying to read the file that was opened at this line as ascii on the sandbox server of for CMU's janeway instance. image Adding .encoding("utf-8") to the prettify() function was injecting \n through the document. Unfortunately, I was unable to recreate the same bug on my local but this line is just being more explicit.

ajrbyers commented 5 years ago

Can you confirm that the sandbox has the latest version of pan doc? This error occurs a lot with the older 1.x version...

Sent from my iPhone

On 4 Mar 2019, at 16:47, jon kirichacha notifications@github.com wrote:

I was consistently having a problem with bs4 trying to read the file that was opened at this line as ascii on the sandbox server of for CMU's janeway instance.

Adding .encoding("utf-8") to the prettify() function was injecting \n through the document. Unfortunately, I was unable to recreate the same bug on my local but this line is just being more explicit.

You can view, comment on, or merge this pull request online at:

https://github.com/ajrbyers/pandoc_plugin/pull/11

Commit Summary

explicit encoding utf8 with open File Changes

M views.py (2) Patch Links:

https://github.com/ajrbyers/pandoc_plugin/pull/11.patch https://github.com/ajrbyers/pandoc_plugin/pull/11.diff — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

hachacha commented 5 years ago

Yes, I did an exhaustive check. When the file is converted it's all fine, but when the stdout is being opened by bs4 it's reading as ascii for some reason. I get the error at bs4 not pandoc.... I checked the versions of bs4, python, and pandoc and they're all up-to-date. the sys.getdefaultencoding() also prints utf8 for python on the server.

$ pandoc -v
pandoc 2.5
$ sudo apt-get purge pandoc 
...Removing pandoc (2.5-1) ......
$ pandoc -v
The program 'pandoc' is currently not installed....
$ wget 'https://github.com/jgm/pandoc/releases/download/2.5/pandoc-2.5-1-amd64.deb'
$ sudo dpkg -i pandoc-2.5-1-amd64.deb
.......
$ pandoc -v
pandoc 2.5

After this I still get the error.

ajrbyers commented 5 years ago

Thanks for being so thorough. I wonder if we should put together a testing package of simple docs with different encoding and expected outcomes for this plugin?