search

hplgit / doconce

Lightweight markup language - document once, include anywhere
http://hplgit.github.io/doconce/doc/web/index.html
Other
312 stars 60 forks source link

Sphinx 1.8.4 - Build Process Error #176

Open mimeiners opened 5 years ago

mimeiners commented 5 years ago

Dear Doconce Users,

For some time the command 'doconce sphinx_dir theme=$theme $name' causes errors. I need to run it twice to obtain automate_sphinx.py in my work dir. It seems that sphinx-build has been changed and path variables behave different. Has someone experienced similar behaviour? Shall I track this inside the relevant doconce class?

Regards

Mirco Meiners

KGHustad commented 5 years ago

I have also noticed that sphinx hasn't been working as it should lately. DocOnce's integration with Sphinx is quite fragile, and can easily break when changes are made to Sphinx. I don't have the time to look into it now, but if you want to attempt to fix it, then that's great. Just remember to do the fix in a separate branch and create a pull request so that I can review it before it's merged.

mimeiners commented 5 years ago

Alright, I will have a look into the class and the sphinx-build process. I will open a sphinx branch and keep you informed about my fixing.

mimeiners commented 5 years ago

Dear Kristian,

I had a look into the Sphinx process and I was able to identify the relevant code segments. As you mentioned, Sphinx implementation in Doconce is quite fragile and any modification to sphinx-quickstart by its developers causes a fail in Doconce, since sphinx_dir is a system-call to sphinx-quickstart.

I tested sphinx_dir calls under Python 2.7 with Sphinx 1.8.5 and Python 3.7 with Sphinx 2.1.2. The questionary in sphinx-quickstart version 2.1.2 is much shorter than in previous Sphinx versions, hence tmp_sphinx_gen.sh has to be modified accordingly. All of this is done in bin/doconce from line 364 on.

How should we proceed, since this is a question of Python 2.7 to Python 3.x, too? Right? For Python 2.7 it is a minor fix, for Python 3.x it includes porting of automake_sphinx.py, too.

Summer greetings

Mirco

KGHustad commented 5 years ago

I took a quick look at the sphinx-quickstart source code: https://github.com/sphinx-doc/sphinx/blob/master/sphinx/cmd/quickstart.py

If we could provide all information via command line arguments to sphinx_dir, then that is probably a better solution than hardcoding answers to the questions asked by the prompt as we have seen that these can change between versions. Maybe we can't provide all the information via command line arguments, but we should least use them where possible, and then there might be a few things that have to be answered via the prompt.

Another approach would be to call directly into the Sphinx API. It seems like the function handling the interactive prompt simply adds keys to a dictionary, so we could probably populate that dictionary in DocOnce, and then call the generate function in the file linked above.

I won't have time to fix this myself anytime soon, but if you're willing, I think you could find a solution with one of these two approaches. As for Python 2/3 differences, I'm not too concerned about fixing this for Python 2, as Python 2 is nearing EOL.

mimeiners commented 5 years ago

Hallo Kristian,

Thanks for your quick comments.

Okay, I will give it a try to fix the Sphinx issue. I will review both solution paths and I'll keep you informed about my ongoing. As already agreed, I will use branch sphinx for testing.

Best regards

Mirco

Am 01.08.2019 um 18:58 schrieb Kristian Gregorius Hustad notifications@github.com:

I took a quick look at the sphinx-quickstart source code: https://github.com/sphinx-doc/sphinx/blob/master/sphinx/cmd/quickstart.py https://github.com/sphinx-doc/sphinx/blob/master/sphinx/cmd/quickstart.py If we could provide all information via command line arguments to sphinx_dir, then that is probably a better solution that hardcoding answers to the questions asked by the prompt as we have seen that these can change between versions. Maybe we can't provide all the information via command line arguments, but we should least use them where possible, and then there might be a few things that have to be answered via the prompt.

Another approach would be to call directly into the Sphinx API. It seems like the function handling the interactive prompt simply adds keys to a dictionary, so we could probably populate that dictionary in DocOnce, and then call the generate function in the file linked above.

I won't have time to fix this myself anytime soon, but if you're willing, I think you one of these two approaches. As for Python 2/3 differences, I'm not too concerned about fixing this for Python 2, as Python 2 is nearing EOL.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/hplgit/doconce/issues/176?email_source=notifications&email_token=ABSMECIHNELQSXK4TQB5G7TQCMIZ7A5CNFSM4GYKILN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3LHUEY#issuecomment-517372435, or mute the thread https://github.com/notifications/unsubscribe-auth/ABSMECP4OVH7XNF3P5S5LSDQCMIZ7ANCNFSM4GYKILNQ.

mimeiners commented 5 years ago

Dear Kristian,

currently I prepare my lecture slides for the coming winter term. I use this scenario to work on the sphinx issue in doconce.

As my lecture slides are in German as well as in English I got stucked due to unicode issues. This becomes unstructured, if I switch between python 2.7 and python 3.7; esp. module re (regex) in python 3.7 causes error messages during doconce calls format latex.

I am writing this to you to sync appropriately into your doconce workflow.

How can I best support your work and push modification into the repositorium? What is your approach to switch the code from python 2.7 to python 3.x? Working on Sphinx format results in rewriting some functions of doconce, e.g. automake_sphinx.py.

Best regards

Mirco

KGHustad commented 5 years ago

I'm not sure that I understand your questions exactly, but I will try my best to answer.

As my lecture slides are in German as well as in English I got stucked due to unicode issues. This becomes unstructured, if I switch between python 2.7 and python 3.7; esp. module re (regex) in python 3.7 causes error messages during doconce calls format latex.

Could you post the error message and traceback?

How can I best support your work and push modification into the repositorium?

Create a separate git branch, commit the changes there, and push that branch either to this repo (please don't push directly to master) or to your own fork, and then create a pull request

What is your approach to switch the code from python 2.7 to python 3.x?

We have used autotranslation tools previously, but they don't fix the regular expressions, which is the key issue in our case. My approach for the regular expressions have been to fix them by hand. Generally what needs to be done is to double-escape \ characters. '\\' is a string with a single backslash. '\\\\' is a string with a double backslash, which re will interpret as a single literal backslash.

mimeiners commented 5 years ago

I'm not sure that I understand your questions exactly, but I will try my best to answer.

You made a great job and answered quite well ;o)

As my lecture slides are in German as well as in English I got stucked due to unicode issues. This becomes unstructured, if I switch between python 2.7 and python 3.7; esp. module re (regex) in python 3.7 causes error messages during doconce calls format latex.

Could you post the error message and traceback?

I will do that.

How can I best support your work and push modification into the repositorium?

Create a separate git branch, commit the changes there, and push that branch either to this repo (please don't push directly to master) or to your own fork, and then create a pull request

Alright, made two braches, sphinx_v1 and sphinx_v2. sphinx_v1 is the branch to which I committed my latest modification dealing with utf-8 (unicode) for German language with python 2.7 and sphinx vers. 1.8.5.

Branch sphinx_v2 holds modification for python 3.7 and sphinx vers. 2.1.2

What is your approach to switch the code from python 2.7 to python 3.x?

We have used autotranslation tools previously, but they don't fix the regular expressions, which is the key issue in our case. My approach for the regular expressions have been to fix them by hand. Generally what needs to be done is to double-escape \ characters. '\\' is a string with a single backslash. '\\\\' is a string with a double backslash, which re will interpret as a single literal backslash.

Yep, this is what I meant when writing about errors during doconce call for format latex. The behaviour of module re is different in py 2.7 and py 3.7 which causes errors processing language German in doconce, I called it unicode errors.

Okay, I will post the error, but in the meantime you might have a look into my branches.

KGHustad commented 5 years ago

I had a look at your branches.

I saw you made some improvements to German language support, so I made a separate commit with these: https://github.com/hplgit/doconce/commit/a68daba272aef97001846c5511f7b793a54c539f

There is no need to specify the encoding of the .py files for Python 3. Also, I would rather not have """ Module description. """ strings at the top of the files.

Does sphinx work for you with the changes you have made in these branches?

mimeiners commented 5 years ago

__

I had a look at your branches.

I saw you made some improvements to German language support, so I made a separate commit with these: a68daba

Yep!

There is no need to specify the encoding of the .py files for Python 3. Also, I would rather not have """ Module description. """ strings at the top of the files.

Okay!

Does sphinx work for you with the changes you have made in these branches? Yep, with the modifications in branch sphinx_v1 it works with python 2.7 and sphinx.

If I run doconce with python 3.7, sphinx runs with a work-around in my make.sh:

system python2 automake_sphinx.py

In my next step, I will translate that file to python 3. Maybe it is no longer necessary, if I am able to link with a dictionary into sphinx-build (one of your ideas to optimise sphinx processing).

But doconce format pdflatex gives the following error with python 3.7:

Traceback (most recent call last): File "/opt/local/bin/doconce", line 1690, in bg_session = main() File "/opt/local/bin/doconce", line 1679, in main retval = eval(command + '()') File "", line 1, in File "/opt/local/bin/doconce", line 103, in format bg_session = doconce.doconce.format_driver() File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/doconce/doconce.py", line 5543, in format_driver out_filename, bg_session = file2file(filename_preprocessed, format, basename) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/doconce/doconce.py", line 4355, in file2file filestr, bg_session = doconce2format(filestr, format) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/doconce/doconce.py", line 4847, in doconce2format tex_blocks, format) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/doconce/latex.py", line 643, in latex_code pattern = re.compile(r'!b%s *\n', re.UNICODE) % admon TypeError: unsupported operand type(s) for %: 're.Pattern' and 'str'

KGHustad commented 5 years ago

This is the line that is failing https://github.com/hplgit/doconce/blob/a7fef2b2d50a90a6c2937a08d3ceac9135f72a3f/lib/doconce/latex.py#L642

From your traceback, I see that this line is set to

pattern = re.compile(r'!b%s *\n', re.UNICODE) % admon

in your local latex.py. I assume this is a change you made in attempt to fix the problem, but I cannot provide much help in debugging code that I cannot see. Could you check if converting this file to latex works with the latest version of DocOnce?

mimeiners commented 5 years ago

latest DocOnce version

exporting publish database papers.pub to papers.bib: Traceback (most recent call last): File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/sre_parse.py", line 1021, in parse_template this = chr(ESCAPES[this][1]) KeyError: '\e'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/local/bin/doconce", line 1690, in bg_session = main() File "/opt/local/bin/doconce", line 1679, in main retval = eval(command + '()') File "", line 1, in File "/opt/local/bin/doconce", line 103, in format bg_session = doconce.doconce.format_driver() File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/doconce/doconce.py", line 5543, in format_driver out_filename, bg_session = file2file(filename_preprocessed, format, basename) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/doconce/doconce.py", line 4355, in file2file filestr, bg_session = doconce2format(filestr, format) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/doconce/doconce.py", line 4847, in doconce2format tex_blocks, format) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/doconce/latex.py", line 664, in latex_code filestr = re.sub(pattern, r'\end{block}', filestr) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/re.py", line 192, in sub return _compile(pattern, flags).sub(repl, string, count) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/re.py", line 309, in _subx template = _compile_repl(template, pattern) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/re.py", line 300, in _compile_repl return sre_parse.parse_template(repl, pattern) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/sre_parse.py", line 1024, in parse_template raise s.error('bad escape %s' % this, len(this)) re.error: bad escape \e at position 0

mimeiners commented 5 years ago

I had a look at your branches.

I saw you made some improvements to German language support, so I made a separate commit with these: a68daba

There is no need to specify the encoding of the .py files for Python 3. Also, I would rather not have """ Module description. """ strings at the top of the files.

Does sphinx work for you with the changes you have made in these branches?

master branch does not work neither sphinx (py27) version 1.8.5 nor sphinx (py3) version 2.2.0

I committed and pushed necessary modifications into branches sphinx_v1 and sphinx_v2. It is just a quick and dirty work-around inside bin/doconce for different sphinx-quickbuild calls in version 1 and version 2, as I made for previous sphinx version with Hans-Petter. I am working on a more robust implementation as we discussed.

KGHustad commented 5 years ago

I think this should fix the error you posted with latex. https://github.com/hplgit/doconce/commit/9c8fc12a2f5a095f7f13eded972c1fab7ce3c693