Unable to compile IETF document

[manuelfuenmayor]

In relation to https://github.com/metanorma/rfc-asciidoc-rfc/issues/26

I can't get to compile on IETF flavor, not even a simple document.

It seems like I get a different error message every time a compile. But I'd say the most recurrent one is:

$ bundle exec metanorma -t ietf -x html draft-ribose-asciirfc.adoc
...
RFC XML: Line 000055:20 element "middle" incomplete; missing required element "section"
Traceback (most recent call last):
  File "c:\program files\libreoffice\program\python-core-3.5.5\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\program files\libreoffice\program\python-core-3.5.5\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Manuel\AppData\Roaming\Python\Python35\Scripts\xml2rfc.exe\__main__.py", line 4, in <module>
    if sys.argv[0].endswith("__main__.py"):
  File "C:\Users\Manuel\AppData\Roaming\Python\Python35\site-packages\xml2rfc\__init__.py", line 14, in <module>
    from xml2rfc.parser import  XmlRfcError, CachingResolver, XmlRfcParser, XmlRfc
  File "C:\Users\Manuel\AppData\Roaming\Python\Python35\site-packages\xml2rfc\parser.py", line 20, in <module>
    from xml2rfc.writers import base
  File "C:\Users\Manuel\AppData\Roaming\Python\Python35\site-packages\xml2rfc\writers\__init__.py", line 2, in <module>
    from xml2rfc.writers.base import RfcWriterError
  File "C:\Users\Manuel\AppData\Roaming\Python\Python35\site-packages\xml2rfc\writers\base.py", line 30, in <module>
    from xml2rfc.util.unicode import ( punctuation, unicode_replacements, unicode_content_tags, bare_unicode_tags,
  File "C:\Users\Manuel\AppData\Roaming\Python\Python35\site-packages\xml2rfc\util\unicode.py", line 248, in <module>
    punctuation_re = re.compile(r'[%s]'%''.join(list(punctuation.keys())))
  File "c:\program files\libreoffice\program\python-core-3.5.5\lib\re.py", line 224, in compile
    return _compile(pattern, flags)
  File "c:\program files\libreoffice\program\python-core-3.5.5\lib\re.py", line 293, in _compile
    p = sre_compile.compile(pattern, flags)
  File "c:\program files\libreoffice\program\python-core-3.5.5\lib\sre_compile.py", line 536, in compile
    p = sre_parse.parse(p, flags)
  File "c:\program files\libreoffice\program\python-core-3.5.5\lib\sre_parse.py", line 829, in parse
    p = _parse_sub(source, pattern, 0)
  File "c:\program files\libreoffice\program\python-core-3.5.5\lib\sre_parse.py", line 437, in _parse_sub
    itemsappend(_parse(source, state, nested + 1))
  File "c:\program files\libreoffice\program\python-core-3.5.5\lib\sre_parse.py", line 575, in _parse
    raise source.error(msg, len(this) + 1 + len(that))
sre_constants.error: bad character range \u2002-▒ at position 4

This is the document I've been trying to build: rfc-asciidoc-rfc.zip

[opoudjis]

If I include include::sections/00-abstract.adoc[] and include::sections/06-sections.adoc[], I get it compiling OK on my Mac, so this seems to be a PC issue. The character range error does seem to be a python issue rather than a metanorma issue.

@CAMOBAP, any chance you could look at this?

[manuelfuenmayor]

I asked Nermina to compile the document as well, and she is also struggling with it. Maybe a Windows issue?

[CAMOBAP]

@manuel489 @anermina to be on the same page

1. how usually you call metanorma on windows:

   • docker
   • PowerShell
   • cmd
   • other shell?

2. could you please confirm that issue exists (or not) for docker?

   • docker run -v %cd%:/metanorma -v %USERPROFILE%\.fontist\fonts:/config/fonts -w /metanorma -it metanorma/mn metanorma -t ietf -x html draft-ribose-asciirfc.adoc --agree-to-terms - for cmd
   • docker run -v ${pwd.Path}:/metanorma -v ${env:USERPROFILE}\.fontist\fonts:/config/fonts -w /metanorma -it metanorma/mn metanorma -t ietf -x html draft-ribose-asciirfc.adoc --agree-to-terms - for PowerShell

3. without docker. can you confirm that execution of chcp 65001 command before bundle exec metanorma -t ietf -x html draft-ribose-asciirfc.adoc helps to avoid this issue?

[manuelfuenmayor]

@CAMOBAP to answer your queries:

how usually you call metanorma on windows:

I use mostly git bash to compile metanorma documents.

could you please confirm that issue exists (or not) for docker?

I use docker in ubuntu (virtual machine). And I've tried to compile there but I can't get the latest version of metanorma-ietf by using bundle update (I'm stuck in version 2.1.3).

without docker. can you confirm that execution of chcp 65001 command before bundle exec metanorma -t ietf -x html draft-ribose-asciirfc.adoc helps to avoid this issue?

I've changed the code page to 65001 on Powershell. The issue persists.

In addition, I've also tried to compile in Ubuntu... I'm getting issues as well:

...
RFC XML: xref target RFC7990 does not exist in the document
RFC XML: xref target RFC7991 does not exist in the document
RFC XML: xref target RFC7749 does not exist in the document
RFC XML: xref target RFC7991 does not exist in the document
RFC XML: xref target AsciiDoc does not exist in the document
RFC XML: xref target Asciidoctor does not exist in the document
RFC XML: xref target AsciiDoc does not exist in the document
RFC XML: xref target Asciidoctor does not exist in the document
RFC XML: xref target RFC7764 does not exist in the document
RFC XML: xref target asciidoctor_bibliography does not exist in the document
Cannot continue processing
No such file: draft-ribose-asciirfc.rfc.xml

This is weird...

[manuelfuenmayor]

The file I've been trying to build is: rfc-asciidoc-rfc.zip (from https://github.com/metanorma/rfc-asciidoc-rfc/issues/26)

[opoudjis]

Unassigning myself, as this is a PC issue. @CAMOBAP if there is something I need to do here, let me know

[CAMOBAP]

In progress

[CAMOBAP]

Tested on VM and Actual OS (via Bootcamp)

@manuel489 BTW what is the language selected in your OS? English or some other?

[manuelfuenmayor]

@CAMOBAP English (Sometimes I switch to Spanish, but most of the time I use English.)

[CAMOBAP]

Regarding No such file: draft-ribose-asciirfc.rfc.xml AFAIK it's known issue

I have tested git bash also, the same cannot reproduce the original issue.

@manuel489 I need more info from your side

• please try to compile in cmd or powersehll also
• also please post here output for env, locale -i, locale -u, locale -s commands from your Git Bash

[manuelfuenmayor]

Hi @CAMOBAP, Attending to your query, this zip file contains the requested info in both git bash and powershell: compilation-info-manuel.zip

Something weird happened, when I compile directly on rfc-asciidoctor-rfc, I get the already known No such file: draft-ribose-asciirfc.rfc.xml message, but if I compile on a separate folder (like the zip file I uploaded several comments above), I get a different message. The files I just shared reflect this "new" message.

[CAMOBAP]

Thanks, @manuel489 may I also ask you to attach generated *rfc..xml files here

Short summary. The issue happens on the python side. Also, I see that there is python from Libreoffice used for some reason. So I need also to execute:

• where python
• python -c "import sysconfig;print('{}'.format('\n'.join(['{} = {}'.format(v, sysconfig.get_config_var(v)) for v in sorted(sysconfig.get_config_vars(), key=lambda s: s.lower())])))"

Maybe this distribution of python has been build with specific flags

[manuelfuenmayor]

@CAMOBAP Ok, I attach generated *rfc..xml: rfc-asciidoc-rfc_compilation.zip

As per the commands, where python output in git bash (in powershell shows nothing):

$ where python
C:\Program Files\LibreOffice\program\python.exe
C:\Users\Manuel\AppData\Local\Microsoft\WindowsApps\python.exe

python -c "import sysconfig;... command in git bash (same output for powershell):

$ python -c "import sysconfig;print('{}'.format('\n'.join(['{} = {}'.format(v, sysconfig.get_config_var(v)) for v in sorted(sysconfig.get_config_vars(), key=lambda s: s.lower())])))"
abiflags =
base = C:\Program Files\LibreOffice\program\python-core-3.5.5
BINDIR = C:\Program Files\LibreOffice\program\python-core-3.5.5\bin
BINLIBDEST = C:\Program Files\LibreOffice\program\python-core-3.5.5\Lib
EXE = .exe
exec_prefix = C:\Program Files\LibreOffice\program\python-core-3.5.5
EXT_SUFFIX = .pyd
INCLUDEPY = C:\Program Files\LibreOffice\program\python-core-3.5.5\Include
installed_base = C:\Program Files\LibreOffice\program\python-core-3.5.5
installed_platbase = C:\Program Files\LibreOffice\program\python-core-3.5.5
LIBDEST = C:\Program Files\LibreOffice\program\python-core-3.5.5\Lib
platbase = C:\Program Files\LibreOffice\program\python-core-3.5.5
prefix = C:\Program Files\LibreOffice\program\python-core-3.5.5
projectbase = C:\Program Files\LibreOffice\program\python-core-3.5.5\bin
py_version = 3.5.5
py_version_nodot = 35
py_version_short = 3.5
SO = .pyd
srcdir = C:\Program Files\LibreOffice\program\python-core-3.5.5\bin
userbase = C:\Users\Manuel\AppData\Roaming\Python
VERSION = 35

[CAMOBAP]

@manuel489 thanks you a lot for your support, because of you I have a lot of data to analyze but still not enough to propose a fix so could you please also do (only in git bash):

1. run command /c/Users/Manuel/AppData/Local/Microsoft/WindowsApps/python.exe --version
2. run command env PATH="/c/Users/Manuel/AppData/Local/Microsoft/WindowsApps/:$PATH" bundle exec metanorma -t ietf -x html draft-ribose-asciirfc.adoc
3. run command xml2rfc --version if xml2rfc version less them 3.5 please update it with pip install xml2rfc --upgrade

Also, I have found similar tickets on xml2rfc https://trac.tools.ietf.org/misc/outcomes/ticket/3

[manuelfuenmayor]

@CAMOBAP Ok, these are my results:

1. I get "Permission denied":

   $ /c/Users/Manuel/AppData/Local/Microsoft/WindowsApps/python.exe --version
   bash: /c/Users/Manuel/AppData/Local/Microsoft/WindowsApps/python.exe: Permission denied

I find this a little bit strange because if I execute ls -l on that folder:

$ ls /c/Users/Manuel/AppData/Local/Microsoft/WindowsApps/ -l
total 0
drwxr-xr-x 1 Manuel 197121 0 Jun  9 17:12 Backup/
drwxr-xr-x 1 Manuel 197121 0 Oct  2  2020 Microsoft.DesktopAppInstaller_8wekyb3d8bbwe/
drwxr-xr-x 1 Manuel 197121 0 Jun  9 17:12 Microsoft.MicrosoftEdge_8wekyb3d8bbwe/
-rwxr-xr-x 1 Manuel 197121 0 Apr 15 17:17 MicrosoftEdge.exe*
-rwxr-xr-x 1 Manuel 197121 0 Oct  2  2020 python.exe*
-rwxr-xr-x 1 Manuel 197121 0 Oct  2  2020 python3.7.exe*
-rwxr-xr-x 1 Manuel 197121 0 Oct  2  2020 python3.exe*

It says that I have enough permissions on the python.exe file.

2. Basically, it throws me the same error message: compilation-output-git_bash.txt

3. I was using an outdated version of xml2rfc. Now after executing the update, I have 3.9.1 version.

   $ xml2rfc --version
   xml2rfc 3.9.1

   I still get the same compilation error though.

[CAMOBAP]

@manuel489 but the update of xml2rfc didn't help with the original issue, right?

[manuelfuenmayor]

@manuel489 but the update of xml2rfc didn't help with the original issue, right?

No, it didn't.

[CAMOBAP]

I think as a temporal solution Python from chocolatey may help

1. choco install -y python
2. refreshenv
3. try to compile again

[manuelfuenmayor]

@CAMOBAP I did what you proposed, and still get the same error message.

[manuelfuenmayor]

Additional info:

$ where python
C:\Python39\python.exe
C:\Program Files\LibreOffice\program\python.exe
C:\Users\Manuel\AppData\Local\Microsoft\WindowsApps\python.exe

$ python -c "import sysconfig;print('{}'.format('\n'.join(['{} = {}'.format(v, sysconfig.get_config_var(v)) for v in sorted(sysconfig.get_config_vars(), key=lambda s: s.lower())])))"
<string>:1: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
abiflags =
base = C:\Python39
BINDIR = C:\Python39
BINLIBDEST = C:\Python39\Lib
EXE = .exe
exec_prefix = C:\Python39
EXT_SUFFIX = .cp39-win_amd64.pyd
INCLUDEPY = C:\Python39\Include
installed_base = C:\Python39
installed_platbase = C:\Python39
LIBDEST = C:\Python39\Lib
platbase = C:\Python39
platlibdir = lib
prefix = C:\Python39
projectbase = C:\Python39
py_version = 3.9.6
py_version_nodot = 39
py_version_short = 3.9
SO = .cp39-win_amd64.pyd
srcdir = C:\Python39
TZPATH =
userbase = C:\Users\Manuel\AppData\Roaming\Python
VERSION = 39

[Intelligent2013]

Sorry for intervention, but looks like a few versions of Python installed on Win and/or parts from the different version are using.

C:\Python39\python.exe C:\Program Files\LibreOffice\program\python.exe C:\Users\Manuel\AppData\Local\Microsoft\WindowsApps\python.exe

@manuel489 could you check some things in Windows:

1. open cmd (not powershell, not git bash, just Start->cmd) and run:
   • C:\Python39\python.exe --version
   • C:\Program Files\LibreOffice\program\python.exe --version
   • C:\Users\Manuel\AppData\Local\Microsoft\WindowsApps\python.exe --version
2. open cmd and run:
   • C:\Python39\python.exe then run these commands

     >>> import sys
     >>> print(sys.maxunicode)

   • run C:\Program Files\LibreOffice\program\python.exe and C:\Users\Manuel\AppData\Local\Microsoft\WindowsApps\python.exe and repeat previous steps.

which values did you get 1114111 or 65535?

[manuelfuenmayor]

Hi @Intelligent2013, thank you for getting involved 😃

These are my results from cmd:

1. C:\Users\Manuel>C:\Python39\python.exe --version
   Python 3.9.6

C:\Users\Manuel>C:\Program Files\LibreOffice\program\python.exe --version
'C:\Program' is not recognized as an internal or external command,
operable program or batch file.

C:\Users\Manuel>C:\Users\Manuel\AppData\Local\Microsoft\WindowsApps\python.exe --version
(Nothing shown.)

2. The value is 1114111:

   C:\Users\Manuel>C:\Python39\python.exe
   Python 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)] on win32
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import sys
   >>> print(sys.maxunicode)
   1114111
   >>>

Thanks!

[Intelligent2013]

C:\Users\Manuel>C:\Program Files\LibreOffice\program\python.exe --version 'C:\Program' is not recognized as an internal or external command, operable program or batch file.

@manuel489 please enclose in the double quotes: "C:\Program Files\LibreOffice\program\python.exe" --version

[manuelfuenmayor]

Thanks @Intelligent2013, here is the result:

C:\Users\Manuel>"C:\Program Files\LibreOffice\program\python.exe" --version
Python 3.5.5

[Intelligent2013]

@manuel489 thank you. Could you also run "C:\Program Files\LibreOffice\program\python.exe" and:

>>> import sys
>>> print(sys.maxunicode)

[manuelfuenmayor]

@Intelligent2013, ok:

C:\Users\Manuel>"C:\Program Files\LibreOffice\program\python.exe"
Python 3.5.5 (default, Jan 31 2019, 00:04:27) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.maxunicode)
1114111
>>>

[CAMOBAP]

After testing on @manuel489 's machine some more details:

1. issue doesn't reproduce on python 3.9, if it loads xml2rfc from python 3.9
2. also issue randomly doesn't reproduce even on 3.5.5 from LibreOffice

Suggestion is to upgrade LibreOffice

Next issue will be resolved in the scope of https://github.com/metanorma/rfc-asciidoc-rfc/issues/32

[CAMOBAP]

@manuel489 please let me know if upgrade will not work for you

[manuelfuenmayor]

@CAMOBAP I've updated LibreOffice to 7.0.6. Unfortunately, I still cannot build the document. This time, the last line of the message says "Cannot continue processing". This is the log: compilation_msg_after_upgrade.txt

[CAMOBAP]

@manuel489 because this is separate issue which I mentioned in my comment above.

I'm going to close this one because workaround for the original issue was found