Closed efieleke-tausight closed 5 days ago
I was using version 2024.06.24, BTW
We are seeing that the old doc2Text is not stripping away these spaces between words, given the same file. So this appears to be a regression.
For reference, here is how we call into docwire. In this case, GetMappedExtension is returning "docx" (and the file really is a docx file).
auto iStream = std::make_shared<boost::iostreams::stream<boost::iostreams::array_source>>(buffer, bufferSize);
docwire::data_source dataSource(
docwire::seekable_stream_ptr { iStream },
docwire::file_extension("." + GetMappedExtension(extension)));
std::basic_stringstream<char> outStream;
std::move(dataSource) |
docwire::ParseDetectedFormat<docwire::OfficeFormatsParserProvider, docwire::MailParserProvider,
docwire::OcrParserProvider>(_parameters) |
docwire::PlainTextExporter() |
outStream;
It seems that the problematic spaces are encoded with a text tag with single space only:
<w:t> </w:t>
There is no "space=preserve" attribute so it's on the edge of the official OOXML standard I would say. It is possible that older versions treat it differently because there were some optimizations in the code in latest releases, including this parser.
What needs to be done:
All versions down to 5.0.8 are not sending text tags to parsing chain if text contains only spaces unless xml tag has "preserve=space" attribute:
if (space_preserve || !std::all_of(s.begin(), s.end(), [](auto c){return isspace(static_cast<unsigned char>(c));}))
send_tag(tag::Text{.text = s});
The stripping was introduced in version 5.0.2:
Version 5.0.2:
...
* Ignore all XML text nodes that contains only whitespaces to fix crashes with some ODF and OOXML documents
* Add support for T tag in OOXML formats with whitespace preserve attribute
This is a commit (in previous, private repository) that introduced it: https://github.com/docwire/doctotext_old_private/commit/ad676e298e5de7b9a3d22e26609332ef45f3a8e4 And pull request: https://github.com/docwire/doctotext_old_private/pull/948 It would be good to find the issue with problematic documents and recheck them after new modifications.
http://officeopenxml.com/WPtextSpacing.php :
As with any XML, to preserve spaces in OOXML, the space attribute must be set to preserve: <w:t xml:space="preserve">.
So it seems that the document is not correct according to the specification. Of course if this is a common situation a workaround for this would be perfect but we need to make sure that it will not introduce errors for other documents.
It seems that the space=preserve is set in the document header instead of w:t tag. Maybe supporting this can be a workaround for the issue.
Expected Outcome: All space characters in all Office document types are preserved exactly as they would appear in standard software applications, such as when opening a DOCX file in Microsoft Word.
Agreed for documents that keeps the official standard. When implementing workarounds for documents that do not keep the official standard we can only base on available test documents including attached in this issue. If we can collect more documents with this problem that would be perfect.
The implementation is complete but it will be merged later, together with other improvements.
When I convert the attached document using docwire, it removes some spaces between words. For example, instead of "Motion Picture Association" it has "MotionPictureAssociation"
A little further down, instead of "Dear Mr. Carson:" it has "DearMr.Carson:"
This file is rife with examples of this problem. Perhaps the space character in the Word document isn't a standard space. But if I copy the text from Word and paste it into Notepad, Notepad has the proper spaces between words. 123mpaa.docx