Crash on Mac not on windows with same project

[Niwreg]

Hi,

I'm have installed Manuskript both on my Mac and on an windows 10 machine. I've got an NAS share where the project folder is stored (not into 1 file just the folder structure) When opening on windows there is no issue but when I open the project on Mac it crash. the logs on the Mac report the following:

cat 2023-05-31_11-52-12_manuskript#35931.log
2023-05-31 11:52:12,452 - manuskript.logging - INFO - Logging to file: /Users/gerwinvanderkamp/Library/Application Support/manuskript/manuskript/logs/2023-05-31_11-52-12_manuskript#35931.log 2023-05-31 11:52:12,474 - manuskript - INFO - Operating System: macOS-11.7.6-x86_64-i386-64bit 2023-05-31 11:52:12,474 - manuskript - INFO - Hardware: x86_64 / i386 2023-05-31 11:52:12,474 - manuskript - INFO - Running in a frozen (packaged) state. 2023-05-31 11:52:12,474 - manuskript - DEBUG - sys.frozen = True 2023-05-31 11:52:12,474 - manuskript - DEBUG - sys._MEIPASS = /Applications/manuskript.app/Contents/MacOS 2023-05-31 11:52:12,474 - manuskript - DEBUG - sys.executable = '/Applications/manuskript.app/Contents/MacOS/manuskript' 2023-05-31 11:52:12,474 - manuskript - DEBUG - sys.argv = ['/Applications/manuskript.app/Contents/MacOS/manuskript'] 2023-05-31 11:52:12,475 - manuskript - DEBUG - sys.path = ['/Applications/manuskript.app/Contents/MacOS/base_library.zip', '/Applications/manuskript.app/Contents/MacOS/..', '/Applications/manuskript.app/Contents/MacOS/lib-dynload', '/Applications/manuskript.app/Contents/MacOS'] 2023-05-31 11:52:12,475 - manuskript - DEBUG - sys.prefix = '/Applications/manuskript.app/Contents/MacOS' 2023-05-31 11:52:12,475 - manuskript - INFO - Manuskript 0.15.0-041fda5-041fda5 (Python 3.10.8) 2023-05-31 11:52:12,475 - manuskript - INFO - PyQt 5.15.7 (compiled against Qt 5.15.2) 2023-05-31 11:52:12,475 - manuskript - INFO - Qt 5.15.2 (runtime) 2023-05-31 11:52:12,495 - manuskript - INFO - lxml.etree 4.9.1.0 2023-05-31 11:52:12,496 - manuskript - INFO - libxml 2.9.4 (compiled: 2.9.4) 2023-05-31 11:52:12,496 - manuskript - INFO - libxslt 1.1.29 (compiled: 1.1.29) 2023-05-31 11:52:12,501 - manuskript - INFO - pyEnchant N/A (libenchant: N/A) 2023-05-31 11:52:12,502 - manuskript - INFO - pySpellChecker N/A 2023-05-31 11:52:12,502 - manuskript - INFO - Symspellpy N/A 2023-05-31 11:52:12,502 - manuskript - INFO - * Markdown N/A 2023-05-31 11:52:12,503 - manuskript - INFO - Web rendering engine: QTextEdit 2023-05-31 11:52:12,566 - manuskript.main - INFO - Preferred translation: ['en-GB'] (based on available ui languages) 2023-05-31 11:52:12,568 - manuskript.main - INFO - Loaded translation: manuskript_en_GB.qm 2023-05-31 11:52:15,361 - manuskript.ui.welcome - INFO - Last accessed directory "/Volumes/home/Manuscript/Marean" loaded. 2023-05-31 11:52:18,530 - manuskript.loadSave - INFO - Loading: /Volumes/home/Manuscript/Marean/Marean.msk 2023-05-31 11:52:18,530 - manuskript.loadSave - INFO - Detected file format version: 1. Zip: False. 2023-05-31 11:52:18,530 - manuskript.load_save.version_1 - DEBUG - Loading /Volumes/home/Manuscript/Marean/Marean.msk (folder) 2023-05-31 11:52:23,109 - manuskript.logging - CRITICAL - An unhandled exception has occurred! Traceback (most recent call last): File "manuskript/ui/welcome.py", line 165, in openFile File "manuskript/mainWindow.py", line 591, in loadProject File "manuskript/mainWindow.py", line 886, in loadDatas File "manuskript/loadSave.py", line 68, in loadProject File "manuskript/load_save/version_1.py", line 693, in loadProject File "codecs.py", line 322, in decode UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 2: invalid start byte

Before it opened on the Mac without any issues. Only when opened with windows it gives an error. How can I fix this?

[TheShadowOfHassen]

It looks like it's a corrupted file. Can you start a new project and see if that opens?

[Niwreg]

If I start an new project on the Mac and save it I can reopen it. If I then open it on windows and save it I can't open it anymore in the Mac. But I still can open it on windows.

[TheShadowOfHassen]

So could you create a dummy project and upload one when you can run it on a Mac and then the same project once you uploaded it to the NAS and opened it with windows. I want to see if anything is different between them.

[Niwreg]

Sure

I created an project, saved it on the nas and verified it still opened and it did (projectmac.zip)

then I opened it on my windows machine and saved it opened it on the Mac again and indeed it won't open. (projectwindows.zip)

[TheShadowOfHassen]

That's odd. I put them in a git repository to see if Git would say any changes, and they were identical.

Also on my Ubuntu install I could open both.

Are you the latest Manuskript (0.15.0)?

[TheJackiMonster]

I guess we use text file mode somewhere in Manuskript to store data. Windows uses a different set of characters for line endings for example than any UNIX system like macOS or Linux. So that issue might be related to that. That should be avoidable if Manuskript would properly use standardized UTF-8 as intended.

I assume we need some interoperability test cases to prevent some issues like that in the future. But for now I assume we need to fix that in a way to allow Windows and macOS open projects in both forms because otherwise it might break existing projects. I would definitely set this as issue to be fixed with next release. This might cause a lot of problems when switching to the GTK rewrite some day. ^^'

[Niwreg]

Yes i'm running both machines on 0.15.0

I did some more checking on a file saved under windows and a file under mac and there might indeed be some crlf issues

cat -e 0-Scene_1.md gTrilogy/bug1169/outline/1-Book_2/0-Section_1/00-Ch title: Scene 1^M$ ID: 188^M$ type: md^M$ compile: 2^M$ setGoal: 500^M$ charCount: 40^M$

vs

cat -e 0-Scene_1.md gTrilogy/bug1169/outline/0-Book_1/0-Section_1/00-Ch title: Scene 1$ ID: 4$ type: md$ summaryFull: Saved on mac$ compile: 2$ setGoal: 500$ charCount: 43$ $ $

Other then that i'm using webdav to connect to the nas from the windows machine and i noticed an ._DAV file in the folders. too bad there is no indication on which file or folder it crashes.

[Niwreg]

Saving the file as a single file does make it available on the Mac again.

[TheJackiMonster]

Yes i'm running both machines on 0.15.0

I did some more checking on a file saved under windows and a file under mac and there might indeed be some crlf issues

cat -e 0-Scene_1.md gTrilogy/bug1169/outline/1-Book_2/0-Section_1/00-Ch title: Scene 1^M$ ID: 188^M$ type: md^M$ compile: 2^M$ setGoal: 500^M$ charCount: 40^M$

vs

cat -e 0-Scene_1.md gTrilogy/bug1169/outline/0-Book_1/0-Section_1/00-Ch title: Scene 1$ ID: 4$ type: md$ summaryFull: Saved on mac$ compile: 2$ setGoal: 500$ charCount: 43$ $ $

Other then that i'm using webdav to connect to the nas from the windows machine and i noticed an ._DAV file in the folders. too bad there is no indication on which file or folder it crashes.

Okay, this pretty much confirms my thought so far. Thanks for investigating.

[Dreaded-Gnu]

Would enforce of specific newline like \n for all platforms be a valid fix? That would be the easiest way to handle this cross platform issue.

[Niwreg]

I found the windows install file with this patch :) currently testing let you know the results

[Niwreg]

Just tested the Mac part of it but alas it still crashes:

python3 ./manuskript CRITICAL> An unhandled exception has occurred! Traceback (most recent call last): File "/Users/gerwinvanderkamp/Documents/ManuskriptSRC/manuskript/bin/../manuskript/ui/welcome.py", line 165, in openFile self.mw.loadProject(filename) File "/Users/gerwinvanderkamp/Documents/ManuskriptSRC/manuskript/bin/../manuskript/mainWindow.py", line 932, in loadProject self.loadDatas(project) File "/Users/gerwinvanderkamp/Documents/ManuskriptSRC/manuskript/bin/../manuskript/mainWindow.py", line 1231, in loadDatas errors = loadSave.loadProject(project) File "/Users/gerwinvanderkamp/Documents/ManuskriptSRC/manuskript/bin/../manuskript/loadSave.py", line 68, in loadProject return v1.loadProject(project, zip=isZip) File "/Users/gerwinvanderkamp/Documents/ManuskriptSRC/manuskript/bin/../manuskript/load_save/version_1.py", line 693, in loadProject files[os.path.join(p, f)] = fo.read() File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/codecs.py", line 322, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xec in position 2: invalid continuation byte gerwinvanderkamp@Gerwins-MacBook-Pro bin %

[Niwreg]

Not hindered by any python experience I've added an line to the load_save\version_1.py to see on which files it trips and it is the ERROR> UTF8 ISSUE:/Volumes/home/Manuscript/BugTrilogy/bug1169/summary.txt

This file is 0 bytes on disk and hexdumps doesn't return anything.

log line is implemented around line 693:

try: filename = os.path.join(dirpath, f) LOGGER.error("UTF8 ISSUE:" + filename) with open(filename, "r", encoding="utf8", newline="\n") as fo: files[os.path.join(p, f)] = fo.read() except PermissionError as e: LOGGER.error("Cannot open file " + filename + ": " + e.strerror) errors.append(fo) filesWithPermissionErrors.append(filename)

Is this useful in fixing the issue?

EDIT No not helpful:

Ran it again an it fails consistently now on an ._DAV file in an subdirectory.

ERROR> UTF8 ISSUE:/Volumes/home/Manuscript/BugTrilogy/bug1169/outline/1-Book2.md/0-Section1.md/06-Chapter_7/._DAV/4-Scene_5.md.pag

SO I'll guess there is an issue somewhere around here:||https://github.com/olivierkes/manuskript/blob/6ee7e881bee47d3934dc6eef58666d0dd58202a6/manuskript/load_save/version_1.py#L678

where there is an check if the folder/file starts with an .

[Niwreg]

This works for me now but it might be needing extra checking

[TheJackiMonster]

I merged both contributions since the extra check seems to fix the issue and more consistency with the files should help anyway. Thanks for the contributions.