Sort of worked - Githubissues

richd67 commented 2 years ago

I tried this, I think the steps in the text need a bit of a fuller description but I eventually got there but it only pulled 8 files into the assets folder then seemed to crash with the following error UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 63: character maps to

Tried to run again but no further progress??

romilly commented 2 years ago

I'm sorry you've encountered problems.

It would be really helpful if you could give me a little more information.

Which version of the software did you install, and which version of Python are you using?
Which of the steps were unclear?
Could you copy the entire error output? The message you quoted should have been the last line of a multi-line stack trace. It's not possible to see where things went wrong without the full stack trace.

I'll add better debugging output to the application but I won't be able to publish the new version immediately.

On Fri, 17 Jun 2022 at 15:12, Prof D @.***> wrote:

I tried this, I think the steps in the text need a bit of a fuller description but I eventually got there but it only pulled 8 files into the assets folder then seemed to crash with the following error UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 63: character maps to

Tried to run again but no further progress??

— Reply to this email directly, view it on GitHub https://github.com/romilly/logseq-migration/issues/1, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABEOPTIGHU4FYRYAJFQTWDVPSBWVANCNFSM5ZCNMVJQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- @. @. skype:romilly.cocking web: http://blog.rareschool.com/

richd67 commented 2 years ago

Python 3.10
I have actually never used Python before so it was probably the syntax was not so clear to me and thus exactly how to get stuff to execute. The directory for the migrater.py is actually \logseq-migration-master\scr\migration anf not just \scr.
I never worked out the step of localise_assets??

Hope this helps C:\Users\77780758\Downloads\logseq-migration-master\logseq-migration-master\src\migration>python3 migrater.py C:\Users\77780758\Downloads\Logseqroam migrating C:\Users\77780758\Downloads\Logseqroam downloaded https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2FRichdata%2FH5RtETzoVN.docx?alt=media&token=754a00bc-96a4-4537-9d06-327e95b78a8d to ..\assets\OSS 12.06.20 Paper 1 - Director's Report.docx downloaded https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2FRichdata%2FaCKhCgmIcU.pdf?alt=media&token=410a5b52-9318-4ea6-9cc0-706d32b68083 to ..\assets\OSS 12.06.20 Paper 2 - Management Report.pdf downloaded https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2FRichdata%2FpvfQzz-be5.docx?alt=media&token=29ffd216-2baf-416f-817e-0a2ddf0fb606 to ..\assets\OSS Board 12.06.20 - Research Update NR.docx downloaded https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2FRichdata%2F1cuEKc59QF.docx?alt=media&token=a0d787d6-07ed-44f5-bbaa-46d16cff3630 to ..\assets\OSS 12.06.20 Paper 3 - Fundraising Strategy 2020-23.docx downloaded https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2FRichdata%2FdmXekkai52.docx?alt=media&token=09d9c9f7-04e1-42e2-ab5c-35160a6a330e to ..\assets\OSS 12.06.20 Paper 4 - Fundraising Plan June 2020.docx downloaded https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2FRichdata%2FnEK27e8MST.doc?alt=media&token=bc1887d5-2a67-46bc-96b3-23710840c825 to ..\assets\OSS 12.06.20 Paper 5 - Risk Register.doc downloaded https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2FRichdata%2Fc2oUoGTl5_.doc?alt=media&token=8a216822-e965-4247-a480-986bccd06625 to ..\assets\OSS 12.06.20 Paper 6 - Risk Paper.doc downloaded https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2FRichdata%2FKNNcilcAIO.docx?alt=media&token=20c2d4bf-e167-4735-ae16-a101e8267190 to ..\assets\OSS 12.06.20 Paper 7 - Options 2020-21.docx Traceback (most recent call last): File "C:\Users\77780758\Downloads\logseq-migration-master\logseq-migration-master\src\migration\migrater.py", line 98, in main() File "C:\Users\77780758\Downloads\logseq-migration-master\logseq-migration-master\src\migration\migrater.py", line 94, in main migrate(vault_directory) File "C:\Users\77780758\Downloads\logseq-migration-master\logseq-migration-master\src\migration\migrater.py", line 59, in migrate process_files(assets_dir, assets_from_page_dir, directory) File "C:\Users\77780758\Downloads\logseq-migration-master\logseq-migration-master\src\migration\migrater.py", line 65, in process_files process_file(assets_dir, assets_from_page_dir, file, subdir) File "C:\Users\77780758\Downloads\logseq-migration-master\logseq-migration-master\src\migration\migrater.py", line 78, in process_file lines = find_asset_references(numbered_lines) File "C:\Users\77780758\Downloads\logseq-migration-master\logseq-migration-master\src\migration\migrater.py", line 84, in find_asset_references return list((number, line) for File "C:\Users\77780758\Downloads\logseq-migration-master\logseq-migration-master\src\migration\migrater.py", line 84, in return list((number, line) for File "C:\Users\77780758\Downloads\logseq-migration-master\logseq-migration-master\src\migration\migrater.py", line 77, in numbered_lines = enumerate(line for line in md) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 63: character maps to

C:\Users\77780758\Downloads\logseq-migration-master\logseq-migration-master\src\migration>python3 migrater.py C:\Users\77780758\Downloads\Logseqroam migrating C:\Users\77780758\Downloads\Logseqroam Traceback (most recent call last): File "C:\Users\77780758\Downloads\logseq-migration-master\logseq-migration-master\src\migration\migrater.py", line 98, in main() File "C:\Users\77780758\Downloads\logseq-migration-master\logseq-migration-master\src\migration\migrater.py", line 94, in main migrate(vault_directory) File "C:\Users\77780758\Downloads\logseq-migration-master\logseq-migration-master\src\migration\migrater.py", line 59, in migrate process_files(assets_dir, assets_from_page_dir, directory) File "C:\Users\77780758\Downloads\logseq-migration-master\logseq-migration-master\src\migration\migrater.py", line 65, in process_files process_file(assets_dir, assets_from_page_dir, file, subdir) File "C:\Users\77780758\Downloads\logseq-migration-master\logseq-migration-master\src\migration\migrater.py", line 78, in process_file lines = find_asset_references(numbered_lines) File "C:\Users\77780758\Downloads\logseq-migration-master\logseq-migration-master\src\migration\migrater.py", line 84, in find_asset_references return list((number, line) for File "C:\Users\77780758\Downloads\logseq-migration-master\logseq-migration-master\src\migration\migrater.py", line 84, in return list((number, line) for File "C:\Users\77780758\Downloads\logseq-migration-master\logseq-migration-master\src\migration\migrater.py", line 77, in numbered_lines = enumerate(line for line in md) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 63: character maps to

romilly commented 2 years ago

Thanks for the fast response.

I'm confused.

The latest version of the software needs Python, pip3 and the logseq-migration package to be installed but doesn't require you to run any Python code directly.

When you ran pip3 install logseq-migration, you should have seen a message saying that it installed logseq-migration-0.1.4

You should then have been able to run localise-assets fred in a command window (the same one you used to install logseq-migration) where 'fred' is the name of the directory containing your vault..

However, it looks as if the problem is that one or more of the markdown files in your graph has a character encoding that my Python code doesn't know how to handle.

I've added some more debugging info which should help us track down the file that's causing the problem.

If you could run pip3 install logseq-migration --upgrade and then try to run 'localise_assets xxx` where xxx is the path to your vault, we'll see what report you get.

The migration still won't work, but it should help us track down the exact cause of the problem.

richd67 commented 2 years ago

yes logseq-migration-0.1.4 was installed

for the localise_assets I get the error, 'localise_assets' is not recognized as an internal or external command

romilly commented 2 years ago

It looks as if the installation setup needs to be modified for WIndows :( I'll do some research and respond tomorrow.

On Fri, 17 Jun 2022 at 16:32, Prof D @.***> wrote:

yes logseq-migration-0.1.4 was installed

for the localise_assets I get the error, 'localise_assets' is not recognized as an internal or external command

— Reply to this email directly, view it on GitHub https://github.com/romilly/logseq-migration/issues/1#issuecomment-1158994281, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABEOPTNV66PGNP6Y32Z2KTVPSLAVANCNFSM5ZCNMVJQ . You are receiving this because you commented.Message ID: @.***>

-- @. @. skype:romilly.cocking web: http://blog.rareschool.com/

romilly commented 2 years ago

One more thing to try (if you have time): pip3 install logseq-migration --upgrade should upgrade you to version 0.1.7 then run python3 a Python window should open. In that window, type import logseq.migration.migrater as mig mig.migrate('fred')

where 'fred' is the path to the vault.

richd67 commented 2 years ago

Tried your suggestion this is what I got import logseq.migration.migrater as mig Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'logseq'

romilly commented 2 years ago

Did you upgrade the package first? It should be 0.1.7

On Fri, 17 Jun 2022, 20:06 Prof D, @.***> wrote:

Tried your suggestion this is what I got import logseq.migration.migrater as mig Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'logseq'

— Reply to this email directly, view it on GitHub https://github.com/romilly/logseq-migration/issues/1#issuecomment-1159157113, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABEOPRQ34LUG4XHMOXCE7TVPTECLANCNFSM5ZCNMVJQ . You are receiving this because you commented.Message ID: @.***>

richd67 commented 2 years ago

Apologies no I didnt but this is what I got

mig.migrate(C:\Users\77780758\Downloads\Logseqroam) ^ SyntaxError: invalid syntax

didnt seem to like C:

romilly commented 2 years ago

Running it in the way I suggested, the C:\Users\77780758\Downloads\Logseqroam needs to be in quotes so that Python knows it's a string.

If you want to have another go, upgrade the package, and when you invoke Python, start it like this:

python3 -X utf8 then import logseq.migration.migrater as mig mig.migrate('C:\Users\77780758\Downloads\Logseqroam')

There is a chance that will actually do the migration! If not, at least we should find out the markdown file that's causing the problem.

I think I know why localise_assets' is not working. If you want, we can have a shot at sorting that out tomorrow.

On Fri, 17 Jun 2022, 20:38 Prof D, @.***> wrote:

Apologies no I didnt but this is what I got

mig.migrate(C:\Users\77780758\Downloads\Logseqroam) ^ SyntaxError: invalid syntax

didnt seem to like C:

— Reply to this email directly, view it on GitHub https://github.com/romilly/logseq-migration/issues/1#issuecomment-1159176463, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABEOPRBFLUE75VT4AYV7KTVPTH3PANCNFSM5ZCNMVJQ . You are receiving this because you commented.Message ID: @.***>

richd67 commented 2 years ago

Something still not quite right

mig.migrate('C:\Users\77780758\Downloads\Logseqroam') File "", line 1 mig.migrate('C:\Users\77780758\Downloads\Logseqroam') ^ SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape

romilly commented 2 years ago

Silly me. Python treats the \ as an escape character. Try replacing the \ chars with /. I think Python will cope with that.

Sorry to be making such a meal of this. It's almost a decade since I used Windows in anger. Tomorrow I will try to install a Windows virtual machine so I can do some testing at this end.

On Fri, 17 Jun 2022 at 21:03, Prof D @.***> wrote:

Something still not quite right

mig.migrate('C:\Users\77780758\Downloads\Logseqroam') File "", line 1 mig.migrate('C:\Users\77780758\Downloads\Logseqroam') ^ SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape

— Reply to this email directly, view it on GitHub https://github.com/romilly/logseq-migration/issues/1#issuecomment-1159191829, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABEOPXXGOQW7UT3TGPUV4LVPTKYBANCNFSM5ZCNMVJQ . You are receiving this because you commented.Message ID: @.***>

-- @. @. skype:romilly.cocking web: http://blog.rareschool.com/

romilly commented 2 years ago

It turns out that installing Python on Windows is a more fragile process than on any other platform; some tempting installation options mean that you cannot run programs like localiise_assets.

I installed Python for all users, adding Python 3.10 to the path, and I can run localises_assets without problems. That doen;t help you much though.

I am not sure how you installed Python; if this application is your only reason to have Python installed it's probably best to leave the installtation as it is.

I've updated the logseq-migration package so that it will give us more debug infprmation.

If you upgrade the package, start python and then type

import logseq.migration.migrater as mig
DEBUG=True
mig.migrate('C:/Users/77780758/Downloads/Logseqroam')

It should produce lots of output, listing each markdown file it finds before it reads it.

It will fall over again, but this time we should be able to find the file that causes the problem. Then we can hopefully figure out why.

I suspect it's a problem with file encodings, in which case there may be a simple fix.

richd67 commented 2 years ago

Ok heres what happened

Traceback (most recent call last): File "", line 1, in File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 60, in migrate process_files(assets_dir, assets_from_page_dir, directory) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 66, in process_files process_file(assets_dir, assets_from_page_dir, file, subdir) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 83, in process_file lines = find_asset_references(numbered_lines) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 89, in find_asset_references return list((number, line) for File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 89, in return list((number, line) for File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 79, in numbered_lines = enumerate(line for line in md) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 63: character maps to

romilly commented 2 years ago

Did you upgrade the packed first?

romilly commented 2 years ago

*package

romilly commented 2 years ago

pip3 install logseq-migrate --upgrade should take you up to version 0.1.10

richd67 commented 2 years ago

Sorry I hadnt noticed your updated package but having downloaded the new package and run pip as suggested this is what I get

ERROR: Could not find a version that satisfies the requirement logseq-migrate (from versions: none) ERROR: No matching distribution found for logseq-migrate

romilly commented 2 years ago

I've now upped the version to 0.1.11. The latest version may have fixed the bug that is causing the crash. It appears that the default file encoding for Python on Windows is not utf8, which is probably the encoding used when logseq imported your roam graph.

romilly commented 2 years ago

It's logseq-migration, not logseq-migrate

On Sat, 18 Jun 2022 at 16:14, Prof D @.***> wrote:

Sorry I hadnt noticed your updated package but having downloaded the new package and run pip as suggested this is what I get

ERROR: Could not find a version that satisfies the requirement logseq-migrate (from versions: none) ERROR: No matching distribution found for logseq-migrate

— Reply to this email directly, view it on GitHub https://github.com/romilly/logseq-migration/issues/1#issuecomment-1159481607, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABEOPVD4W7IUZEL6HY5JQDVPXRWTANCNFSM5ZCNMVJQ . You are receiving this because you commented.Message ID: @.***>

-- @. @. skype:romilly.cocking web: http://blog.rareschool.com/

richd67 commented 2 years ago

Yes now have 0.1.11

latest python output

Traceback (most recent call last): File "", line 1, in File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 68, in migrate process_files(assets_dir, assets_from_page_dir, directory) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 74, in process_files process_file(assets_dir, assets_from_page_dir, file, subdir) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 95, in process_file process_assets(file_path, lines, assets_dir, assets_from_page_dir) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 44, in process_assets content = update_markdown_content(asset_dir, file_path, lines, relative_asset_dir) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 53, in update_markdown_content content = md.readlines() File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 2310: character maps to

romilly commented 2 years ago

If you set DEBUG=True before you ran the migrate function, it should have printed the fiel it was downloading just before the stack trace started , That's; the information we need

However, it did get further this time. (To character 2301). It looks as if the file endcoding isn't utf8, but we can't check that until we know what file is causing the problem.

richd67 commented 2 years ago

Just tried that but got exactly the same outcome?

romilly commented 2 years ago

You're get the same outcome. But there will be lines before the part of the stack trace youy've sent me. Those are the lines I need to see.

There should be a line that says something like 'updating fred.md' just before things go wrong and the stack trace starts. That's the line we need.

Also, when the migrate program started it should have printed 'debug mode'. Did it?

richd67 commented 2 years ago

Nothing before the stack? Exactly as pasted above?

romilly commented 2 years ago

could you do a screenshot or copy the lines in the command window at the point where where you ran the app, with the first few lines of output after that?

richd67 commented 2 years ago

Here is the copy and paste

import logseq.migration.migrater as mig DEBUG=True mig.migrate('C:/Users/77780758/Downloads/Logseqroam') Traceback (most recent call last): File "", line 1, in File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 68, in migrate process_files(assets_dir, assets_from_page_dir, directory) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 74, in process_files process_file(assets_dir, assets_from_page_dir, file, subdir) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 95, in process_file process_assets(file_path, lines, assets_dir, assets_from_page_dir) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 44, in process_assets content = update_markdown_content(asset_dir, file_path, lines, relative_asset_dir) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 53, in update_markdown_content content = md.readlines() File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 2310: character maps to

romilly commented 2 years ago

It looks as if the python session isn't running the upgraded version of the package.

I've uploaded version 0.1.12 which will print its version number as soon as it starts. That way we can confirm which version is running.

I'm sorry this is turning out to be such a marathon. (I'm also experiencing Windows update hell - is it usual for a Windows update to take an hour or more?)

On Sat, 18 Jun 2022 at 17:08, Prof D @.***> wrote:

Here is the copy and paste

import logseq.migration.migrater as mig DEBUG=True mig.migrate('C:/Users/77780758/Downloads/Logseqroam') Traceback (most recent call last): File "", line 1, in File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 68, in migrate process_files(assets_dir, assets_from_page_dir, directory) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 74, in process_files process_file(assets_dir, assets_from_page_dir, file, subdir) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 95, in process_file process_assets(file_path, lines, assets_dir, assets_from_page_dir) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 44, in process_assets content = update_markdown_content(asset_dir, file_path, lines, relative_asset_dir) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 53, in update_markdown_content content = md.readlines() File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 2310: character maps to

— Reply to this email directly, view it on GitHub https://github.com/romilly/logseq-migration/issues/1#issuecomment-1159489838, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABEOPUFTPDNFY3M6MOLOGTVPXYANANCNFSM5ZCNMVJQ . You are receiving this because you commented.Message ID: @.***>

-- @. @. skype:romilly.cocking web: http://blog.rareschool.com/

richd67 commented 2 years ago

OK upgraded to 0.1.12

and this is what I got?

import logseq.migration.migrater as mig DEBUG=True mig.migrate('C:/Users/77780758/Downloads/Logseqroam') migrate version 0.1.12 debug mode Off Traceback (most recent call last): File "", line 1, in File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 69, in migrate process_files(assets_dir, assets_from_page_dir, directory) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 75, in process_files process_file(assets_dir, assets_from_page_dir, file, subdir) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 96, in process_file process_assets(file_path, lines, assets_dir, assets_from_page_dir) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 44, in process_assets content = update_markdown_content(asset_dir, file_path, lines, relative_asset_dir) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 53, in update_markdown_content content = md.readlines() File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 2310: character maps to

romilly commented 2 years ago

That's puzzling. I'll do more local testing tomorrow.

On Sat, 18 Jun 2022, 18:46 Prof D, @.***> wrote:

OK upgraded to 0.1.12

and this is what I got?

import logseq.migration.migrater as mig DEBUG=True mig.migrate('C:/Users/77780758/Downloads/Logseqroam') migrate version 0.1.12 debug mode Off Traceback (most recent call last): File "", line 1, in File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 69, in migrate process_files(assets_dir, assets_from_page_dir, directory) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 75, in process_files process_file(assets_dir, assets_from_page_dir, file, subdir) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 96, in process_file process_assets(file_path, lines, assets_dir, assets_from_page_dir) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 44, in process_assets content = update_markdown_content(asset_dir, file_path, lines, relative_asset_dir) File "C:\Users\77780758\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\logseq\migration\migrater.py", line 53, in update_markdown_content content = md.readlines() File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 2310: character maps to

— Reply to this email directly, view it on GitHub https://github.com/romilly/logseq-migration/issues/1#issuecomment-1159522898, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABEOPWZAE5SCBRX7IRTMZTVPYDODANCNFSM5ZCNMVJQ . You are receiving this because you commented.Message ID: @.***>

richd67 commented 2 years ago

No problem, hopefully this is helpful for you.

romilly commented 2 years ago

Very helpful, and I'm very grateful for your patience and cooperation.

I've written a lot of open source software, and some is quite widely used. I've not been able to test in Windows until now.

I now have a Windows Virtual Machine set up. It's been quite an eye-opener.

Today I'll add proper logging to the app and add logging to the acceptance tests. I'll let you know how it goes. When it's ready it should pinpoint the file or files that are causing the problem and convert the rest.

On Sat, 18 Jun 2022, 20:35 Prof D, @.***> wrote:

No problem, hopefully this is helpful for you.

— Reply to this email directly, view it on GitHub https://github.com/romilly/logseq-migration/issues/1#issuecomment-1159546029, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABEOPURV2DD6NK6FH32T2LVPYQJ5ANCNFSM5ZCNMVJQ . You are receiving this because you commented.Message ID: @.***>

romilly commented 2 years ago

Whew!

I've updated the app to version 0.2, so please start by getting pip to upgrade it.

Now change into the directory where you imported your graph - C:/Users/77780758/Downloads/Logseqroam

start Python from that directory, and type

import logseq.migration.migrater as mig mig.migrate('.')

The last line has a period enclosed in single quotation marks.

This should print the version number and then eventually print 'done'. It should also create a very detailed log file called 'migration.log' in the same directory.

I've modified the application so that it will now try to continue after encountering the sort of errors we had before.

The log file will show all the successful operations, but there will be also some lines that start ERROR:

Those error lines should show the names of the file(s) that caused problems, and we can then try to see what the problem is.

Any files that contain data saved in Roam that don't cause problems should be correctly migrated.

Please let me know how this goes!

richd67 commented 2 years ago

Contents of the migration.log

INFO:root:migrator version 0.2 INFO:root:migrating directory C:\Users\77780758\Downloads\logseq-migration-master\logseq-migration-master DEBUG:root:processing .\README.md DEBUG:root:processing .\ROADMAP.md DEBUG:root:processing .\plan\journal.md INFO:root:run completed

romilly commented 2 years ago

That ran in the directory where you installed my code, not the directory where you imported your file, wihich is not what happens when I run it in windows. I think the script that runs Python is changing the working driectory

Could you try import logseq.migration.migrater as mig mig.migrate('C:/Users/77780758/Downloads/Logseqroam')

richd67 commented 2 years ago

So here is the contents of the log file.

INFO:root:migrator version 0.2 INFO:root:migrating directory C:\Users\77780758\Downloads\Logseqroam DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\02-06-2020.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\03-06-2020.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\04-06-2020.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\05-06-2020.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\08-06-2020.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\09-06-2020.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_06_10.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_06_11.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_06_12.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_06_13.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_06_15.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_06_16.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_06_18.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_06_20.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_06_22.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_06_23.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_06_25.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_06_26.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_06_29.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_07_01.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_07_05.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_07_06.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_07_07.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_07_09.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_07_13.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_07_14.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_07_15.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_07_16.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_07_20.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_07_22.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_07_23.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_07_24.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_07_26.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_07_27.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_07_28.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_07_29.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_07_30.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_07_31.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_08_02.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_08_03.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_08_05.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_08_06.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_08_07.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_08_08.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_08_10.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_08_13.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_08_16.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_08_17.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_08_18.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_08_19.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_08_21.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_08_24.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_08_25.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_08_26.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_08_27.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_08_28.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_08_31.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_09_01.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_09_02.md DEBUG:root:processing C:/Users/77780758/Downloads/Logseqroam\journals\2020_09_03.md DEBUG:root:updating markdown file C:/Users/77780758/Downloads/Logseqroam\journals\2020_09_03.md

romilly commented 2 years ago

That looks like the beginning of the log but not the complete file. Would it be possible for you to send me the original? You can attach a file to a note here in issues.

romilly commented 2 years ago

Alternatively, if that's the whole file, it crashed when trying to read the journal entry 2020_09_03.md. Let me know and we can discuss what to do next.

romilly / logseq-migration

Sort of worked #1