Closed willbasky closed 5 years ago
There's something "wrong" with the attachments definitions in the note export. @willbasky could you provide that particular note as a sample?
I guess I can't, it is personal, sorry.
I exported notes to enex without tags and got this kind of error.
$ enex2md --disk backup_ever.enex
Traceback (most recent call last):
File "/usr/local/bin/enex2md", line 5, in <module>
from enex2md.cli import app
File "/usr/local/lib/python3.6/dist-packages/enex2md/cli.py", line 8, in <module>
from enex2md.convert import Converter
File "/usr/local/lib/python3.6/dist-packages/enex2md/convert.py", line 11, in <module>
from dateutil.parser import parse
File "/home/metaxis/.local/lib/python3.6/site-packages/dateutil/parser.py", line 158
l.append("%s=%s" % (attr, `value`))
^
SyntaxError: invalid syntax
@willbasky Ok, let's have a look at the second issue. What are the versions of the installed packages + Python? A.k.a. the output of the following two commands:
python -V pip freeze
$ python -V
Python 2.7.15+
$ pip -V
pip 19.3.1 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)
$ pip freeze
acme==0.31.0
apt-xapian-index==0.47
asciinema==2.0.0
asn1crypto==0.24.0
beautifulsoup4==4.8.1
certbot==0.27.0
certbot-nginx==0.23.0
certifi==2018.1.18
chardet==3.0.4
Click==7.0
colorlog==3.1.2
command-not-found==0.3
ConfigArgParse==0.11.0
configobj==5.0.6
cryptography==2.1.4
cupshelpers==1.0
dbus-python==1.2.12
defer==1.0.6
devscripts===2.17.12ubuntu1.1
distro-info===0.18ubuntu0.18.04.1
enex2md==0.3.4
future==0.15.2
gpg==1.10.0
greenlet==0.4.12
html2text==2019.9.26
httpie==0.9.8
httplib2==0.9.2
idna==2.6
jack-select==1.3.1
josepy==1.1.0
keyring==10.6.0
keyrings.alt==3.0
language-selector==0.1
lxml==4.4.1
macaroonbakery==1.1.3
Mako==1.0.7
MarkupSafe==1.0
mock==2.0.0
msgpack==0.5.6
neovim==0.2.0
netifaces==0.10.4
notify2==0.3
olefile==0.45.1
parsedatetime==2.4
pbr==3.1.1
pexpect==4.2.1
Pillow==5.1.0
protobuf==3.0.0
pycairo==1.18.1
pycrypto==2.6.1
pycups==1.9.73
Pygments==2.2.0
PyGObject==3.34.0
pymacaroons==0.13.0
PyNaCl==1.1.2
pyOpenSSL==17.5.0
pyparsing==2.2.0
pyRFC3339==1.0
python-apt==1.6.4
python-dateutil==1.5
python-debian==0.1.32
python-magic==0.4.16
pytz==2018.3
pyudev==0.21.0
pyxattr==0.6.0
pyxdg==0.26
PyYAML==3.12
reportlab==3.4.0
requests==2.18.4
requests-toolbelt==0.8.0
requests-unixsocket==0.1.5
scour==0.36
screen-resolution-extra==0.0.0
SecretStorage==2.3.1
Shredder===2.6.1.Penetrating.Pineapple
six==1.12.0
soupsieve==1.9.5
ssh-import-id==5.7
sshuttle==0.78.3
system-service==0.3
systemd-python==234
ubuntu-drivers-common==0.0.0
ufw==0.36
unattended-upgrades==0.1
unidiff==0.5.4
urllib3==1.22
xkit==0.0.0
youtube-dl==2018.3.14
zope.component==4.3.0
zope.event==4.2.0
zope.hookable==4.0.4
zope.interface==4.3.2
Thank you @willbasky for the details, those explain what's going on. This tool only supports Python 3.6+, and you are using 2.7.15. Python 2 support has never been on the agenda for enex2md.
Since Python 2 is end-of-life in less than two months, I strongly suggest for you to move to Python 3.
Ok. I switched to python 3 and got following error:
$ alias python=python3
$ python -V
Python 3.6.8
$ enex2md --disk backup_ever.enex
Traceback (most recent call last):
File "/usr/local/bin/enex2md", line 5, in <module>
from enex2md.cli import app
File "/usr/local/lib/python3.6/dist-packages/enex2md/cli.py", line 8, in <module>
from enex2md.convert import Converter
File "/usr/local/lib/python3.6/dist-packages/enex2md/convert.py", line 11, in <module>
from dateutil.parser import parse
File "/home/metaxis/.local/lib/python3.6/site-packages/dateutil/parser.py", line 158
l.append("%s=%s" % (attr, `value`))
^
SyntaxError: invalid syntax
@janikarh ^
Could you please rerun pip freeze
and post the results. I need to check the exact versions, since I am unable to reproduce the issue on macOS or Fedora.
$ python -V
Python 3.6.8
$ pip -V
pip 19.3.1 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)
$ pip freeze
acme==0.31.0
apt-xapian-index==0.47
asciinema==2.0.0
asn1crypto==0.24.0
beautifulsoup4==4.8.1
certbot==0.27.0
certbot-nginx==0.23.0
certifi==2018.1.18
chardet==3.0.4
Click==7.0
colorlog==3.1.2
command-not-found==0.3
ConfigArgParse==0.11.0
configobj==5.0.6
cryptography==2.1.4
cupshelpers==1.0
dbus-python==1.2.12
defer==1.0.6
devscripts===2.17.12ubuntu1.1
distro-info===0.18ubuntu0.18.04.1
enex2md==0.3.4
future==0.15.2
gpg==1.10.0
greenlet==0.4.12
html2text==2019.9.26
httpie==0.9.8
httplib2==0.9.2
idna==2.6
jack-select==1.3.1
josepy==1.1.0
keyring==10.6.0
keyrings.alt==3.0
language-selector==0.1
lxml==4.4.1
macaroonbakery==1.1.3
Mako==1.0.7
MarkupSafe==1.0
mock==2.0.0
msgpack==0.5.6
neovim==0.2.0
netifaces==0.10.4
notify2==0.3
olefile==0.45.1
parsedatetime==2.4
pbr==3.1.1
pexpect==4.2.1
Pillow==5.1.0
protobuf==3.0.0
pycairo==1.18.1
pycrypto==2.6.1
pycups==1.9.73
Pygments==2.2.0
PyGObject==3.34.0
pymacaroons==0.13.0
PyNaCl==1.1.2
pyOpenSSL==17.5.0
pyparsing==2.2.0
pyRFC3339==1.0
python-apt==1.6.4
python-dateutil==1.5
python-debian==0.1.32
python-magic==0.4.16
pytz==2018.3
pyudev==0.21.0
pyxattr==0.6.0
pyxdg==0.26
PyYAML==3.12
reportlab==3.4.0
requests==2.18.4
requests-toolbelt==0.8.0
requests-unixsocket==0.1.5
scour==0.36
screen-resolution-extra==0.0.0
SecretStorage==2.3.1
Shredder===2.6.1.Penetrating.Pineapple
six==1.12.0
soupsieve==1.9.5
ssh-import-id==5.7
sshuttle==0.78.3
system-service==0.3
systemd-python==234
ubuntu-drivers-common==0.0.0
ufw==0.36
unattended-upgrades==0.1
unidiff==0.5.4
urllib3==1.22
xkit==0.0.0
youtube-dl==2018.3.14
zope.component==4.3.0
zope.event==4.2.0
zope.hookable==4.0.4
zope.interface==4.3.2
one.enex.zip i exported one note and got error:
$ enex2md --disk one.enex
Traceback (most recent call last):
File "/usr/local/bin/enex2md", line 5, in <module>
from enex2md.cli import app
File "/usr/local/lib/python3.6/dist-packages/enex2md/cli.py", line 8, in <module>
from enex2md.convert import Converter
File "/usr/local/lib/python3.6/dist-packages/enex2md/convert.py", line 11, in <module>
from dateutil.parser import parse
File "/home/metaxis/.local/lib/python3.6/site-packages/dateutil/parser.py", line 158
l.append("%s=%s" % (attr, `value`))
^
SyntaxError: invalid syntax
@willbasky Ok, I might now know, what's going on. Your python-dateutil
version is 1.5
which is way too old for enex2md
. I have failed to pin the versions on the setup.py
of enex2md
package, which might be the reason why an old version is "accepted". Thanks for the info, I'll ping you when new version is ready.
@willbasky I just released version 0.4.0
, which has version pinning added. This will make sure, that all the dependencies are at the minimum level. Could you please reinstall the package, and try again? Please consider using a Python virtual environment to ensure a clean installation.
Upgreded. One.enex
converted well.
Then I set python virtual environment, install enex2md and try to convert *.enex
with all my notes and it gave error:
$ pip -V
pip 9.0.1 from /home/metaxis/Загрузки/env/lib/python3.6/site-packages (python 3.6)
(env)
$ python -V
Python 3.6.8
(env)
$ pip freeze
beautifulsoup4==4.8.1
Click==7.0
enex2md==0.4.0
html2text==2019.9.26
lxml==4.4.1
pkg-resources==0.0.0
python-dateutil==2.8.1
six==1.13.0
soupsieve==1.9.5
(env)
$ enex2md --disk backup_ever.enex
Processing input file: backup_ever.enex, writing output to DISK.
Traceback (most recent call last):
File "/home/metaxis/Загрузки/env/bin/enex2md", line 11, in <module>
sys.exit(app())
File "/home/metaxis/Загрузки/env/lib/python3.6/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/home/metaxis/Загрузки/env/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/home/metaxis/Загрузки/env/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/metaxis/Загрузки/env/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/metaxis/Загрузки/env/lib/python3.6/site-packages/enex2md/cli.py", line 25, in app
converter.convert()
File "/home/metaxis/Загрузки/env/lib/python3.6/site-packages/enex2md/convert.py", line 33, in convert
notes = self._parse_notes(tree)
File "/home/metaxis/Загрузки/env/lib/python3.6/site-packages/enex2md/convert.py", line 99, in _parse_notes
attachment['filename'] = resource.xpath('resource-attributes/file-name')[0].text
IndexError: list index out of range
(env)
Ok, happy to hear you got the environment working. We are back to where we started, there's something strange with the xml data. Something I have not come across yet. I will try to think some way to circumvent the issue by bypassing those attachments which does not have necessary (known) data to extract them. I'll ping you once again, when next release is out.
@willbasky v0.4.1 is out, please upgrade and try again. I tested this by intentionally breaking the enex, let's see how this behaves with the "real data". Let me know how it goes.
Upgraded to 0.4.1, now it converted 4 notes from 916 and stop with error:
$ enex2md --disk backup_ever.enex
Processing input file: backup_ever.enex, writing output to DISK.
Skipping attachment on note with title "Встреча с ребром адама" because the name xml element is missing (resource/resource-attributes/file-name).
Processed 916 note(s).
Traceback (most recent call last):
File "/home/metaxis/Загрузки/env/bin/enex2md", line 11, in <module>
sys.exit(app())
File "/home/metaxis/Загрузки/env/lib/python3.6/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/home/metaxis/Загрузки/env/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/home/metaxis/Загрузки/env/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/metaxis/Загрузки/env/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/metaxis/Загрузки/env/lib/python3.6/site-packages/enex2md/cli.py", line 25, in app
converter.convert()
File "/home/metaxis/Загрузки/env/lib/python3.6/site-packages/enex2md/convert.py", line 38, in convert
self._write_markdown(notes, output_folder)
File "/home/metaxis/Загрузки/env/lib/python3.6/site-packages/enex2md/convert.py", line 354, in _write_markdown
output_file.writelines("%s\n" % l for l in self._format_note(note))
File "/home/metaxis/Загрузки/env/lib/python3.6/site-packages/enex2md/convert.py", line 289, in _format_note
note_content.append(f"- Updated at: {note['updated']}")
KeyError: 'updated'
(env)
I looked what Встреча с ребром адама
note is. Nothing spacial. It has name. There is text only.
I exported it as single and it converted well. It is strange.
@willbasky There's no attachment on that note, did I understand correctly? If so, then Evernote is generating really "interesting" xml with extra elements.
The updated
part required an update from my side, v0.4.2 is ready for testing. Sorry this is moving forward with one step at a time, but you are the first external user to provide bug reports. Thank you for that, by the way!
It is ok we were moving step by step. My pleasure to help you.
Yes, there was no attachments... i guess
Congratulate! You did it!
$ enex2md --disk backup_ever.enex
Processing input file: backup_ever.enex, writing output to DISK.
Skipping attachment on note with title "Встреча с ребром адама" because the name xml element is missing (resource/resource-attributes/file-name).
Processed 916 note(s).
(env)
Wait, it skipped some attachment. I checked there is link with icon, maybe it is this attachment.
Happy to hear, that it worked now!
If you can find that part from the enex xml-bundle and provide that segment here, I would be happy to check what's going on there.
there is attachment
<mime>image/png</mime>
<width>16</width>
<height>16</height>
and it inserted to text within
<div><en-media alt="http://l-files.livejournal.net/userhead/1144?v=1364988178" hash="f843775848f8e56e97e53420e5db24b2" style="cursor: default;" type="image/png"/></div>
<div><strong><a href="http://www.livejournal.com/update.bml">rebro_a_dama</a></strong> .</div>
Ok, that's an image, not an attachment per se. Does it appear on the markdown output?
Nope, the icon was eliminated.
Ok, I will need to look into that on another day.
@willbasky Is that note a web clipping, not a note you wrote yourself? If so, could you share the whole note xml?
I wrote it myself. It is the same as other ones, but it has resource.
<resource>
<data encoding="base64">
iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAAXNSR0IArs4c6QAAAARnQU1BAACx jwv8YQUAAAAJcEhZcwAACxEAAAsRAX9kX5EAAALiSURBVDhPdZL9T81RHMf9GURdT3XvshmGrRWJ VMYSpmzlaeW23ZpWslFR5nE2lGwe5uq5XMoVYcSSHm7NxdQ0qZuunlT33tKW233q5fRtSuKHz87n nPN+v87nfM6ZA8wK41Xfrq+XFeQFe3I70Jua+KX03Vw++i/trIWB4rBxk2YbH5PXUnXQA832RehU MtpPL8ZUtJixtjv2P/UzzPavhc3Ongc4Pp2jWx2ItaWYmwFyDJdk/Kj0x9F9j9EPSUL6H4D5uS+O zhysrWpubJBja9dye6OCy36e5IR4M1KdSH++jJGaNCH/C9ByPoQG1UJ6c+eT5e/FW3UyTlMDn/OP 8uhUFBd8lvAgXE5zhpx3qStmA55HufM+QcatAAV1mUpclgbGh94yPqznS9ExStMiuCIqqYiUU7ZL RnaApwSRzLnBCgqCF5CzWUFu0DJcwugwNzJuaZRy26COb2UnKTm+m+z1XpRHLKJUQKYAFRGb0KXP xfzEj8LQVVj79TgtesaE0W5qFAAxH9JTfSWWXo0X37VrqDocNQ2oPOAtzD5Uxrihu6ZizNIqnTza VyMADVI+ATRoUqlJnovl2Toe7nKfBmh3LqDpzHxMj31p1p7HNfwe64CooL8ehwTQiyvp+fm9DmPh SvQn5nF2/bxpwKM97gy/8KNtXwyGvdGMdL+RxK5BYTbpJIhVwAxRsXTuU2J56kd9vMc0YCJeJ7ph UybxIzqBn035k80rvshY5jkBqsPalIf1UCK22CQs5f6SeQbgdxgPxKOO8KH7Ybp4SvEKg/V0laZx PSaQjkilkMzUTyXFiaHcPbqDO8og7h7ZTlHcFux9VdiNz3i134NbhwIpSQpFkxwm7c8A3E8NR60K oVOTgvFeijTaeqtx9r4UFdTi6KmkS5vBN1FJT8VZCgQoL2GbBJEAatVWBquysLwrYLT9iWhcPS7z 5E+c6IXLrMM5UEtHdBwGEV1laVwKXy2szPkF/N3ovQOX9lwAAAAASUVORK5CYII=
</data>
<mime>image/png</mime>
<width>16</width>
<height>16</height>
</resource>
Hmm. Really odd it gets skipped. Need to look into it more.
How looks xml of note with image converted well?
Images should and do work well, based on my own tests with several web clipped and hand-crafted notes.
Have you try to make xml of note with fields i showed?
I did, still not sure if I added them to the right/same places as they appear on your note. My test looked ok.
Full xml structure of this note. Text changed to blahblah:
<note>
<title>blahblah</title>
<content>
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE en-note SYSTEM "http://xml.evernote.com/pub/enml2.dtd"> <en-note style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div>blahblah</div><div><br/></div><div><en-media alt="http://l-files.livejournal.net/userhead/1144?v=1364988178" hash="f843775848f8e56e97e53420e5db24b2" style="cursor: default;" type="image/png"/></div><div><strong><a href="http://www.livejournal.com/update.bml">rebro_a_dama</a></strong> .</div><div>blahblah</div><div><br/></div><div>blahblah <a href="http://www.pushkin.edu.ru/%20/t%20_blank">институте</a> blahblah</div><div>blahblah</div><div><br/></div><div>blahblah</div><div><br/></div><div>blahblah</div><div><br/></div><div>blahblah</div><div><br/></div><div>blahblah</div><div>blahblah</div><div>blahblah</div><div>blahblah</div><div>blahblah</div><div>blahblah</div><div><br/></div><div>blahblah</div><div>blahblah</div><div>blahblah</div><div>blahblah</div><div>3. blahblah</div><div>blahblah</div><div><br/></div><div>blahblah</div><div><br/></div><div>blahblah</div><div>blahblah</div><div><br/></div></en-note>
</content>
<created>20150425T082652Z</created>
<updated>20150425T104052Z</updated>
<resource>
<data encoding="base64">
iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAAXNSR0IArs4c6QAAAARnQU1BAACx jwv8YQUAAAAJcEhZcwAACxEAAAsRAX9kX5EAAALiSURBVDhPdZL9T81RHMf9GURdT3XvshmGrRWJ VMYSpmzlaeW23ZpWslFR5nE2lGwe5uq5XMoVYcSSHm7NxdQ0qZuunlT33tKW233q5fRtSuKHz87n nPN+v87nfM6ZA8wK41Xfrq+XFeQFe3I70Jua+KX03Vw++i/trIWB4rBxk2YbH5PXUnXQA832RehU MtpPL8ZUtJixtjv2P/UzzPavhc3Ongc4Pp2jWx2ItaWYmwFyDJdk/Kj0x9F9j9EPSUL6H4D5uS+O zhysrWpubJBja9dye6OCy36e5IR4M1KdSH++jJGaNCH/C9ByPoQG1UJ6c+eT5e/FW3UyTlMDn/OP 8uhUFBd8lvAgXE5zhpx3qStmA55HufM+QcatAAV1mUpclgbGh94yPqznS9ExStMiuCIqqYiUU7ZL RnaApwSRzLnBCgqCF5CzWUFu0DJcwugwNzJuaZRy26COb2UnKTm+m+z1XpRHLKJUQKYAFRGb0KXP xfzEj8LQVVj79TgtesaE0W5qFAAxH9JTfSWWXo0X37VrqDocNQ2oPOAtzD5Uxrihu6ZizNIqnTza VyMADVI+ATRoUqlJnovl2Toe7nKfBmh3LqDpzHxMj31p1p7HNfwe64CooL8ehwTQiyvp+fm9DmPh SvQn5nF2/bxpwKM97gy/8KNtXwyGvdGMdL+RxK5BYTbpJIhVwAxRsXTuU2J56kd9vMc0YCJeJ7ph UybxIzqBn035k80rvshY5jkBqsPalIf1UCK22CQs5f6SeQbgdxgPxKOO8KH7Ybp4SvEKg/V0laZx PSaQjkilkMzUTyXFiaHcPbqDO8og7h7ZTlHcFux9VdiNz3i134NbhwIpSQpFkxwm7c8A3E8NR60K oVOTgvFeijTaeqtx9r4UFdTi6KmkS5vBN1FJT8VZCgQoL2GbBJEAatVWBquysLwrYLT9iWhcPS7z 5E+c6IXLrMM5UEtHdBwGEV1laVwKXy2szPkF/N3ovQOX9lwAAAAASUVORK5CYII=
</data>
<mime>image/png</mime>
<width>16</width>
<height>16</height>
</resource>
</note>
@willbasky, an analysis:
There is an image reference in the note content:
<en-media alt="http://l-files.livejournal.net/userhead/1144?v=1364988178" hash="f843775848f8e56e97e53420e5db24b2" style="cursor: default;" type="image/png"/>
. Pay attention to the hash attribute.
That same hash should be present in the resource
section of the xml, to link the image to the actual base64 encoded image data. Since the hash is missing from the resource, the linking is not possible, thus the image is never generated in the markdown.
Since the said resource has no resource/resource-attributes/file-name
either, it cannot be extracted as an attachment.
Unfortunately enex2md
is not able to process that kind of data, since some key elements are missing. One solution could be to generate a random name for the attachments with no name, but it would be hard to identify the said attachment after conversion. Would that option suit your needs?
Why the image be showed at evernote even though wrong hash and lack of name are?
Your guess is as good as mine, only Evernote knows, why their exporter works as it does. Could be a bug. Any more issues you noticed, which we could take a look at, or can I close this issue?
I suppose we can to grab a pie off the shelf. Thank you for collaboration!
Thanks for reporting! Do let me know, if you find anything else.
OS: Kubuntu 18.04