search

awdeorio / mailmerge

A simple, command line mail merge tool.
MIT License
140 stars 41 forks source link

Encoding UTF-8 #42

Closed lightmagic1 closed 5 years ago

lightmagic1 commented 5 years ago

mailmerge_template.txt


TO: {{email}}
SUBJECT: Testing mailmerge
FROM: My Self <myself@mydomain.com>
Content-Type: text/html; charset="utf-8"

<!doctype html>
<html lang="pt-br">

<head>
    <title> Page title </title>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <style type="text/css">
        * {
            margin: 0;
            border: 0;
            padding: 0;
            color: red;

            font-family: Arial, Helvetica, sans-serif;
        }
    </style>
</head>

<body>

<h1>ççç ÇÇÇ áéíóú !.</h1>
</body>

</html>

Terminal:

>mailmerge
>>> message 0
TO: light.magic1@gmail.com
SUBJECT: Testing mailmerge
FROM: My Self <myself@mydomain.com>
Content-Type: text/html; charset="utf-8"

<!doctype html>
<html lang="pt-br">

<head>
    <title> Page title </title>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <style type="text/css">
        * {
            margin: 0;
            border: 0;
            padding: 0;
            color: red;

            font-family: Arial, Helvetica, sans-serif;
        }
    </style>
</head>

<body>

<h1>ççç ÇÇÇ áéíóú !.</h1>
</body>

</html>
>>> encoding utf-8
>>> sent message 0 DRY RUN
>>> No attachments were sent with the emails.
>>> Limit was 1 messages.  To remove the limit, use the --no-limit option.
>>> This was a dry run.  To send messages, use the --no-dry-run option.

Recieved mail:

image

Am i doing anything wrong? What should i do to those characters appears?

awdeorio commented 5 years ago

Here's a couple things to try.

Check your Python version. I expect Python 3.

$ python --version

Check the encoding of the template. I expect UTF-8.

$ file mailmerge_template.txt
lightmagic1 commented 5 years ago

Here's a couple things to try.

Check your Python version. I expect Python 3.

$ python --version

Check the encoding of the template. I expect UTF-8.

$ file mailmerge_template.txt
λ python --version
Python 3.7.3
λ file mailmerge_template.txt
mailmerge_template.txt: HTML document, UTF-8 Unicode text, with CRLF line terminators

anything else that i should try?

:(

awdeorio commented 5 years ago

Digging a little deeper, it looks like this is not related to actually sending the message. On my machine, I get the expected output at the terminal.

$ python3 --version
Python 3.7.2
$ file mailmerge_*
mailmerge_database.csv: ASCII text
mailmerge_server.conf:  ASCII text
mailmerge_template.txt: HTML document text, UTF-8 Unicode text
$ mailmerge 
>>> message 0
TO: myself@mydomain.com
SUBJECT: Testing mailmerge
FROM: My Self <myself@mydomain.com>
Content-Type: text/html; charset="utf-8"

<!doctype html>
<html lang="pt-br">

<head>
    <title> Page title </title>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <style type="text/css">
        * {
            margin: 0;
            border: 0;
            padding: 0;
            color: red;

            font-family: Arial, Helvetica, sans-serif;
        }
    </style>
</head>

<body>

<h1>ççç ÇÇÇ áéíóú !.</h1>
</body>

</html>

>>> encoding utf-8
>>> sent message 0 DRY RUN
>>> Limit was 1 messages.  To remove the limit, use the --no-limit option.
>>> This was a dry run.  To send messages, use the --no-dry-run option.

Next thing to check is the file encoding of the database.

lightmagic1 commented 5 years ago

mailmerge_database.csv: ASCII text, with CRLF line terminators mailmerge_server.conf: ASCII text, with CRLF line terminators mailmerge_template.txt: HTML document, UTF-8 Unicode text, with CRLF line terminators

awdeorio commented 5 years ago

That looks OK to me. Next thing to try: use the smallest email you can to try and reproduce the bug. Something like this:

mailmerge_template.txt

TO: {{email}}
SUBJECT: Testing mailmerge
FROM: My Self <myself@mydomain.com>
ççç ÇÇÇ áéíóú
lightmagic1 commented 5 years ago

TO: {{email}} SUBJECT: Testing mailmerge FROM: My Self myself@mydomain.com ççç ÇÇÇ áéíóú

mailmerge_template.txt

TO: {{email}}
SUBJECT: Testing mailmerge
FROM: My Self <myself@mydomain.com>
ççç ÇÇÇ áéíóú

Output

λ mailmerge --no-dry-run
>>> message 0
TO: light.magic1@gmail.com
SUBJECT: Testing mailmerge
FROM: My Self <myself@mydomain.com>
ççç ÇÇÇ áéíóú
>>> encoding utf-8
>>> Read SMTP server configuration from mailmerge_server.conf
>>>   host = smtp.gmail.com
>>>   port = 465
>>>   username = light.magic1@gmail.com
>>>   security = SSL/TLS
>>> password for light.magic1@gmail.com on smtp.gmail.com:
>>> sent message 0
>>> No attachments were sent with the emails.
>>> Limit was 1 messages.  To remove the limit, use the --no-limit option.

Message in my inbox:

image

awdeorio commented 5 years ago

Let's try it without mailmerge, just with Python.

$ cp mailmerge_template.txt test.py

Edit test.py to look like this:

print("ççç ÇÇÇ áéíóú")

Now run it.

$ file test.py
test.py: UTF-8 Unicode text
$ python3 test.py 
ççç ÇÇÇ áéíóú
lightmagic1 commented 5 years ago 
print("ççç ÇÇÇ áéíóú")
λ cp mailmerge_template.txt test.py

λ file test.py
test.py: UTF-8 Unicode text, with no line terminators

λ python3 test.py
ççç ÇÇÇ áéíóú
awdeorio commented 5 years ago

Well, looks like that's not it. Is mailmerge using Python2 or Python3?

$ which mailmerge 
/usr/local/bin/mailmerge
$ head /usr/local/bin/mailmerge
#!/usr/local/opt/python/bin/python3.7
# -*- coding: utf-8 -*-
import re
import sys

from mailmerge.__main__ import cli

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(cli())
lightmagic1 commented 5 years ago

Well, looks like that's not it. Is mailmerge using Python2 or Python3?

$ which mailmerge 
/usr/local/bin/mailmerge
$ head /usr/local/bin/mailmerge
#!/usr/local/opt/python/bin/python3.7
# -*- coding: utf-8 -*-
import re
import sys

from mailmerge.__main__ import cli

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(cli())

Anaconda CondaEnv Python 3.7.3

λ pip list
Package       Version
------------- --------
backports.csv 1.0.7
certifi       2019.3.9
chardet       3.0.4
Click         7.0
configparser  3.7.4
future        0.17.1
Jinja2        2.10.1
mailmerge     1.8
MarkupSafe    1.1.1
pip           19.0.3
setuptools    41.0.0
wheel         0.33.1
wincertstore  0.2
λ which mailmerge
/c/Anaconda3/envs/qrcode/Scripts/mailmerge

λ head /c/Anaconda3/envs/qrcode/Scripts/mailmerge
Python 3.7.3 (default, Mar 27 2019, 17:13:21) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32

>>> #!/usr/local/opt/python/bin/python3.7
>>> # -*- coding: utf-8 -*-
>>> import re
>>> import sys
>>> from mailmerge.__main__ import cli
ççç ÇÇÇ áéíóú
>>> if __name__ == '__main__':
...      sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
...      sys.exit(cli())
...
>>> message 0
TO: light.magic1@gmail.com
SUBJECT: Testing mailmerge
FROM: My Self <myself@mydomain.com>
ççç ÇÇÇ áéíóú
>>> encoding utf-8
>>> sent message 0 DRY RUN
>>> No attachments were sent with the emails.
>>> Limit was 1 messages.  To remove the limit, use the --no-limit option.
>>> This was a dry run.  To send messages, use the --no-dry-run option.

That's weird o_o

awdeorio commented 5 years ago

Mailmerge works with either Python2 or Python3. Next thing to try: let's see if Anaconda is causing a problem. We'll install mailmerge from source, using the absolute path to a non-Anaconda python executable. If you're on Windows, use WSL or Cygwin. If you're on OSX, use homebrew to install Python.

Make sure that your Python executable is not Anaconda. Make sure that PYTHONPATH is not set.

$ which python3
/usr/local/bin/python3
$ echo $PYTHONPATH
# <blank>

Get the source code

$ git clone https://github.com/awdeorio/mailmerge.git
Cloning into 'mailmerge'...

Install using the absolute path to Python 3. Install virtual environment with a local copy of pip.

$ /usr/local/bin/python3 -m venv venv
$ source venv/bin/activate
$ which pip
/Users/awdeorio/Downloads/mailmerge/venv/bin/pip
$ pip install -e .

Verify that you have a local copy of mailmerge installed.

$ which mailmerge
/Users/awdeorio/Downloads/mailmerge/venv/bin/mailmerge
$ mailmerge --version
mailmerge, version 1.9

Now, try your test again.

awdeorio commented 5 years ago

Looks like this issue has gone stale. Please reopen if you have more information!