search

taers232c / GAMADV-XTD3

Command line tool to manage Google Workspace
691 stars 86 forks source link

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 1643: invalid start byte #331

Closed sborrill closed 1 month ago

sborrill commented 1 year ago

On GAMADV-XTD3 6.42.10, I was running: gam all users forward threads to RECIPIENT query "\"TEXT SEARCH\"" max_to_forward 0 doit

After a while, it failed with:

Traceback (most recent call last):
  File "__init__.py", line 67320, in ProcessGAMCommand
  File "__init__.py", line 62995, in setForward
  File "__init__.py", line 61010, in forwardMessagesThreads
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 1643: invalid start byte

I have tracked this down to the specific message causing it. It is only the forwarding action that seems to trigger this. print message even with showbody and headers all does not.

taers232c commented 1 year ago

Send me a Meet/Zoom invitation

Ross

sborrill commented 1 year ago

What timezone are you in and so what time UTC is best for you?

taers232c commented 1 year ago

I'm in California (PST UTC-8) and am generally available starting at 7:30AM. It's now 8:09

Ross

On Tue, Feb 28, 2023 at 7:48 AM sborrill @.***> wrote:

What timezone are you in and so what time UTC is best for you?

— Reply to this email directly, view it on GitHub https://github.com/taers232c/GAMADV-XTD3/issues/331#issuecomment-1448416572, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCTYL4FISIRA7DWEDJTBBTWZYM6DANCNFSM6AAAAAAVKNUYYM . You are receiving this because you commented.Message ID: @.***>

-- Ross Scroggs @.***

taers232c commented 1 year ago

I think that I've fixed this. The problem seems to be that there were Windows Latin-1 curly quotes in the subject. Is it possible to send ross.scroggs @ rdschool.org a test message with the same subject? I can use that to check my fix.

Thanks

sborrill commented 1 year ago

The subject appear to be just Fw: Miss Baines, nothing else, so probably not worth sending.

taers232c commented 1 year ago

Test with 6.50.01 and let me know the result.

Thanks

On Thu, Mar 2, 2023 at 8:19 AM sborrill @.***> wrote:

The subject appear to be just Fw: Miss Baines, nothing else, so probably not worth sending.

— Reply to this email directly, view it on GitHub https://github.com/taers232c/GAMADV-XTD3/issues/331#issuecomment-1452146398, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCTYL5LM73AVD5VICDLYITW2DCCVANCNFSM6AAAAAAVKNUYYM . You are receiving this because you commented.Message ID: @.***>

-- Ross Scroggs @.***

sborrill commented 1 year ago

Fault is still there with 6.50.02

taers232c commented 1 year ago

Can the problematic message be forwarded to ross.scroggs @ rdschool.org http://rdschool.org/ from gmail.com http://gmail.com/?

Ross

Ross Scroggs @.***

On Mar 3, 2023, at 2:22 AM, sborrill @.***> wrote:

Fault is still there with 6.50.02

— Reply to this email directly, view it on GitHub https://github.com/taers232c/GAMADV-XTD3/issues/331#issuecomment-1453298361, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCTYL5W2PGVLN6VEQT6PRLW2HA4RANCNFSM6AAAAAAVKNUYYM. You are receiving this because you commented.

taers232c commented 1 year ago

I might not need the message, send me the trap traceback from 6.50.07; that will let me know what line in the code is the problem.

Ross

On Fri, Mar 3, 2023 at 6:10 AM Ross Scroggs @.***> wrote:

Can the problematic message be forwarded to ross.scroggs @ rdschool.org from gmail.com?

Ross

Ross Scroggs @.***

On Mar 3, 2023, at 2:22 AM, sborrill @.***> wrote:

Fault is still there with 6.50.02

— Reply to this email directly, view it on GitHub https://github.com/taers232c/GAMADV-XTD3/issues/331#issuecomment-1453298361, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCTYL5W2PGVLN6VEQT6PRLW2HA4RANCNFSM6AAAAAAVKNUYYM . You are receiving this because you commented.Message ID: @.***>

-- Ross Scroggs @.***

sborrill commented 1 year ago

Traceback from 6.50.02 was: File "init.py", line 67639, in ProcessGAMCommand File "init.py", line 63297, in setForward File "init.py", line 61327, in forwardMessagesThreads UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 1643: invalid start byte

6.50.09 does not give a trace back, rather it says: Recipient: sar1@REDACTED, Message: 1784be1bb0ac2fb4, Not Forwarded: 'utf-8' codec can't decode byte 0x92 in position 1643: invalid start byte

taers232c commented 1 year ago

I'm now trapping the error; add this to the command and report: altcharset latin1

Ross

On Wed, Mar 8, 2023 at 2:31 AM sborrill @.***> wrote:

Traceback from 6.50.02 was: File "init.py", line 67639, in ProcessGAMCommand File "init.py", line 63297, in setForward File "init.py", line 61327, in forwardMessagesThreads UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 1643: invalid start byte

6.50.09 does not give a trace back, rather it says: Recipient: @.***, Message: 1784be1bb0ac2fb4, Not Forwarded: 'utf-8' codec can't decode byte 0x92 in position 1643: invalid start byte

— Reply to this email directly, view it on GitHub https://github.com/taers232c/GAMADV-XTD3/issues/331#issuecomment-1459953872, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCTYL2AFOGEUIWA3E3L3KDW3BNWVANCNFSM6AAAAAAVKNUYYM . You are receiving this because you commented.Message ID: @.***>

-- Ross Scroggs @.***

sborrill commented 1 year ago

Like this?

Command: gam user ORIGINAL forward message to DESTINATION ids 1784be1bb0ac2fb4 altcharset >>>latin1<<<

ERROR: Invalid argument

taers232c commented 1 year ago

Yes. But it fails for me as well, so it's bug fixing time.

On Wed, Mar 8, 2023 at 7:28 AM sborrill @.***> wrote:

Like this?

Command: gam user ORIGINAL forward message to DESTINATION ids 1784be1bb0ac2fb4 altcharset >>>latin1<<<

ERROR: Invalid argument

— Reply to this email directly, view it on GitHub https://github.com/taers232c/GAMADV-XTD3/issues/331#issuecomment-1460333704, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCTYL4GKYDVAT37GSFYHALW3CQRFANCNFSM6AAAAAAVKNUYYM . You are receiving this because you commented.Message ID: @.***>

-- Ross Scroggs @.***

taers232c commented 1 year ago

Bug fixed in 6.50.10

On Wed, Mar 8, 2023 at 7:43 AM Ross Scroggs @.***> wrote:

Yes. But it fails for me as well, so it's bug fixing time.

On Wed, Mar 8, 2023 at 7:28 AM sborrill @.***> wrote:

Like this?

Command: gam user ORIGINAL forward message to DESTINATION ids 1784be1bb0ac2fb4 altcharset >>>latin1<<<

ERROR: Invalid argument

— Reply to this email directly, view it on GitHub https://github.com/taers232c/GAMADV-XTD3/issues/331#issuecomment-1460333704, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCTYL4GKYDVAT37GSFYHALW3CQRFANCNFSM6AAAAAAVKNUYYM . You are receiving this because you commented.Message ID: @.***>

-- Ross Scroggs @.***

-- Ross Scroggs @.***

sborrill commented 1 year ago

Traceback (most recent call last): File "init.py", line 67710, in ProcessGAMCommand File "init.py", line 63367, in setForward File "init.py", line 61411, in forwardMessagesThreads File "email\message.py", line 208, in as_bytes File "email\generator.py", line 116, in flatten File "email\generator.py", line 181, in _write File "email\generator.py", line 218, in _dispatch File "email\generator.py", line 276, in _handle_multipart File "email\generator.py", line 116, in flatten File "email\generator.py", line 181, in _write File "email\generator.py", line 218, in _dispatch File "email\generator.py", line 436, in _handle_text File "email\generator.py", line 253, in _handle_text File "email\generator.py", line 155, in _write_lines File "email\generator.py", line 410, in write UnicodeEncodeError: 'ascii' codec can't encode character '\x92' in position 38: ordinal not in range(128)

taers232c commented 1 year ago

6.50.11 should handle this trap.

On Wed, Mar 8, 2023 at 8:32 AM sborrill @.***> wrote:

Traceback (most recent call last): File "init.py", line 67710, in ProcessGAMCommand File "init.py", line 63367, in setForward File "init.py", line 61411, in forwardMessagesThreads File "email\message.py", line 208, in as_bytes File "email\generator.py", line 116, in flatten File "email\generator.py", line 181, in _write File "email\generator.py", line 218, in _dispatch File "email\generator.py", line 276, in _handle_multipart File "email\generator.py", line 116, in flatten File "email\generator.py", line 181, in _write File "email\generator.py", line 218, in _dispatch File "email\generator.py", line 436, in _handle_text File "email\generator.py", line 253, in _handle_text File "email\generator.py", line 155, in _write_lines File "email\generator.py", line 410, in write UnicodeEncodeError: 'ascii' codec can't encode character '\x92' in position 38: ordinal not in range(128)

— Reply to this email directly, view it on GitHub https://github.com/taers232c/GAMADV-XTD3/issues/331#issuecomment-1460463857, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCTYLY4JPSOC5UMQQPGRX3W3CYAJANCNFSM6AAAAAAVKNUYYM . You are receiving this because you commented.Message ID: @.***>

-- Ross Scroggs @.***

sborrill commented 1 year ago

Behaviour is the same with 6.50.11 (same trap)

taers232c commented 1 year ago

Show the trap details. Thanks

Ross Scroggs @.***

On Mar 9, 2023, at 12:52 AM, sborrill @.***> wrote:

Behaviour is the same with 6.50.11 (same trap)

— Reply to this email directly, view it on GitHub https://github.com/taers232c/GAMADV-XTD3/issues/331#issuecomment-1461591164, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCTYLZLFFHGQEAK6SHX2IDW3GK6XANCNFSM6AAAAAAVKNUYYM. You are receiving this because you commented.

sborrill commented 1 year ago

Absolutely identical to previous:

Traceback (most recent call last): File "init.py", line 67710, in ProcessGAMCommand File "init.py", line 63367, in setForward File "init.py", line 61411, in forwardMessagesThreads File "email\message.py", line 208, in as_bytes File "email\generator.py", line 116, in flatten File "email\generator.py", line 181, in _write File "email\generator.py", line 218, in _dispatch File "email\generator.py", line 276, in _handle_multipart File "email\generator.py", line 116, in flatten File "email\generator.py", line 181, in _write File "email\generator.py", line 218, in _dispatch File "email\generator.py", line 436, in _handle_text File "email\generator.py", line 253, in _handle_text File "email\generator.py", line 155, in _write_lines File "email\generator.py", line 410, in write UnicodeEncodeError: 'ascii' codec can't encode character '\x92' in position 38: ordinal not in range(128)

C:\GAMADV-XTD3>gam version GAMADV-XTD3 6.50.11 - https://github.com/taers232c/GAMADV-XTD3 - pyinstaller Ross Scroggs ross.scroggs@gmail.com Python 3.11.2 64-bit final Windows 10 10.0.20348 SP0 Multiprocessor Free AMD64

taers232c commented 1 year ago

6.50.12 now handles the trap and gives an error message. You need to get the original recipient to do a "show original" on the message and look for a line like this:

Content-Type: text/plain; charset="UTF-8"; format=flowed; delsp=yes

Try that charset in altcharset

Ross

On Thu, Mar 9, 2023 at 5:53 AM sborrill @.***> wrote:

Absolutely identical to previous:

Traceback (most recent call last): File "init.py", line 67710, in ProcessGAMCommand File "init.py", line 63367, in setForward File "init.py", line 61411, in forwardMessagesThreads File "email\message.py", line 208, in as_bytes File "email\generator.py", line 116, in flatten File "email\generator.py", line 181, in _write File "email\generator.py", line 218, in _dispatch File "email\generator.py", line 276, in _handle_multipart File "email\generator.py", line 116, in flatten File "email\generator.py", line 181, in _write File "email\generator.py", line 218, in _dispatch File "email\generator.py", line 436, in _handle_text File "email\generator.py", line 253, in _handle_text File "email\generator.py", line 155, in _write_lines File "email\generator.py", line 410, in write UnicodeEncodeError: 'ascii' codec can't encode character '\x92' in position 38: ordinal not in range(128)

C:\GAMADV-XTD3>gam version GAMADV-XTD3 6.50.11 - https://github.com/taers232c/GAMADV-XTD3 - pyinstaller Ross Scroggs @.*** Python 3.11.2 64-bit final Windows 10 10.0.20348 SP0 Multiprocessor Free AMD64

— Reply to this email directly, view it on GitHub https://github.com/taers232c/GAMADV-XTD3/issues/331#issuecomment-1462106022, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCTYL7H64LTARLXOAVDS23W3HOD3ANCNFSM6AAAAAAVKNUYYM . You are receiving this because you commented.Message ID: @.***>

-- Ross Scroggs @.***

sborrill commented 1 year ago

It seems that whatever altcharset you specify, it still says "'ascii' codec can't encode character". Is that right?

The charset is iso-8859-1, but specifying that to altcharset makes no difference.

The original mail has &#8217; in its text/html part. The text/plain part has 0xC292