astroidmail / astroid

A graphical threads-with-tags style, lightweight and fast, e-mail client for Notmuch
http://astroidmail.github.io
Other
612 stars 65 forks source link

fail_safe_name () / isspace ()fails on some utf-8 strings #194

Closed benjumanji closed 8 years ago

benjumanji commented 8 years ago

the backtrace from gdb:

#0  0x00007f160ac050de in isspace () from /usr/lib/libc.so.6
#1  0x00000000006a972c in Astroid::UstringUtils::trim_left (str=...) at src/utils/ustring_utils.cc:23
#2  0x00000000006a996e in Astroid::UstringUtils::trim (str=...) at src/utils/ustring_utils.cc:54
#3  0x00000000006a3ae8 in Astroid::Address::fail_safe_name (this=0x7f15ebffe880) at src/utils/address.cc:91
#4  0x00000000005d00be in Astroid::NotmuchThread::get_authors (this=0x7f15e4085bc0, nm_thread=0x7f15e40835c0) at src/db.cc:594
#5  0x00000000005cfafd in Astroid::NotmuchThread::load (this=0x7f15e4085bc0, nm_thread=0x7f15e40835c0) at src/db.cc:510
#6  0x00000000005cf70d in Astroid::NotmuchThread::NotmuchThread (this=0x7f15e4085bc0, t=0x7f15e40835c0, __in_chrg=<optimized out>, __vtt_parm=<optimized out>) at src/db.cc:476
#7  0x000000000063f211 in Astroid::QueryLoader::loader (this=0x2cd3990) at src/modes/thread_index/query_loader.cc:182
#8  0x0000000000643eb6 in std::__invoke_impl<void, void (Astroid::QueryLoader::* const&)(), Astroid::QueryLoader*>(std::__invoke_memfun_deref, void (Astroid::QueryLoader::* const&)(), Astroid::QueryLoader*&&) (__f=@0x2d21810: (void (Astroid::QueryLoader::*)(Astroid::QueryLoader * const)) 0x63ef72 <Astroid::QueryLoader::loader()>, 
    __t=<unknown type in /usr/local/bin/astroid, CU 0x534dcb, DIE 0x5678ec>) at /usr/include/c++/6.1.1/functional:235
#9  0x0000000000643e43 in std::__invoke<void (Astroid::QueryLoader::* const&)(), Astroid::QueryLoader*>(void (Astroid::QueryLoader::* const&)(), Astroid::QueryLoader*&&) (
    __fn=@0x2d21810: (void (Astroid::QueryLoader::*)(Astroid::QueryLoader * const)) 0x63ef72 <Astroid::QueryLoader::loader()>, 
    __args#0=<unknown type in /usr/local/bin/astroid, CU 0x534dcb, DIE 0x5678ec>) at /usr/include/c++/6.1.1/functional:260
#10 0x0000000000643df4 in std::_Mem_fn_base<void (Astroid::QueryLoader::*)(), true>::operator()<Astroid::QueryLoader*>(Astroid::QueryLoader*&&) const (this=0x2d21810, 
    __args#0=<unknown type in /usr/local/bin/astroid, CU 0x534dcb, DIE 0x5678ec>) at /usr/include/c++/6.1.1/functional:613
#11 0x0000000000643dbf in std::_Bind_simple<std::_Mem_fn<void (Astroid::QueryLoader::*)()> (Astroid::QueryLoader*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x2d21808)
    at /usr/include/c++/6.1.1/functional:1400
#12 0x0000000000643d0d in std::_Bind_simple<std::_Mem_fn<void (Astroid::QueryLoader::*)()> (Astroid::QueryLoader*)>::operator()() (this=0x2d21808) at /usr/include/c++/6.1.1/functional:1389
#13 0x0000000000643cec in std::thread::_State_impl<std::_Bind_simple<std::_Mem_fn<void (Astroid::QueryLoader::*)()> (Astroid::QueryLoader*)> >::_M_run() (this=0x2d21800)
    at /usr/include/c++/6.1.1/thread:196
#14 0x00007f160b77435f in std::execute_native_thread_routine (__p=0x2d21800) at /build/gcc-multilib/src/gcc/libstdc++-v3/src/c++11/thread.cc:83
#15 0x00007f160af80454 in start_thread () from /usr/lib/libpthread.so.0
#16 0x00007f160acc17df in clone () from /usr/lib/libc.so.6

Maybe relevant?

$ uname -a
Linux greyskull 4.7.0-1-ARCH #1 SMP PREEMPT Mon Aug 8 22:05:58 CEST 2016 x86_64 GNU/Linux
$ locale
LANG=en_GB.utf8
LC_CTYPE="en_GB.utf8"
LC_NUMERIC="en_GB.utf8"
LC_TIME="en_GB.utf8"
LC_COLLATE="en_GB.utf8"
LC_MONETARY="en_GB.utf8"
LC_MESSAGES="en_GB.utf8"
LC_PAPER="en_GB.utf8"
LC_NAME="en_GB.utf8"
LC_ADDRESS="en_GB.utf8"
LC_TELEPHONE="en_GB.utf8"
LC_MEASUREMENT="en_GB.utf8"
LC_IDENTIFICATION="en_GB.utf8"
LC_ALL=

Built from source. It dies on startup. Here is the log:

[info ] (18:04:11): logging to: crash.log                                                                                                                                                      
[debug] (18:04:11): loading gtk..                                                                                                                                                              
[debug] (18:04:11): HOME: /home/ben                                                                                                                                                            
[info ] (18:04:11): cf: loading: "/home/ben/.config/astroid/config"                                                                                                                            
[info ] (18:04:11): cf: version: 6                                                                                                                                                             
[info ] (18:04:11): notmuch db: /home/ben/mail                                                                                                                                                 
[info ] (18:04:11): date: init.                                                                                                                                                                
[info ] (18:04:11): searches: loading saved searches..                                                                                                                                         
[debug] (18:04:11): searches: loading history..                                                                                                                                                
[info ] (18:04:11): ac: initializing accounts..                                                                                                                                                
[info ] (18:04:11): ac: setup account: charlie for Charlie Root (default: 1)                                                                                                                   
[info ] (18:04:11): plugins: starting manager..                                                                                                                                                
[debug] (18:04:11): plugins: adding path: /usr/local/share/astroid/plugins                                                                                                                     
[debug] (18:04:11): plugin: adding path: /home/ben/.config/astroid/plugins                                                                                                                     
[debug] (18:04:11): plugins: refreshing..                                                                                                                                                      
[debug] (18:04:11): plugins: found 0 plugins.                                                                                                                                                  
[info ] (18:04:11): global actions: set up.                                                                                                                                                    
[info ] (18:04:11): poll: setting up.                                                                                                                                                          
[debug] (18:04:11): poll: interval: 60                                                                                                                                                         
[debug] (18:04:11): poll: requested..                                                                                                                                                          
[warn ] (18:04:11): astroid: starting a new window..                                                                                                                                           
[info ] (18:04:11): poll: polling: /home/ben/.config/astroid/poll.sh                                                                                                                           
[info ] (18:04:11): db: open db read-only, waiting for lock..                                                                                                                                  
[debug] (18:04:11): db: read-only got lock.                                                                                                                                                    
[debug] (18:04:11): db: open time: 2.307 ms.                                                                                                                                                   
[debug] (18:04:11): poll: revision before poll: 4996                                                                                                                                           
[ERROR] (18:04:11): poll: poll script does not exist or is not a regular file.                                                                                                                 
[info ] (18:04:11): db: closing db.                                                                                                                                                            
[debug] (18:04:11): db: ro: waiting for lock to close..                                                                                                                                        
[debug] (18:04:11): db: ro: closing..                                                                                                                                                          
[info ] (18:04:11): db: open db read-only, waiting for lock..                                                                                                                                  
[debug] (18:04:11): db: read-only got lock.                                                                                                                                                    
[debug] (18:04:11): db: open time: 0.213 ms.                                                                                                                                                   
[info ] (18:04:11): db: loaded 6 tags.                                                                                                                                                         
[info ] (18:04:11): db: closing db.                                                                                                                                                            
[debug] (18:04:11): db: ro: waiting for lock to close..                                                                                                                                        
[debug] (18:04:11): db: ro: closing..                                                                                                                                                          
[debug] (18:04:11): mw: init, id: 1                                                                                                                                                            
[info ] (18:04:11): astroid: got query: inbox: tag:inbox                                                                                                                                       
[info ] (18:04:11): re: using installed resource: /usr/local/share/astroid/ui/no-mail.png                                                                                                      
[debug] (18:04:11): pm: add pane                                                                                                                                                               
[info ] (18:04:11): db: open db read-only, waiting for lock..                                                                                                                                  
[debug] (18:04:11): db: read-only got lock.      
[debug] (18:04:11): db: open time: 0.213 ms.                                                                                                                                                   
[info ] (18:04:11): db: loaded 6 tags.                                                                                                                                                         
[info ] (18:04:11): db: closing db.                                                                                                                                                            
[debug] (18:04:11): db: ro: waiting for lock to close..                                                                                                                                        
[debug] (18:04:11): db: ro: closing..                                                                                                                                                          
[debug] (18:04:11): mw: init, id: 1                                                                                                                                                            
[info ] (18:04:11): astroid: got query: inbox: tag:inbox                                                                                                                                       
[info ] (18:04:11): re: using installed resource: /usr/local/share/astroid/ui/no-mail.png                                                                                                      
[debug] (18:04:11): pm: add pane                                                                                                                                                               
[info ] (18:04:11): db: open db read-only, waiting for lock..                                                                                                                                  
[debug] (18:04:11): db: read-only got lock.                                                                                                                                                    
[debug] (18:04:11): db: open time: 0.765 ms.                                                                                                                                                   
[debug] (18:04:11): mw: does not have focus, will not grab modal.                                                                                                                              
[debug] (18:04:11): mw: does not have focus, will not grab modal.                                                                                                                              
[debug] (18:04:11): mw: does not have focus, will not grab modal.                                                                                                                              
[info ] (18:04:11): poll: emitted poll state: 0                                                                                                                                                
[info ] (18:04:11): poll: emitted poll state: 0                                                                                                                                                
[debug] (18:04:11): ql: refresh stats..                                                                                                                                                        
[info ] (18:04:11): db: open db read-only, waiting for lock..                                                                                                                                  
[debug] (18:04:11): db: read-only got lock.                                                                                                                                                    
[debug] (18:04:11): db: open time: 0.408 ms.                                                                                                                                                   
[info ] (18:04:11): db: closing db.                                                                                                                                                            
[debug] (18:04:11): db: ro: waiting for lock to close..                                                                                                                                        
[debug] (18:04:11): db: ro: closing..                                                                                                                                                          
[debug] (18:04:11): ti: got refresh stats.                                                                                                                                                     
[debug] (18:04:11): ql: loaded 100 threads.                                                                                                                                                    
[debug] (18:04:11): ql: loaded 200 threads.                                                                                                                                                    
[debug] (18:04:11): ql: loaded 300 threads.                                                                                                                                                    
[debug] (18:04:11): ql: loaded 400 threads.

Then it dies with a SIGSEV.

Thoughts?

gauteh commented 8 years ago

Hm, weird - hard to see how that code can result in a segfault. Does it happen consistently?

Could you try pulling from master again and see if it still happens?

Regards, Gaute

benjumanji commented 8 years ago

Yes, completely consistently, with HEAD (v0.6-29-g3e892d3f). I know it's some imput in my mail that's provoking it, because when I had only imported a small amount of mail, it worked flawlessly. Is there any way to figure out what input it's choking on?

gauteh commented 8 years ago

Benjamin Edwards writes on august 14, 2016 23:13:

Yes, completely consistently, with HEAD (v0.6-29-g3e892d3f). I know it's some imput in my mail that's provoking it, because when I had only imported a small amount of mail, it worked flawlessly. Is there any way to figure out what input it's choking on?

You could try running it in gdb, and see if the backtrace ('bt') gives more information than the core. Also, you could put the following debug code into src/utils/address.cc:

diff --git a/src/utils/address.cc b/src/utils/address.cc
index 9e0cfca..b406bdd 100644
--- a/src/utils/address.cc
+++ b/src/utils/address.cc
@@ -88,6 +88,8 @@ namespace Astroid {
   }

   ustring Address::fail_safe_name () {
+    log << debug << "getting fsafe name: " << _name << endl;
+
     UstringUtils::trim (_name);
     if (_name.length () == 0) {
       return _email;
gauteh commented 8 years ago

Sorry, I just saw that you already used gdb. If the debug info from address doesn't work, I'll put in debug messages in the trace of functions so thta it is possible to find the message-id/message-file.

benjumanji commented 8 years ago

Attached log. Spoiler: 󾔰 C0ngratuLations 󾔰 is the value that makes it puke. I guess it's probably not valid utf-8? Anyway.. the raw bytes:

f3 be 94 b0 20 43 30 6e 67 72 61 74 75 4c 61 74 69 6f 6e 73 20 f3 be 94 b0
gauteh commented 8 years ago

Benjamin Edwards writes on august 16, 2016 0:37:

Attached log. Spoiler: 󾔰 C0ngratuLations 󾔰 is the value that makes it puke. I guess it's probably not valid utf-8? Anyway.. the raw bytes:

f3 be 94 b0 20 43 30 6e 67 72 61 74 75 4c 61 74 69 6f 6e 73 20 f3 be 94 b0

That looks like a spam email, would it be possible to send the actual email (gzipped or something) so that I could include it in the test-suite?

I am unsure if this is an astroid bug, the header is fetched through notmuch, which uses gmime if possible. What notmuch version do you have?

-gaute

gauteh commented 8 years ago

Gaute Hope writes on august 16, 2016 8:39:

Benjamin Edwards writes on august 16, 2016 0:37:

Attached log. Spoiler: 󾔰 C0ngratuLations 󾔰 is the value that makes it puke. I guess it's probably not valid utf-8? Anyway.. the raw bytes:

f3 be 94 b0 20 43 30 6e 67 72 61 74 75 4c 61 74 69 6f 6e 73 20 f3 be 94 b0

That looks like a spam email, would it be possible to send the actual email (gzipped or something) so that I could include it in the test-suite?

I am unsure if this is an astroid bug, the header is fetched through notmuch, which uses gmime if possible. What notmuch version do you have?

-gaute

I am pretty sure the output is valid utf-8, python can handle it at least. I think it is the isspace test that doesn't handle it. Would be great with a complete test email.

benjumanji commented 8 years ago

Is there another patch you could offer that would make tracking down the path to the email easy?

Very happy to send the complete email.

On Tue, 16 Aug 2016, 08:26 Gaute Hope, notifications@github.com wrote:

Gaute Hope writes on august 16, 2016 8:39:

Benjamin Edwards writes on august 16, 2016 0:37:

Attached log. Spoiler: [image: [?]] C0ngratuLations [image: [?]] is the value that makes it puke. I guess it's probably not valid utf-8? Anyway.. the raw bytes:

f3 be 94 b0 20 43 30 6e 67 72 61 74 75 4c 61 74 69 6f 6e 73 20 f3 be 94
b0

That looks like a spam email, would it be possible to send the actual email (gzipped or something) so that I could include it in the test-suite?

I am unsure if this is an astroid bug, the header is fetched through notmuch, which uses gmime if possible. What notmuch version do you have?

-gaute

I am pretty sure the output is valid utf-8, python can handle it at least. I think it is the isspace test that doesn't handle it. Would be great with a complete test email.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/astroidmail/astroid/issues/194#issuecomment-240024236, or mute the thread https://github.com/notifications/unsubscribe-auth/AARufVb3w-pN5w4RHK2I70i-t94N0EYcks5qgWY9gaJpZM4Jj8H- .

gauteh commented 8 years ago

Yes, but not super-easy, maybe you could just grep for the string?

gauteh commented 8 years ago

By the way, your crash.txt log (while very useful!) does show all the authors of your emails, so you might want to delete it and not have it here publicly.

benjumanji commented 8 years ago

Ok, I was just being lazy. I'll have a root about for it when I am home from work.

I'll also delete that log, good point!

gauteh commented 8 years ago

Ok! Sure :+1:

benjumanji commented 8 years ago

Voila!

spam.gz

gauteh commented 8 years ago

Benjamin Edwards writes on august 16, 2016 20:46:

Voila!

Alright, please try latest master.

benjumanji commented 8 years ago

It worked! I can now open all my mail. Thanks for the quick turn around!

gauteh commented 8 years ago

Benjamin Edwards writes on august 16, 2016 22:07:

It worked! I can now open all my mail. Thanks for the quick turn around!

Great!