search

RWilton / Arioc

Arioc: GPU-accelerated DNA short-read alignment
BSD 3-Clause "New" or "Revised" License
55 stars 8 forks source link

pthread_mutex_unlock returned 22 for AriocE #17

Closed a7032018 closed 3 years ago

a7032018 commented 3 years ago

Hi, I was testing Arioc with provided sample dataset. I created the index for yeast reference and saw a warning message for both gapped and ungapped indexes (ApplicationException ([0x00079555] CppCommon/RaiiMutex.cpp 107): pthread_mutex_unlock returned 22). Below is the log for making gapped reference index. I was able to generate SAM files for both unpaired and paired reads. Just want to know if the pthread warning is critical and may affect the result. I am planning to use Arioc for a gigantic dataset, and hopefully it won't affect the result. During the compilation, I did receive the warning message but was able to get the binary files. I am wondering if it is the reason I got the pthread error. I have CUDA 11.2 installed in Ubuntu 14.04 LTS

/usr/local/cuda/include/cub/util_cpp_dialect.cuh:129:13: warning: CUB requires C++14. Please pass -std=c++14 to your compiler. Define CUB_IGNORE_DEPRECATED_CPP_DIALECT to suppress this message. CUB_COMPILER_DEPRECATION(C++14, pass -std=c++14 to your compiler);"

Thanks!

162707206 [000131a1] AriocE v1.43.2991.21102 (release) +2021-04-12T13:07:35 162707206 [000131a1] Copyright (c) 2015-2021 Johns Hopkins University. All righ ts reserved. 162707206 [000131a1] compiled : 2021-04-14T15:55:58 (GNU g++ v5.4.0 ) 162707206 [000131a1] data type sizes : int=4 long=8 *=8 Jvalue5=5 Jvalue8= 8 JtableHeader=5 (little-endian) 162707206 [000131a1] computer name : CBU5143610 162707206 [000131a1] executable file : /home/apps/arioc/bin/AriocE 162707206 [000131a1] configuration file : /home/apps/arioc/bin/R/S_ce revisiae/AriocE.gapped.cfg 162707219 [000131a1] AriocE::encodeR: encoding 17 files (34 sequences) (112 CPU threads available)... 162709417 [000131a1] AriocE::encodeR: encoded 17 files (34 sequences) 162711552 [000131a1] AriocE::computeJlistSizes: sorting C list (1073741824 hash keys)... 163102797 [000131a1] AriocE::computeJlistSizes: sorted C list 163105773 [000131a1] AriocE::computeJlistSizes: m_nJ=24485122 (220907+24264215) m_iCshort=5225 m_iCzero=22744289 163105773 [000131a1] AriocE::computeJlistSizes: log2(nJ) # J-lists 163105773 [000131a1] AriocE::computeJlistSizes: 0 21950821 163105773 [000131a1] AriocE::computeJlistSizes: 1 704636 163105773 [000131a1] AriocE::computeJlistSizes: 2 43501 163105773 [000131a1] AriocE::computeJlistSizes: 3 23287 163105773 [000131a1] AriocE::computeJlistSizes: 4 16819 163105773 [000131a1] AriocE::computeJlistSizes: 5 4622 163105773 [000131a1] AriocE::computeJlistSizes: 6 573 163105773 [000131a1] AriocE::computeJlistSizes: 7 26 163105773 [000131a1] AriocE::computeJlistSizes: 9 4 163112301 [000131a1] AriocE::computeJlistSizes: completed: J table uses 24490347 elements for 24485122 J values (0.021% for list counts in J lists) 163114647 [000131a1] AriocE::buildJlists: building J lists for 17 files (34 sequences) (112 CPU threads available)... 163116165 [000131a1] AriocE::buildJlists: built J lists for 17 files (34 sequences) 163116190 [000131a1] AriocE::sortJcpu: sorting J lists on 112 worker threads... 163206607 [000131a1] AriocE::sortJcpu: J list sort complete 163238533 [000131a1] AriocE::validateHJ: start 163247005 [000131a1] AriocE::validateHJ: H table uses 22744289/1073741824 ( 2.118%) hash keys 163247005 [000131a1] AriocE::validateHJ: J table contains 24490347 J values (list counts in lists: 5225 (0.021%)) 163247005 [000131a1] AriocE::validateHJ: maximum nJ=892 for hash key 0x295C5954 163247005 [000131a1] AriocE::validateHJ: completed 163247006 [000131a1] AriocE::validateSubIds: start 163249042 [000131a1] AriocE::validateSubIds: validated 1073741824 hash values 163249042 [000131a1] AriocE::validateSubIds: # subIds # hash keys avg J-list size 163249042 [000131a1] AriocE::validateSubIds: 0 0 0.0 163249042 [000131a1] AriocE::validateSubIds: 1 22193080 1.0 163249042 [000131a1] AriocE::validateSubIds: 2 438996 2.1 163249042 [000131a1] AriocE::validateSubIds: 3 45131 3.1 163249042 [000131a1] AriocE::validateSubIds: 4 12937 4.3 163249042 [000131a1] AriocE::validateSubIds: 5 6546 5.7 163249042 [000131a1] AriocE::validateSubIds: 6 5218 7.4 163249042 [000131a1] AriocE::validateSubIds: 7 5482 9.1 163249042 [000131a1] AriocE::validateSubIds: 8 8909 12.1 163249042 [000131a1] AriocE::validateSubIds: 9 5069 13.6 163249042 [000131a1] AriocE::validateSubIds: 10 3334 16.9 163249042 [000131a1] AriocE::validateSubIds: 11 3647 20.4 163249042 [000131a1] AriocE::validateSubIds: 12 7642 28.0 163249042 [000131a1] AriocE::validateSubIds: 13 6322 21.2 163249042 [000131a1] AriocE::validateSubIds: 14 1197 39.2 163249042 [000131a1] AriocE::validateSubIds: 15 359 56.8 163249042 [000131a1] AriocE::validateSubIds: 16 416 84.3 163249042 [000131a1] AriocE::validateSubIds: 17 4 474.5 163249042 [000131a1] AriocE::validateSubIds: completed 163249042 [000131a1] AriocE::finalizeJ: finalizing J lists on 112 worker threads... 163309783 [000131a1] AriocE::finalizeJ: completed 163316296 [000131a1] RaiiMutex::~RaiiMutex: pthread_mutex_destroy returned 16 163316672 [000136c3] ApplicationException ([0x00079555] CppCommon/RaiiMutex.cpp 107): pthread_mutex_unlock returned 22

RWilton commented 3 years ago

Thank you for reporting the compile-time warning from CUDA CUB implementation and the runtime error in AriocE.

As far as I know, the two messages you describe are unrelated to each other. I have not seen the compile-time warning message before, but I have never built Arioc with GNU g++ v5.4.0 either.

As for the pthreads error emitted by AriocE: yes, this error should be considered critical in that the encoded index files might be left in an indeterminate state at the point where the AriocE application terminates abnormally. However, because the GNU pthreads implementation has evolved over the years, my starting point with troubleshooting pthreads-related errors is always to compile using the most recent available GNU g++ distribution I can install that remains compatible with the Nvidia CUDA compiler. (GNU g++ v5.4.0 dates back to 2015...)

So to troubleshoot both messages: would it be possible for you to recompile everything with a more recent GNU C++ compiler?

-- rw

a7032018 commented 3 years ago

I compiled the src by GCC 9 and G++ 9. Same warning message pops up. Just realize a solution to change flag from "GPPFLAGS = -std=c++11" and "NVCCFLAGS = -std=c++11" to "GPPFLAGS = -std=c++14" and "NVCCFLAGS = -std=c++14" in makefile solve the compilation issue. However, pthread issue is not gone when running AriocE.

124607895 [00012ae8] AriocE v1.43.2991.21102 (release) +2021-04-12T13:07:35 124607896 [00012ae8] Copyright (c) 2015-2021 Johns Hopkins University. All rights reserved. 124607896 [00012ae8] compiled : 2021-04-15T12:20:27 (GNU g++ v9.3.0) 124607896 [00012ae8] data type sizes : int=4 long=8 *=8 Jvalue5=5 Jvalue8=8 JtableHeader=5 (little-endian) 124607896 [00012ae8] computer name : CBU5143610 124607896 [00012ae8] executable file : /home/apps/arioc/bin/AriocE 124607896 [00012ae8] configuration file : /home/apps/arioc/R/S_cerevisiae/AriocE.nongapped.cfg 124607912 [00012ae8] AriocE::encodeR: encoding 17 files (34 sequences) (112 CPU threads available)... 124611780 [00012ae8] AriocE::encodeR: encoded 17 files (34 sequences) 124617813 [00012ae8] AriocE::computeJlistSizes: sorting C list (1073741824 hash keys)... 125038820 [00012ae8] AriocE::computeJlistSizes: sorted C list 125042024 [00012ae8] AriocE::computeJlistSizes: m_nJ=24482946 (85527+24397419) m_iCshort=2248 m_iCzero=23028745 125042024 [00012ae8] AriocE::computeJlistSizes: log2(nJ) # J-lists 125042024 [00012ae8] AriocE::computeJlistSizes: 0 22280497 125042024 [00012ae8] AriocE::computeJlistSizes: 1 677079 125042024 [00012ae8] AriocE::computeJlistSizes: 2 33216 125042024 [00012ae8] AriocE::computeJlistSizes: 3 21701 125042024 [00012ae8] AriocE::computeJlistSizes: 4 14004 125042024 [00012ae8] AriocE::computeJlistSizes: 5 2110 125042024 [00012ae8] AriocE::computeJlistSizes: 6 138 125049394 [00012ae8] AriocE::computeJlistSizes: completed: J table uses 24485194 elements for 24482946 J values (0.009% for list counts in J lists) 125057318 [00012ae8] AriocE::buildJlists: building J lists for 17 files (34 sequences) (112 CPU threads available)... 125105044 [00012ae8] AriocE::buildJlists: built J lists for 17 files (34 sequences) 125105117 [00012ae8] AriocE::sortJcpu: sorting J lists on 112 worker threads... 125224910 [00012ae8] AriocE::sortJcpu: J list sort complete 125338188 [00012ae8] AriocE::validateHJ: start 125403661 [00012ae8] AriocE::validateHJ: H table uses 23028745/1073741824 ( 2.145%) hash keys 125403662 [00012ae8] AriocE::validateHJ: J table contains 24485194 J values (list counts in lists: 2248 (0.009%)) 125403662 [00012ae8] AriocE::validateHJ: maximum nJ=82 for hash key 0x1CD1186B 125403663 [00012ae8] AriocE::validateHJ: completed 125403663 [00012ae8] AriocE::validateSubIds: start 125409660 [00012ae8] AriocE::validateSubIds: validated 1073741824 hash values 125409660 [00012ae8] AriocE::validateSubIds: # subIds # hash keys avg J-list size 125409660 [00012ae8] AriocE::validateSubIds: 0 0 0.0 125409660 [00012ae8] AriocE::validateSubIds: 1 22526309 1.0 125409660 [00012ae8] AriocE::validateSubIds: 2 406893 2.0 125409660 [00012ae8] AriocE::validateSubIds: 3 36909 3.1 125409661 [00012ae8] AriocE::validateSubIds: 4 11883 4.5 125409661 [00012ae8] AriocE::validateSubIds: 5 6251 5.9 125409661 [00012ae8] AriocE::validateSubIds: 6 6062 7.9 125409661 [00012ae8] AriocE::validateSubIds: 7 5327 9.7 125409661 [00012ae8] AriocE::validateSubIds: 8 7036 12.3 125409661 [00012ae8] AriocE::validateSubIds: 9 5271 14.8 125409661 [00012ae8] AriocE::validateSubIds: 10 3836 17.0 125409661 [00012ae8] AriocE::validateSubIds: 11 3621 21.1 125409661 [00012ae8] AriocE::validateSubIds: 12 5495 26.6 125409661 [00012ae8] AriocE::validateSubIds: 13 3179 20.9 125409661 [00012ae8] AriocE::validateSubIds: 14 471 38.4 125409661 [00012ae8] AriocE::validateSubIds: 15 168 69.9 125409661 [00012ae8] AriocE::validateSubIds: 16 34 70.4 125409661 [00012ae8] AriocE::validateSubIds: completed 125409661 [00012ae8] AriocE::finalizeJ: finalizing J lists on 112 worker threads... 125433093 [00012ae8] AriocE::finalizeJ: completed 125433758 [00012ae8] RaiiMutex::~RaiiMutex: pthread_mutex_destroy returned 16 125447990 [000130a2] ApplicationException ([0x00077986] CppCommon/RaiiMutex.cpp 107): pthread_mutex_unlock returned 1

RWilton commented 3 years ago

Thank you again for the detailed troubleshooting help.

Although I like to maintain backward compatibility, I think it's time I modified the Arioc makefile to specify c++14. That's the default used by Nvidia's CUB implementation (which is in turn used by Nvidia Thrust, which is used by Arioc). That would cause a build-time error with any g++ compiler older than v6.1, but requiring a more recent version of g++ is probably better than dealing with build-time warnings over which I have no control. So I will take care of that. Thank you for raising the issue!

The pthreads problem is tougher. Needless to say, the code runs fine on a bunch of other Linux and Windows installations! So I am trying to get a test system put together with the same (or similar) g++ and CUDA versions you are using to see if I can reproduce the problem. I will let you know.

In the meantime: could you please re-run AriocE with fewer CPU threads? (For this test, 16 threads would be fine.) Here is my reasoning: At face value, the error says that a mutex is being destroyed while it is still in use -- but the lookup-table validation code that throws the error explicitly waits for all of the thread synchronization objects it uses to be released before destroying them. So there's something strange going on, but until I can reproduce the problem I am not sure I understand how to troubleshoot it. There is no reason to think that the 112 threads AriocE is currently using are "too many" -- but using only a few threads to validate the lookup table might cause the code to fail differently (and more informatively) simply because whatever inter-thread error is arising might be elicited with different timing.

Anyway, thank you for your patience. I will follow up when I set up a test environment, and I hope you will let me know the result of running AriocE with fewer threads (use maxDOP="16" in the AriocE element of the configuration file).

-- rw

a7032018 commented 3 years ago

Reducing the core to 16 didn't help. I was able to run AriocE with just 1 core for nongapped index without any errors. It is not terribly slower than 16 or 112 cores.

052827290 [0000a612] AriocE v1.43.2991.21102 (release) +2021-04-12T13:07:35 052827290 [0000a612] Copyright (c) 2015-2021 Johns Hopkins University. All rights reserved. 052827290 [0000a612] compiled : 2021-04-15T12:20:27 (GNU g++ v9.3.0) 052827290 [0000a612] data type sizes : int=4 long=8 *=8 Jvalue5=5 Jvalue8=8 JtableHeader=5 (little-endian) 052827290 [0000a612] computer name : CBU5143610 052827290 [0000a612] executable file : /home/apps/arioc/bin/AriocE 052827290 [0000a612] configuration file : /home/apps/arioc/R/S_cerevisiae/AriocE.nongapped.cfg 052827296 [0000a612] AriocE::encodeR: encoding 17 files (34 sequences) (16 CPU threads available)... 052829582 [0000a612] AriocE::encodeR: encoded 17 files (34 sequences) 052830683 [0000a612] AriocE::computeJlistSizes: sorting C list (1073741824 hash keys)... 053026647 [0000a612] AriocE::computeJlistSizes: sorted C list 053029614 [0000a612] AriocE::computeJlistSizes: m_nJ=24482946 (85527+24397419) m_iCshort=2248 m_iCzero=23028745 053029614 [0000a612] AriocE::computeJlistSizes: log2(nJ) # J-lists 053029614 [0000a612] AriocE::computeJlistSizes: 0 22280497 053029615 [0000a612] AriocE::computeJlistSizes: 1 677079 053029615 [0000a612] AriocE::computeJlistSizes: 2 33216 053029615 [0000a612] AriocE::computeJlistSizes: 3 21701 053029615 [0000a612] AriocE::computeJlistSizes: 4 14004 053029615 [0000a612] AriocE::computeJlistSizes: 5 2110 053029615 [0000a612] AriocE::computeJlistSizes: 6 138 053035364 [0000a612] AriocE::computeJlistSizes: completed: J table uses 24485194 elements for 24482946 J values (0.009% for list counts in J lists ) 053037109 [0000a612] AriocE::buildJlists: building J lists for 17 files (34 sequences) (16 CPU threads available)... 053038812 [0000a612] AriocE::buildJlists: built J lists for 17 files (34 sequences) 053038829 [0000a612] AriocE::sortJcpu: sorting J lists on 16 worker threads... 053111507 [0000a612] AriocE::sortJcpu: J list sort complete 053130693 [0000a612] AriocE::validateHJ: start 053138746 [0000a612] AriocE::validateHJ: H table uses 23028745/1073741824 ( 2.145%) hash keys 053138746 [0000a612] AriocE::validateHJ: J table contains 24485194 J values (list counts in lists: 2248 (0.009%)) 053138746 [0000a612] AriocE::validateHJ: maximum nJ=82 for hash key 0x1CD1186B 053138746 [0000a612] AriocE::validateHJ: completed 053138746 [0000a612] AriocE::validateSubIds: start 053142297 [0000a612] AriocE::validateSubIds: validated 1073741824 hash values 053142297 [0000a612] AriocE::validateSubIds: # subIds # hash keys avg J-list size 053142297 [0000a612] AriocE::validateSubIds: 0 0 0.0 053142297 [0000a612] AriocE::validateSubIds: 1 22526309 1.0 053142297 [0000a612] AriocE::validateSubIds: 2 406893 2.0 053142297 [0000a612] AriocE::validateSubIds: 3 36909 3.1 053142297 [0000a612] AriocE::validateSubIds: 4 11883 4.5 053142297 [0000a612] AriocE::validateSubIds: 5 6251 5.9 053142297 [0000a612] AriocE::validateSubIds: 6 6062 7.9 053142297 [0000a612] AriocE::validateSubIds: 7 5327 9.7 053142297 [0000a612] AriocE::validateSubIds: 8 7036 12.3 053142297 [0000a612] AriocE::validateSubIds: 9 5271 14.8 053142297 [0000a612] AriocE::validateSubIds: 10 3836 17.0 053142297 [0000a612] AriocE::validateSubIds: 11 3621 21.1 053142297 [0000a612] AriocE::validateSubIds: 12 5495 26.6 053142297 [0000a612] AriocE::validateSubIds: 13 3179 20.9 053142297 [0000a612] AriocE::validateSubIds: 14 471 38.4 053142297 [0000a612] AriocE::validateSubIds: 15 168 69.9 053142297 [0000a612] AriocE::validateSubIds: 16 34 70.4 053142297 [0000a612] AriocE::validateSubIds: completed 053142297 [0000a612] AriocE::finalizeJ: finalizing J lists on 16 worker threads... 053201992 [0000a612] AriocE::finalizeJ: completed 053202116 [0000a612] RaiiMutex::~RaiiMutex: pthread_mutex_destroy returned 16 053206132 [0000a770] ApplicationException ([0x00042864] CppCommon/RaiiMutex.cpp 107): pthread_mutex_unlock returned 1

But using 1 core didn't solve pthread issue when making gapped index

053952322 [0000a924] AriocE v1.43.2991.21102 (release) +2021-04-12T13:07:35 053952322 [0000a924] Copyright (c) 2015-2021 Johns Hopkins University. All rights reserved. 053952322 [0000a924] compiled : 2021-04-15T12:20:27 (GNU g++ v9.3.0) 053952322 [0000a924] data type sizes : int=4 long=8 *=8 Jvalue5=5 Jvalue8=8 JtableHeader=5 (little-endian) 053952322 [0000a924] computer name : CBU5143610 053952322 [0000a924] executable file : /home/apps/arioc/bin/AriocE 053952322 [0000a924] configuration file : /home/apps/arioc/R/S_cerevisiae/AriocE.gapped.cfg 053952323 [0000a924] AriocE::encodeR: encoding 17 files (34 sequences) (1 CPU thread available)... 054006084 [0000a924] AriocE::encodeR: encoded 17 files (34 sequences) 054007201 [0000a924] AriocE::computeJlistSizes: sorting C list (1073741824 hash keys)... 054341099 [0000a924] AriocE::computeJlistSizes: sorted C list 054344016 [0000a924] AriocE::computeJlistSizes: m_nJ=24485122 (220907+24264215) m_iCshort=5225 m_iCzero=22744289 054344016 [0000a924] AriocE::computeJlistSizes: log2(nJ) # J-lists 054344016 [0000a924] AriocE::computeJlistSizes: 0 21950821 054344016 [0000a924] AriocE::computeJlistSizes: 1 704636 054344017 [0000a924] AriocE::computeJlistSizes: 2 43501 054344017 [0000a924] AriocE::computeJlistSizes: 3 23287 054344017 [0000a924] AriocE::computeJlistSizes: 4 16819 054344017 [0000a924] AriocE::computeJlistSizes: 5 4622 054344017 [0000a924] AriocE::computeJlistSizes: 6 573 054344017 [0000a924] AriocE::computeJlistSizes: 7 26 054344017 [0000a924] AriocE::computeJlistSizes: 9 4 054349742 [0000a924] AriocE::computeJlistSizes: completed: J table uses 24490347 elements for 24485122 J values (0.021% for list counts in J lists) 054350829 [0000a924] AriocE::buildJlists: building J lists for 17 files (34 sequences) (1 CPU thread available)... 054406026 [0000a924] AriocE::buildJlists: built J lists for 17 files (34 sequences) 054406040 [0000a924] AriocE::sortJcpu: sorting J lists on 1 worker threads... 054417644 [0000a924] AriocE::sortJcpu: J list sort complete 054432777 [0000a924] AriocE::validateHJ: start 054439793 [0000a924] AriocE::validateHJ: H table uses 22744289/1073741824 ( 2.118%) hash keys 054439793 [0000a924] AriocE::validateHJ: J table contains 24490347 J values (list counts in lists: 5225 (0.021%)) 054439793 [0000a924] AriocE::validateHJ: maximum nJ=892 for hash key 0x295C5954 054439793 [0000a924] AriocE::validateHJ: completed 054439793 [0000a924] AriocE::validateSubIds: start 054448894 [0000a924] AriocE::validateSubIds: validated 1073741824 hash values 054448894 [0000a924] AriocE::validateSubIds: # subIds # hash keys avg J-list size 054448894 [0000a924] AriocE::validateSubIds: 0 0 0.0 054448894 [0000a924] AriocE::validateSubIds: 1 22193080 1.0 054448894 [0000a924] AriocE::validateSubIds: 2 438996 2.1 054448894 [0000a924] AriocE::validateSubIds: 3 45131 3.1 054448894 [0000a924] AriocE::validateSubIds: 4 12937 4.3 054448894 [0000a924] AriocE::validateSubIds: 5 6546 5.7 054448894 [0000a924] AriocE::validateSubIds: 6 5218 7.4 054448894 [0000a924] AriocE::validateSubIds: 7 5482 9.1 054448894 [0000a924] AriocE::validateSubIds: 8 8909 12.1 054448894 [0000a924] AriocE::validateSubIds: 9 5069 13.6 054448894 [0000a924] AriocE::validateSubIds: 10 3334 16.9 054448894 [0000a924] AriocE::validateSubIds: 11 3647 20.4 054448894 [0000a924] AriocE::validateSubIds: 12 7642 28.0 054448894 [0000a924] AriocE::validateSubIds: 13 6322 21.2 054448894 [0000a924] AriocE::validateSubIds: 14 1197 39.2 054448894 [0000a924] AriocE::validateSubIds: 15 359 56.8 054448894 [0000a924] AriocE::validateSubIds: 16 416 84.3 054448894 [0000a924] AriocE::validateSubIds: 17 4 474.5 054448894 [0000a924] AriocE::validateSubIds: completed 054448894 [0000a924] AriocE::finalizeJ: finalizing J lists on 1 worker threads... 054457048 [0000a924] AriocE::finalizeJ: completed 054500196 [0000a924] RaiiMutex::~RaiiMutex: pthread_mutex_destroy returned 16 054501769 [0000aae7] ApplicationException ([0x00043751] CppCommon/RaiiMutex.cpp 107): pthread_mutex_unlock returned 1

It's very strange that I got the different error when making gapped index using only 16 cores. Not pthread this time but something relates to J hash

052931612 [0000a64b] AriocE v1.43.2991.21102 (release) +2021-04-12T13:07:35 052931612 [0000a64b] Copyright (c) 2015-2021 Johns Hopkins University. All rights reserved. 052931612 [0000a64b] compiled : 2021-04-15T12:20:27 (GNU g++ v9.3.0) 052931612 [0000a64b] data type sizes : int=4 long=8 *=8 Jvalue5=5 Jvalue8=8 JtableHeader=5 (little-endian) 052931612 [0000a64b] computer name : CBU5143610 052931612 [0000a64b] executable file : /home/apps/arioc/bin/AriocE 052931613 [0000a64b] configuration file : /home/apps/arioc/R/S_cerevisiae/AriocE.gapped.cfg 052931614 [0000a64b] AriocE::encodeR: encoding 17 files (34 sequences) (16 CPU threads available)... 052933813 [0000a64b] AriocE::encodeR: encoded 17 files (34 sequences) 052934921 [0000a64b] AriocE::computeJlistSizes: sorting C list (1073741824 hash keys)... 053312075 [0000a64b] AriocE::computeJlistSizes: sorted C list 053315017 [0000a64b] AriocE::computeJlistSizes: m_nJ=24485122 (220907+24264215) m_iCshort=5225 m_iCzero=22744289 053315017 [0000a64b] AriocE::computeJlistSizes: log2(nJ) # J-lists 053315017 [0000a64b] AriocE::computeJlistSizes: 0 21950821 053315017 [0000a64b] AriocE::computeJlistSizes: 1 704636 053315017 [0000a64b] AriocE::computeJlistSizes: 2 43501 053315017 [0000a64b] AriocE::computeJlistSizes: 3 23287 053315017 [0000a64b] AriocE::computeJlistSizes: 4 16819 053315017 [0000a64b] AriocE::computeJlistSizes: 5 4622 053315017 [0000a64b] AriocE::computeJlistSizes: 6 573 053315017 [0000a64b] AriocE::computeJlistSizes: 7 26 053315017 [0000a64b] AriocE::computeJlistSizes: 9 4 053320592 [0000a64b] AriocE::computeJlistSizes: completed: J table uses 24490347 elements for 24485122 J values (0.021% for list counts in J lists) 053322652 [0000a64b] AriocE::buildJlists: building J lists for 17 files (34 sequences) (16 CPU threads available)... 053324752 [0000a64b] AriocE::buildJlists: built J lists for 17 files (34 sequences) 053324767 [0000a64b] AriocE::sortJcpu: sorting J lists on 16 worker threads... 053355056 [0000a64b] AriocE::sortJcpu: J list sort complete 053415558 [0000a64b] AriocE::validateHJ: start 053421430 [0000a64b] ApplicationException ([0x00042571] AriocE/AriocE.R.cpp 1021): J list out of order for hash 0x00004EAC: J=0x00000000 s=0 subId=0 is followed by J=00000000 s=0 subId=0

a7032018 commented 3 years ago

Just want to indicate that my workstation are also running Dfam and BLAST occupying 100 cores. Not sure it has anything to do with the pthread issue. I will test Arioc again once my workstation has less loading.

RWilton commented 3 years ago

Thanks again for the error logs. Using a different number of threads results in a totally different error in a totally different part of the application -- in one case, a threading error and in another case an apparently-empty buffer that should be filled with data -- so there is clearly something unexpected going on here.

I doubt that running AriocE on a shared system is related to the bug you are seeing. We have run the Arioc applications thousands of times in shared-resource environments, and although performance degradation is to be expected (e.g. by saturating disk I/O or GPU memory bandwidth) there is no reason to expect threading or memory errors to occur.

Needless to say, I have been unable to reproduce this error so far either in Linux or in Windows. But as I indicated, I am getting a test system set up with the same software versions as you are using. I will let you know how that goes.

In the meantime, would you mind telling me more about your operating environment? I am specifically interested in:

AriocE isn't using CUDA at this point, so information about your GPU hardware and CUDA driver is not a priority.

-- rw

a7032018 commented 3 years ago

I have Dell T7920 customized workstation with 1TB RAM and 2x Intel Xeon 8180 CPU. OS is Ubuntu 16.04 LTS with default filesystem. 1x Quadro RTX 5000 with CUDA 11.2. This is not an HPC so no job scheduler is involved.

RWilton commented 3 years ago

Thanks for the system info.

I wonder if we have run into a known problem with the pthreads implementation in Ubuntu (https://sourceware.org/bugzilla/show_bug.cgi?id=25847) although this would be difficult to prove, as you might imagine from looking at the bug.

I suspect this particular problem because the original errors you encountered (i.e., the calls to pthread_mutex_unlock and pthread_mutex_destroy) are in calls from C++ destructors that are simply doing housekeeping on mutexes that are no longer in use. After many thousands of hours of execution time in various Redhat and CentOS Linuxen as well as in Windows, the failure mode you're seeing hasn't arisen, so it's unlikely that something in the Arioc code is truly using those mutexes at the point where they are being destructed (which is what the error codes imply).

Having said that: I don't want to just give up and blame the developers of Ubuntu. Instead, I am going to try to get a copy of that OS installed somewhere around here to see if I can get that bug to appear. I can't promise anything in the next few days, however, so if you have a different OS you can try this with, that would be an equally useful (albeit painful) way to go at it. Alternatively, if it's possible for you to grant me remote login privileges on the machine where the error is occurring, I could spend some time with a debugger and see if the error can be more specifically identified and worked around.

There is more info on the Ubuntu manifestation of this bug here: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1899800. Interestingly, they claim that the bug "appeared" in a version of Ubuntu later than 16.04, but frankly I can't follow the reasoning on this except that the bug doesn't happen to appear in their test application with v16.04 (yet it does appear in their repro even though it was supposedly fixed in a version of glibc later than the one associated with v16.04). I don't quite follow that line of thinking -- but in any event if you want to try the latest version of Ubuntu, maybe you'll get lucky...

I am sorry there isn't a more satisfying solution immediately at hand.

-- rw

a7032018 commented 3 years ago

I appreciate your effort solving the problem. I have another workstation with the similar hardware configuration running different Ubuntu version (18.04 with G++/GCC 7.5, CUDA 11.2 with NVIDIA driver 465). Unlike the workstation having issues, this one has no tasks running. I was able to run the entire pipeline (AriocE/P/U) without errors. I will try the pipeline again at the problemed workstation once the workloads are eased.

RWilton commented 3 years ago

It's interesting (and helpful) to hear that the error does not appear with a different version of Ubuntu. That's not a proof that the Ubuntu bug is the underlying problem here, but it's certainly consistent with the other evidence we have.

Again, I doubt that the bug is related to running AriocE concurrently with other tasks. But if you upgrade the Ubuntu installation on CBU5143610, or if you replace it with a different Linux release, I would like to know if the problem persists.

I will keep this "issue" open and update it when I have more to contribute. Please follow up if you have any additional information as well -- whether it's good or bad!

-- rw

a7032018 commented 3 years ago

I recompiled AriocE with my problemed workstation with the CUDA update (11.2->11.3, NVIDIA driver from 460 to 465), and was able to finish the AriocE seamlessly while the workstation was loaded with other tasks. However, compilation of AriocP and U fails.

I read the manual saying each fasta file can only have 1 sequence. If I have millions sequences (NCBI NT for example) as my reference database, do I have to separate these millions sequences into million fasta files with 1 sequence in each file?

~/apps/arioc/src$ make AriocP /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocBase/baseMaxV.gpu.o AriocBase/baseMaxV.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

4 errors detected in the compilation of "AriocBase/baseMaxV.cu". /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocBase/baseAlignG.gpu.o AriocBase/baseAlignG.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

4 errors detected in the compilation of "AriocBase/baseAlignG.cu". /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocBase/baseMaxVw.gpu.o AriocBase/baseMaxVw.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

4 errors detected in the compilation of "AriocBase/baseMaxVw.cu". /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocBase/baseAlignN.gpu.o AriocBase/baseAlignN.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

4 errors detected in the compilation of "AriocBase/baseAlignN.cu". /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocBase/baseLoadRw.gpu.o AriocBase/baseLoadRw.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

4 errors detected in the compilation of "AriocBase/baseLoadRw.cu". touch -c AriocP/AriocP.cpp g++ -std=c++14 -c -Wall -Wno-unknown-pragmas -O3 -m64 -I /usr/local/cuda-11.3/include -o AriocP/AriocP.o AriocP/AriocP.cpp g++ -MM AriocP/AriocP.cpp | sed -e 's~^(.*).o:~AriocP/AriocP.o:~' > AriocP/AriocP.d

~/apps/arioc/src$ make AriocU /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocU/baseFilterQu.gpu.o AriocU/baseFilterQu.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). g++ -MM -xc++ AriocU/baseFilterQu.cu | sed -e 's~^(.).o:~AriocU/baseFilterQu.gpu.o:~' > AriocU/baseFilterQu.gpu.d /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocU/tuSetupN.gpu.o AriocU/tuSetupN.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). g++ -MM -xc++ AriocU/tuSetupN.cu | sed -e 's~^(.).o:~AriocU/tuSetupN.gpu.o:~' > AriocU/tuSetupN.gpu.d /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocU/baseMapGc.gpu.o AriocU/baseMapGc.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). g++ -MM -xc++ AriocU/baseMapGc.cu | sed -e 's~^(.).o:~AriocU/baseMapGc.gpu.o:~' > AriocU/baseMapGc.gpu.d /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocU/baseFilterD.gpu.o AriocU/baseFilterD.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). g++ -MM -xc++ AriocU/baseFilterD.cu | sed -e 's~^(.).o:~AriocU/baseFilterD.gpu.o:~' > AriocU/baseFilterD.gpu.d /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocU/tuAlignN.gpu.o AriocU/tuAlignN.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). g++ -MM -xc++ AriocU/tuAlignN.cu | sed -e 's~^(.).o:~AriocU/tuAlignN.gpu.o:~' > AriocU/tuAlignN.gpu.d /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocU/tuAlignGs.gpu.o AriocU/tuAlignGs.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). g++ -MM -xc++ AriocU/tuAlignGs.cu | sed -e 's~^(.).o:~AriocU/tuAlignGs.gpu.o:~' > AriocU/tuAlignGs.gpu.d /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocU/tuSetupN24.gpu.o AriocU/tuSetupN24.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). g++ -MM -xc++ AriocU/tuSetupN24.cu | sed -e 's~^(.).o:~AriocU/tuSetupN24.gpu.o:~' > AriocU/tuSetupN24.gpu.d /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocU/tuAlignN32.gpu.o AriocU/tuAlignN32.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). g++ -MM -xc++ AriocU/tuAlignN32.cu | sed -e 's~^(.).o:~AriocU/tuAlignN32.gpu.o:~' > AriocU/tuAlignN32.gpu.d /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocU/tuAlignGs10.gpu.o AriocU/tuAlignGs10.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). g++ -MM -xc++ AriocU/tuAlignGs10.cu | sed -e 's~^(.*).o:~AriocU/tuAlignGs10.gpu.o:~' > AriocU/tuAlignGs10.gpu.d /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocBase/baseMaxV.gpu.o AriocBase/baseMaxV.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

4 errors detected in the compilation of "AriocBase/baseMaxV.cu". /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocBase/baseAlignG.gpu.o AriocBase/baseAlignG.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

4 errors detected in the compilation of "AriocBase/baseAlignG.cu". /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocBase/baseMaxVw.gpu.o AriocBase/baseMaxVw.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

4 errors detected in the compilation of "AriocBase/baseMaxVw.cu". /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocBase/baseAlignN.gpu.o AriocBase/baseAlignN.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

4 errors detected in the compilation of "AriocBase/baseAlignN.cu". /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocBase/baseLoadRw.gpu.o AriocBase/baseLoadRw.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

4 errors detected in the compilation of "AriocBase/baseLoadRw.cu". g++ -std=c++14 -c -Wall -Wno-unknown-pragmas -O3 -m64 -I /usr/local/cuda-11.3/include -o AriocU/tuSetupN.o AriocU/tuSetupN.cpp g++ -MM AriocU/tuSetupN.cpp | sed -e 's~^(.).o:~AriocU/tuSetupN.d:~' > AriocU/tuSetupN.d g++ -std=c++14 -c -Wall -Wno-unknown-pragmas -O3 -m64 -I /usr/local/cuda-11.3/include -o AriocU/baseMapGc.o AriocU/baseMapGc.cpp g++ -MM AriocU/baseMapGc.cpp | sed -e 's~^(.).o:~AriocU/baseMapGc.d:~' > AriocU/baseMapGc.d touch -c AriocU/AriocU.cpp g++ -std=c++14 -c -Wall -Wno-unknown-pragmas -O3 -m64 -I /usr/local/cuda-11.3/include -o AriocU/baseFilterD.o AriocU/baseFilterD.cpp g++ -MM AriocU/baseFilterD.cpp | sed -e 's~^(.).o:~AriocU/baseFilterD.d:~' > AriocU/baseFilterD.d g++ -std=c++14 -c -Wall -Wno-unknown-pragmas -O3 -m64 -I /usr/local/cuda-11.3/include -o AriocU/tuGpuU.o AriocU/tuGpuU.cpp g++ -MM AriocU/tuGpuU.cpp | sed -e 's~^(.).o:~AriocU/tuGpuU.d:~' > AriocU/tuGpuU.d g++ -std=c++14 -c -Wall -Wno-unknown-pragmas -O3 -m64 -I /usr/local/cuda-11.3/include -o AriocU/tuAlignN.o AriocU/tuAlignN.cpp g++ -MM AriocU/tuAlignN.cpp | sed -e 's~^(.).o:~AriocU/tuAlignN.d:~' > AriocU/tuAlignN.d g++ -std=c++14 -c -Wall -Wno-unknown-pragmas -O3 -m64 -I /usr/local/cuda-11.3/include -o AriocU/stdafx.o AriocU/stdafx.cpp g++ -MM AriocU/stdafx.cpp | sed -e 's~^(.).o:~AriocU/stdafx.d:~' > AriocU/stdafx.d g++ -std=c++14 -c -Wall -Wno-unknown-pragmas -O3 -m64 -I /usr/local/cuda-11.3/include -o AriocU/tuSetupN24.o AriocU/tuSetupN24.cpp g++ -MM AriocU/tuSetupN24.cpp | sed -e 's~^(.).o:~AriocU/tuSetupN24.d:~' > AriocU/tuSetupN24.d g++ -std=c++14 -c -Wall -Wno-unknown-pragmas -O3 -m64 -I /usr/local/cuda-11.3/include -o AriocU/tuClassifyU.o AriocU/tuClassifyU.cpp g++ -MM AriocU/tuClassifyU.cpp | sed -e 's~^(.).o:~AriocU/tuClassifyU.d:~' > AriocU/tuClassifyU.d g++ -std=c++14 -c -Wall -Wno-unknown-pragmas -O3 -m64 -I /usr/local/cuda-11.3/include -o AriocU/tuLoadMU.o AriocU/tuLoadMU.cpp g++ -MM AriocU/tuLoadMU.cpp | sed -e 's~^(.).o:~AriocU/tuLoadMU.d:~' > AriocU/tuLoadMU.d g++ -std=c++14 -c -Wall -Wno-unknown-pragmas -O3 -m64 -I /usr/local/cuda-11.3/include -o AriocU/tuAlignGs.o AriocU/tuAlignGs.cpp g++ -MM AriocU/tuAlignGs.cpp | sed -e 's~^(.).o:~AriocU/tuAlignGs.d:~' > AriocU/tuAlignGs.d g++ -std=c++14 -c -Wall -Wno-unknown-pragmas -O3 -m64 -I /usr/local/cuda-11.3/include -o AriocU/tuAlignN32.o AriocU/tuAlignN32.cpp g++ -MM AriocU/tuAlignN32.cpp | sed -e 's~^(.).o:~AriocU/tuAlignN32.d:~' > AriocU/tuAlignN32.d g++ -std=c++14 -c -Wall -Wno-unknown-pragmas -O3 -m64 -I /usr/local/cuda-11.3/include -o AriocU/tuClassifyU1.o AriocU/tuClassifyU1.cpp g++ -MM AriocU/tuClassifyU1.cpp | sed -e 's~^(.).o:~AriocU/tuClassifyU1.d:~' > AriocU/tuClassifyU1.d g++ -std=c++14 -c -Wall -Wno-unknown-pragmas -O3 -m64 -I /usr/local/cuda-11.3/include -o AriocU/tuTailU.o AriocU/tuTailU.cpp g++ -MM AriocU/tuTailU.cpp | sed -e 's~^(.).o:~AriocU/tuTailU.d:~' > AriocU/tuTailU.d g++ -std=c++14 -c -Wall -Wno-unknown-pragmas -O3 -m64 -I /usr/local/cuda-11.3/include -o AriocU/AppGlobal.o AriocU/AppGlobal.cpp g++ -MM AriocU/AppGlobal.cpp | sed -e 's~^(.).o:~AriocU/AppGlobal.d:~' > AriocU/AppGlobal.d g++ -std=c++14 -c -Wall -Wno-unknown-pragmas -O3 -m64 -I /usr/local/cuda-11.3/include -o AriocU/QReaderU.o AriocU/QReaderU.cpp g++ -MM AriocU/QReaderU.cpp | sed -e 's~^(.).o:~AriocU/QReaderU.d:~' > AriocU/QReaderU.d g++ -std=c++14 -c -Wall -Wno-unknown-pragmas -O3 -m64 -I /usr/local/cuda-11.3/include -o AriocU/tuWriteA.o AriocU/tuWriteA.cpp g++ -MM AriocU/tuWriteA.cpp | sed -e 's~^(.).o:~AriocU/tuWriteA.d:~' > AriocU/tuWriteA.d g++ -std=c++14 -c -Wall -Wno-unknown-pragmas -O3 -m64 -I /usr/local/cuda-11.3/include -o AriocU/tuAlignGs10.o AriocU/tuAlignGs10.cpp g++ -MM AriocU/tuAlignGs10.cpp | sed -e 's~^(.).o:~AriocU/tuAlignGs10.d:~' > AriocU/tuAlignGs10.d g++ -std=c++14 -c -Wall -Wno-unknown-pragmas -O3 -m64 -I /usr/local/cuda-11.3/include -o AriocU/tuClassifyU2.o AriocU/tuClassifyU2.cpp g++ -MM AriocU/tuClassifyU2.cpp | sed -e 's~^(.).o:~AriocU/tuClassifyU2.d:~' > AriocU/tuClassifyU2.d g++ -std=c++14 -c -Wall -Wno-unknown-pragmas -O3 -m64 -I /usr/local/cuda-11.3/include -o AriocU/baseFilterQu.o AriocU/baseFilterQu.cpp g++ -MM AriocU/baseFilterQu.cpp | sed -e 's~^(.).o:~AriocU/baseFilterQu.d:~' > AriocU/baseFilterQu.d g++ -std=c++14 -c -Wall -Wno-unknown-pragmas -O3 -m64 -I /usr/local/cuda-11.3/include -o AriocU/AppMain.o AriocU/AppMain.cpp g++ -MM AriocU/AppMain.cpp | sed -e 's~^(.).o:~AriocU/AppMain.d:~' > AriocU/AppMain.d /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocBase/baseMaxV.gpu.o AriocBase/baseMaxV.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

4 errors detected in the compilation of "AriocBase/baseMaxV.cu". /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocBase/baseAlignG.gpu.o AriocBase/baseAlignG.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

4 errors detected in the compilation of "AriocBase/baseAlignG.cu". /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocBase/baseMaxVw.gpu.o AriocBase/baseMaxVw.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

4 errors detected in the compilation of "AriocBase/baseMaxVw.cu". /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocBase/baseAlignN.gpu.o AriocBase/baseAlignN.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

4 errors detected in the compilation of "AriocBase/baseAlignN.cu". /usr/local/cuda-11.3/bin/nvcc -std=c++14 -c -O3 -m64 -gencode arch=compute_37,code=\"compute_37,sm_37\" -Xcompiler -O3,-m64 -I /usr/local/cuda-11.3/include -o AriocBase/baseLoadRw.gpu.o AriocBase/baseLoadRw.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: calling a host function("AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters()") from a device function("AlignmentScoreParameters::AlignmentScoreParameters") is not allowed

AriocBase/../AriocCommon/AriocAlignmentScorer.h(166): error: identifier "AlignmentScoreFunctionParameters::AlignmentScoreFunctionParameters" is undefined in device code

4 errors detected in the compilation of "AriocBase/baseLoadRw.cu". touch -c AriocU/AriocU.cpp g++ -std=c++14 -c -Wall -Wno-unknown-pragmas -O3 -m64 -I /usr/local/cuda-11.3/include -o AriocU/AriocU.o AriocU/AriocU.cpp g++ -MM AriocU/AriocU.cpp | sed -e 's~^(.*).o:~AriocU/AriocU.o:~' > AriocU/AriocU.d

RWilton commented 3 years ago

Ok, thanks for the information. Let's see if we can sort this out:

1) I just got the CUDA SDK v11.3 and GNU g++ v9.3 installed, and I encountered the same build error for AriocP and AriocU. It looks like these more recent versions of the C++ and CUDA compilers are enforcing more stringent syntax checking on data structures used in both CPU and GPU code. (That's good!) So I have addressed the problem in a new Arioc build that I have just uploaded (v1.43.2994). Please download it and recompile.

2) Even with these current versions of the compilers and with the current CUDA drivers, I still cannot reproduce the pthread error. (That's with a CentOS Linux, of course, not Ubuntu.)

3) Arioc, like BWA or Bowtie, is a tool for aligning many short DNA sequences (reads) against a single large reference genome. As such, it can handle dozens of different reference-sequence "subunits" (e.g., chromosomes), each of which is specified in its own FASTA file. But it's not a BLAST-like tool that is designed to search a curated database of thousands or millions of short DNA sequences.

-- rw

a7032018 commented 3 years ago
  1. I was able to compile and run AriocE/P/U without any faults. I am truly thankful for your time to solve the issue with me.
  2. For my own understanding, BWA or Bowtie accepts packing many sequences into a single fasta file to make the index. I am going to use Arioc to align my short reads to multiple virus genomes (at least 30K of virus genomes, let's say). Would combining all these virus genomes into one single fasta file work, or I have to separate them to single fasta files?
RWilton commented 3 years ago

It looks like we have a working resolution to the pthreads error in AriocE, so I am going to close this github "issue." If the error recurs, either when running with Ubuntu or in another context, then we can revisit the problem.

As for using Arioc to align reads to a database of short DNA sequences or viral genomes: there is a hard-coded 28-bit (268,435,456) upper limit to the number of reference sequences that can be used. However, I have never used Arioc in the "BLASTN-like" manner you are suggesting. I'd be happy to follow up on that with you directly by email.

-- rw