se2p / pynguin

The PYthoN General UnIt Test geNerator is a test-generation tool for Python
https://www.pynguin.eu
MIT License
1.22k stars 74 forks source link

Test case timeout causes race condition/multiple instances #42

Closed EmmaJaneBonestell closed 1 year ago

EmmaJaneBonestell commented 1 year ago

System Info: Pynguin version '0.32.0.dev' Ubuntu 22.04, Python 3.10.6 Or when inside docker, Debian Bullseye Python 3.10.10. Occurs in either.

After a test case times out (with the error message mentioned here : https://github.com/se2p/pynguin/issues/29 ) , Pynguin spawns another instance of itself with the exact same command line being run and starting from the beginning. The previous instance is not terminated, and attempting to terminate either proccess kills the other(s).

Besides the obvious issues with possibly overwriting Pynguin's outputted result/generated files, this also sometimes occurs repeatedly, spawning enough processes to take up all available RAM and throttles the CPU. Unfortunately, I could not reproduce it when running under a debugger capable of dealing with threads (PyCharm), so I have no helpful backtrace to provide, but here's an example of the output with a single '--verbose'.

[21:55:39] INFO     Start Pynguin Test Generation…                                                                                                                                                               generator.py:110
           INFO     Collecting static constants from module under test                                                                                                                                           generator.py:210
           INFO     No constants found                                                                                                                                                                           generator.py:213
           INFO     Setting up runtime collection of constants                                                                                                                                                   generator.py:222
[21:55:50]       ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call

               ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call                                                                                                                                                                             

[21:55:51] INFO     Analyzed project to create test cluster                                                                                                                                                        module.py:1296
           INFO     Modules:      60                                                                                                                                                                               module.py:1297
           INFO     Functions:   486                                                                                                                                                                               module.py:1298
           INFO     Classes:     386                                                                                                                                                                               module.py:1299
           INFO     Using seed 1680058537564621002                                                                                                                                                               generator.py:196
           INFO     Using strategy: Algorithm.DYNAMOSA                                                                                                                                          generationalgorithmfactory.py:288
           INFO     Instantiated 1609 fitness functions                                                                                                                                         generationalgorithmfactory.py:380
           INFO     Using CoverageArchive                                                                                                                                                       generationalgorithmfactory.py:332
           INFO     Using selection function: Selection.TOURNAMENT_SELECTION                                                                                                                    generationalgorithmfactory.py:307
           INFO     No stopping condition configured!                                                                                                                                           generationalgorithmfactory.py:111
           INFO     Using fallback timeout of 600 seconds                                                                                                                                       generationalgorithmfactory.py:112
           INFO     Using crossover function: SinglePointRelativeCrossOver                                                                                                                      generationalgorithmfactory.py:320
           INFO     Using ranking function: RankBasedPreferenceSorting                                                                                                                          generationalgorithmfactory.py:340
           INFO     Start generating test cases                                                                                                                                                                  generator.py:523
[21:55:58] INFO     Initial Population, Coverage: 0.104413                                                                                                                                                   searchobserver.py:69
[21:56:09] INFO     Iteration:       1, Coverage: 0.108763                                                                                                                                                   searchobserver.py:73
[21:56:14] INFO     Iteration:       2, Coverage: 0.110628                                                                                                                                                   searchobserver.py:73
[21:56:18] INFO     Iteration:       3, Coverage: 0.114357                                                                                                                                                   searchobserver.py:73
[21:56:23] INFO     Iteration:       4, Coverage: 0.114978                                                                                                                                                   searchobserver.py:73
[21:56:29] INFO     Iteration:       5, Coverage: 0.114978                                                                                                                                                   searchobserver.py:73
[21:56:33] INFO     Iteration:       6, Coverage: 0.115600                                                                                                                                                   searchobserver.py:73
[21:56:35] INFO     Iteration:       7, Coverage: 0.118086                                                                                                                                                   searchobserver.py:73
[21:56:39] INFO     Iteration:       8, Coverage: 0.118707                                                                                                                                                   searchobserver.py:73
[21:56:44] INFO     Iteration:       9, Coverage: 0.130516                                                                                                                                                   searchobserver.py:73
[21:56:48] INFO     Iteration:      10, Coverage: 0.134245                                                                                                                                                   searchobserver.py:73
[21:56:49] ERROR    Exception in Thread: <Thread(Thread-2061 (_execute_test_case), started daemon 139705112209152)>                                                                                             execution.py:2075
                    ╭─────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────────────────────────────╮                  
                    │ /usr/local/lib/python3.10/threading.py:1016 in _bootstrap_inner                                                                                                                         │                  
                    │                                                                                                                                                                                         │                  
                    │   1013 │   │   │   │   _sys.setprofile(_profile_hook)                                                                                                                                   │                  
                    │   1014 │   │   │                                                                                                                                                                        │                  
                    │   1015 │   │   │   try:                                                                                                                                                                 │                  
                    │ ❱ 1016 │   │   │   │   self.run()                                                                                                                                                       │                  
                    │   1017 │   │   │   except:                                                                                                                                                              │                  
                    │   1018 │   │   │   │   self._invoke_excepthook(self)                                                                                                                                    │                  
                    │   1019 │   │   finally:                                                                                                                                                                 │                  
                    │                                                                                                                                                                                         │                  
                    │ /usr/local/lib/python3.10/threading.py:953 in run                                                                                                                                       │                  
                    │                                                                                                                                                                                         │                  
                    │    950 │   │   """                                                                                                                                                                      │                  
                    │    951 │   │   try:                                                                                                                                                                     │                  
                    │    952 │   │   │   if self._target is not None:                                                                                                                                         │                  
                    │ ❱  953 │   │   │   │   self._target(*self._args, **self._kwargs)                                                                                                                        │                  
                    │    954 │   │   finally:                                                                                                                                                                 │                  
                    │    955 │   │   │   # Avoid a refcycle if the thread is running a function with                                                                                                          │                  
                    │    956 │   │   │   # an argument that has a member that points to the thread.                                                                                                           │                  
                    │                                                                                                                                                                                         │                  
                    │ /usr/local/lib/python3.10/site-packages/pynguin/testcase/execution.py:2178 in _execute_test_case                                                                                        │                  
                    │                                                                                                                                                                                         │                  
                    │   2175 │   │   for idx, statement in enumerate(test_case.statements):                                                                                                                   │                  
                    │   2176 │   │   │   ast_node = self._before_statement_execution(statement, exec_ctx)                                                                                                     │                  
                    │   2177 │   │   │   exception = self.execute_ast(ast_node, exec_ctx)                                                                                                                     │                  
                    │ ❱ 2178 │   │   │   self._after_statement_execution(statement, exec_ctx, exception)                                                                                                      │                  
                    │   2179 │   │   │   if exception is not None:                                                                                                                                            │                  
                    │   2180 │   │   │   │   result.report_new_thrown_exception(idx, exception)                                                                                                               │                  
                    │   2181 │   │   │   │   break                                                                                                                                                            │                  
                    │                                                                                                                                                                                         │                  
                    │ /usr/local/lib/python3.10/site-packages/pynguin/testcase/execution.py:2277 in _after_statement_execution                                                                                │                  
                    │                                                                                                                                                                                         │                  
                    │   2274 │   │   # See comments in _before_statement_execution                                                                                                                            │                  
                    │   2275 │   │   if self.tracer.current_thread_identifier != threading.current_thread().ident:                                                                                            │                  
                    │   2276 │   │   │   # Kill this thread                                                                                                                                                   │                  
                    │ ❱ 2277 │   │   │   raise RuntimeError(                                                                                                                                                  │                  
                    │   2278 │   │   │   │   "The current thread shall not be executed any more, thus I kill it."                                                                                             │                  
                    │   2279 │   │   │   )                                                                                                                                                                    │                  
                    │   2280                                                                                                                                                                                  │                  
                    ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯                  
                    RuntimeError: The current thread shall not be executed any more, thus I kill it.
⠙ Running Pynguin...[21:56:52] INFO     Start Pynguin Test Generation…                                                                                                                                                               generator.py:110
           INFO     Collecting static constants from module under test                                                                                                                                           generator.py:210
           INFO     No constants found                                                                                                                                                                           generator.py:213
           INFO     Setting up runtime collection of constants                                                                                                                                                   generator.py:222
[09:56:52 PM] INFO     Iteration:      11, Coverage: 0.139838                                                                                                                                                searchobserver.py:73
[09:56:57 PM] INFO     Iteration:      12, Coverage: 0.141081                                                                                                                                                searchobserver.py:73
⠼ Running Pynguin...[21:56:57] INFO     Start Pynguin Test Generation…                                                                                                                                                               generator.py:110
           INFO     Collecting static constants from module under test                                                                                                                                           generator.py:210
           INFO     No constants found                                                                                                                                                                           generator.py:213
           INFO     Setting up runtime collection of constants                                                                                                                                                   generator.py:222
[09:57:01 PM] INFO     Iteration:      13, Coverage: 0.146053                                                                                                                                                searchobserver.py:73

⠙ Running Pynguin...[21:57:05] INFO     Start Pynguin Test Generation…                                                                                                                                                               generator.py:110
           INFO     Collecting static constants from module under test                                                                                                                                           generator.py:210
           INFO     No constants found                                                                                                                                                                           generator.py:213
           INFO     Setting up runtime collection of constants                                                                                                                                                   generator.py:222
⠋ Running Pynguin...[21:57:05] INFO     Start Pynguin Test Generation…                                                                                                                                                               generator.py:110
           INFO     Collecting static constants from module under test                                                                                                                                           generator.py:210
           INFO     No constants found                                                                                                                                                                           generator.py:213
           INFO     Setting up runtime collection of constants                                                                                                                                                   generator.py:222
[21:57:06] INFO     Analyzed project to create test cluster                                                                                                                                                        module.py:1296
           INFO     Modules:      60                                                                                                                                                                               module.py:1297
           INFO     Functions:   486                                                                                                                                                                               module.py:1298
           INFO     Classes:     386                                                                                                                                                                               module.py:1299
           INFO     Using seed 1680058610586713330                                                                                                                                                               generator.py:196
           INFO     Using strategy: Algorithm.DYNAMOSA                                                                                                                                          generationalgorithmfactory.py:288
           INFO     Instantiated 1609 fitness functions                                                                                                                                         generationalgorithmfactory.py:380
           INFO     Using CoverageArchive                                                                                                                                                       generationalgorithmfactory.py:332
           INFO     Using selection function: Selection.TOURNAMENT_SELECTION                                                                                                                    generationalgorithmfactory.py:307
           INFO     No stopping condition configured!                                                                                                                                           generationalgorithmfactory.py:111
           INFO     Using fallback timeout of 600 seconds                                                                                                                                       generationalgorithmfactory.py:112
           INFO     Using crossover function: SinglePointRelativeCrossOver                                                                                                                      generationalgorithmfactory.py:320
           INFO     Using ranking function: RankBasedPreferenceSorting                                                                                                                          generationalgorithmfactory.py:340
           INFO     Start generating test cases                                                                                                                                                                  generator.py:523
[09:57:07 PM] INFO     Iteration:      14, Coverage: 0.148539                                                                                                                                                searchobserver.py:73                                                                                                                                                                                                  

[21:57:12] INFO     Analyzed project to create test cluster                                                                                                                                                        module.py:1296
           INFO     Modules:      60                                                                                                                                                                               module.py:1297
           INFO     Functions:   486                                                                                                                                                                               module.py:1298
           INFO     Classes:     386                                                                                                                                                                               module.py:1299
           INFO     Using seed 1680058615406288740                                                                                                                                                               generator.py:196
           INFO     Using strategy: Algorithm.DYNAMOSA                                                                                                                                          generationalgorithmfactory.py:288
           INFO     Instantiated 1609 fitness functions                                                                                                                                         generationalgorithmfactory.py:380
           INFO     Using CoverageArchive                                                                                                                                                       generationalgorithmfactory.py:332
           INFO     Using selection function: Selection.TOURNAMENT_SELECTION                                                                                                                    generationalgorithmfactory.py:307
           INFO     No stopping condition configured!                                                                                                                                           generationalgorithmfactory.py:111
           INFO     Using fallback timeout of 600 seconds                                                                                                                                       generationalgorithmfactory.py:112
           INFO     Using crossover function: SinglePointRelativeCrossOver                                                                                                                      generationalgorithmfactory.py:320
           INFO     Using ranking function: RankBasedPreferenceSorting                                                                                                                          generationalgorithmfactory.py:340
           INFO     Start generating test cases                                                                                                                                                                  generator.py:523
⠹ Running Pynguin...[21:57:13] INFO     Start Pynguin Test Generation…                                                                                                                                                               generator.py:110
           INFO     Collecting static constants from module under test                                                                                                                                           generator.py:210
           INFO     No constants found                                                                                                                                                                           generator.py:213
           INFO     Setting up runtime collection of constants                                                                                                                                                   generator.py:222
⠇ Running Pynguin...[21:57:13] INFO     Start Pynguin Test Generation…                                                                                                                                                               generator.py:110
           INFO     Collecting static constants from module under test                                                                                                                                           generator.py:210
           INFO     No constants found                                                                                                                                                                           generator.py:213
           INFO     Setting up runtime collection of constants                                                                                                                                                   generator.py:222
⠼ Running Pynguin...[21:57:14] INFO     Start Pynguin Test Generation…                                                                                                                                                               generator.py:110
           INFO     Collecting static constants from module under test                                                                                                                                           generator.py:210
           INFO     No constants found                                                                                                                                                                           generator.py:213
           INFO     Setting up runtime collection of constants                                                                                                                                                   generator.py:222
⠸ Running Pynguin...[21:57:14] INFO     Start Pynguin Test Generation…                                                                                                                                                               generator.py:110
           INFO     Collecting static constants from module under test                                                                                                                                           generator.py:210
           INFO     No constants found                                                                                                                                                                           generator.py:213
           INFO     Setting up runtime collection of constants                                                                                                                                                   generator.py:222
[09:57:17 PM] INFO     Iteration:      15, Coverage: 0.157241                                                                                                                                                searchobserver.py:73
[21:57:25] INFO     Initial Population, Coverage: 0.099441                                                                                                                                                   searchobserver.py:69
[21:57:29]     ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call

 ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call                                                                                                                                                                           

[21:57:29] INFO     Initial Population, Coverage: 0.110006                                                                                                                                                   searchobserver.py:69
[21:57:30] INFO     Analyzed project to create test cluster                                                                                                                                                        module.py:1296
           INFO     Modules:      60                                                                                                                                                                               module.py:1297
           INFO     Functions:   486                                                                                                                                                                               module.py:1298
           INFO     Classes:     386                                                                                                                                                                               module.py:1299
           INFO     Using seed 1680058622915104278                                                                                                                                                               generator.py:196
           INFO     Using strategy: Algorithm.DYNAMOSA                                                                                                                                          generationalgorithmfactory.py:288
           INFO     Instantiated 1609 fitness functions                                                                                                                                         generationalgorithmfactory.py:380
           INFO     Using CoverageArchive                                                                                                                                                       generationalgorithmfactory.py:332
           INFO     Using selection function: Selection.TOURNAMENT_SELECTION                                                                                                                    generationalgorithmfactory.py:307
           INFO     No stopping condition configured!                                                                                                                                           generationalgorithmfactory.py:111
           INFO     Using fallback timeout of 600 seconds                                                                                                                                       generationalgorithmfactory.py:112
           INFO     Using crossover function: SinglePointRelativeCrossOver                                                                                                                      generationalgorithmfactory.py:320
[21:57:31] INFO     Using ranking function: RankBasedPreferenceSorting                                                                                                                          generationalgorithmfactory.py:340
           INFO     Start generating test cases                                                                                                                                                                  generator.py:523
[21:57:31] ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call

[21:57:32] INFO     Analyzed project to create test cluster                                                                                                                                                        module.py:1296
           INFO     Modules:      60                                                                                                                                                                               module.py:1297
           INFO     Functions:   486                                                                                                                                                                               module.py:1298
           INFO     Classes:     386                                                                                                                                                                               module.py:1299
           INFO     Using seed 1680058622959935474                                                                                                                                                               generator.py:196
           INFO     Using strategy: Algorithm.DYNAMOSA                                                                                                                                          generationalgorithmfactory.py:288
           INFO     Instantiated 1609 fitness functions                                                                                                                                         generationalgorithmfactory.py:380
           INFO     Using CoverageArchive                                                                                                                                                       generationalgorithmfactory.py:332
           INFO     Using selection function: Selection.TOURNAMENT_SELECTION                                                                                                                    generationalgorithmfactory.py:307
           INFO     No stopping condition configured!                                                                                                                                           generationalgorithmfactory.py:111
           INFO     Using fallback timeout of 600 seconds                                                                                                                                       generationalgorithmfactory.py:112
           INFO     Using crossover function: SinglePointRelativeCrossOver                                                                                                                      generationalgorithmfactory.py:320
           INFO     Using ranking function: RankBasedPreferenceSorting                                                                                                                          generationalgorithmfactory.py:340
           INFO     Start generating test cases                                                                                                                                                                  generator.py:523                                                                                                                                                                                                  

[09:57:39 PM] INFO     Iteration:      17, Coverage: 0.167806                                                                                                                                                searchobserver.py:73
[21:57:40] ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call

ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call

[21:57:40]           ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call

ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call

[21:57:41] INFO     Analyzed project to create test cluster                                                                                                                                                        module.py:1296
           INFO     Modules:      60                                                                                                                                                                               module.py:1297
           INFO     Functions:   486                                                                                                                                                                               module.py:1298
           INFO     Classes:     386                                                                                                                                                                               module.py:1299
[21:57:41] INFO     Analyzed project to create test cluster                                                                                                                                                        module.py:1296
           INFO     Modules:      60                                                                                                                                                                               module.py:1297
           INFO     Functions:   486                                                                                                                                                                               module.py:1298
           INFO     Classes:     386                                                                                                                                                                               module.py:1299
           INFO     Using seed 1680058630145117367                                                                                                                                                               generator.py:196
           INFO     Using strategy: Algorithm.DYNAMOSA                                                                                                                                          generationalgorithmfactory.py:288
           INFO     Instantiated 1609 fitness functions                                                                                                                                         generationalgorithmfactory.py:380
           INFO     Using CoverageArchive                                                                                                                                                       generationalgorithmfactory.py:332
           INFO     Using selection function: Selection.TOURNAMENT_SELECTION                                                                                                                    generationalgorithmfactory.py:307
           INFO     Using seed 1680058630236044284                                                                                                                                                               generator.py:196
           INFO     Using strategy: Algorithm.DYNAMOSA                                                                                                                                          generationalgorithmfactory.py:288
           INFO     No stopping condition configured!                                                                                                                                           generationalgorithmfactory.py:111
           INFO     Using fallback timeout of 600 seconds                                                                                                                                       generationalgorithmfactory.py:112
           INFO     Instantiated 1609 fitness functions                                                                                                                                         generationalgorithmfactory.py:380
           INFO     Using CoverageArchive                                                                                                                                                       generationalgorithmfactory.py:332
           INFO     Using crossover function: SinglePointRelativeCrossOver                                                                                                                      generationalgorithmfactory.py:320
           INFO     Using ranking function: RankBasedPreferenceSorting                                                                                                                          generationalgorithmfactory.py:340
           INFO     Using selection function: Selection.TOURNAMENT_SELECTION                                                                                                                    generationalgorithmfactory.py:307
           INFO     No stopping condition configured!                                                                                                                                           generationalgorithmfactory.py:111
           INFO     Using fallback timeout of 600 seconds                                                                                                                                       generationalgorithmfactory.py:112
           INFO     Using crossover function: SinglePointRelativeCrossOver                                                                                                                      generationalgorithmfactory.py:320
           INFO     Start generating test cases                                                                                                                                                                  generator.py:523
           INFO     Using ranking function: RankBasedPreferenceSorting                                                                                                                          generationalgorithmfactory.py:340
           INFO     Start generating test cases                                                                                                                                                                  generator.py:523
[21:57:42] INFO     Analyzed project to create test cluster                                                                                                                                                        module.py:1296
           INFO     Modules:      60                                                                                                                                                                               module.py:1297
           INFO     Functions:   486                                                                                                                                                                               module.py:1298
           INFO     Classes:     386                                                                                                                                                                               module.py:1299
           INFO     Using seed 1680058630223084926                                                                                                                                                               generator.py:196
           INFO     Using strategy: Algorithm.DYNAMOSA                                                                                                                                          generationalgorithmfactory.py:288
           INFO     Instantiated 1609 fitness functions                                                                                                                                         generationalgorithmfactory.py:380
           INFO     Using CoverageArchive                                                                                                                                                       generationalgorithmfactory.py:332
           INFO     Using selection function: Selection.TOURNAMENT_SELECTION                                                                                                                    generationalgorithmfactory.py:307
           INFO     No stopping condition configured!                                                                                                                                           generationalgorithmfactory.py:111
           INFO     Using fallback timeout of 600 seconds                                                                                                                                       generationalgorithmfactory.py:112
           INFO     Using crossover function: SinglePointRelativeCrossOver                                                                                                                      generationalgorithmfactory.py:320
           INFO     Using ranking function: RankBasedPreferenceSorting                                                                                                                          generationalgorithmfactory.py:340
           INFO     Start generating test cases                                                                                                                                                                  generator.py:523
[21:57:45]      ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call

ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call                                                                                                                                                                                             

[21:57:46] INFO     Analyzed project to create test cluster                                                                                                                                                        module.py:1296
           INFO     Modules:      60                                                                                                                                                                               module.py:1297
           INFO     Functions:   486                                                                                                                                                                               module.py:1298
           INFO     Classes:     386                                                                                                                                                                               module.py:1299
           INFO     Using seed 1680058630454705334                                                                                                                                                               generator.py:196
           INFO     Using strategy: Algorithm.DYNAMOSA                                                                                                                                          generationalgorithmfactory.py:288
           INFO     Instantiated 1609 fitness functions                                                                                                                                         generationalgorithmfactory.py:380
           INFO     Using CoverageArchive                                                                                                                                                       generationalgorithmfactory.py:332
           INFO     Using selection function: Selection.TOURNAMENT_SELECTION                                                                                                                    generationalgorithmfactory.py:307
           INFO     No stopping condition configured!                                                                                                                                           generationalgorithmfactory.py:111
           INFO     Using fallback timeout of 600 seconds                                                                                                                                       generationalgorithmfactory.py:112
           INFO     Using crossover function: SinglePointRelativeCrossOver                                                                                                                      generationalgorithmfactory.py:320
           INFO     Using ranking function: RankBasedPreferenceSorting                                                                                                                          generationalgorithmfactory.py:340
           INFO     Start generating test cases                                                                                                                                                                  generator.py:523
[09:57:48 PM] INFO     Iteration:      18, Coverage: 0.172778                                                                                                                                                searchobserver.py:73
[21:57:51] INFO     Initial Population, Coverage: 0.112492                                                                                                                                                   searchobserver.py:69
[09:57:52 PM] INFO     Initial Population, Coverage: 0.123058                                                                                                                                                searchobserver.py:69
/usr/lib/python3.10/subprocess.py:1070: ResourceWarning: subprocess 2427756 is still running
  _warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.10/subprocess.py:1070: ResourceWarning: subprocess 2427822 is still running
  _warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.10/subprocess.py:1070: ResourceWarning: subprocess 2427899 is still running
  _warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.10/subprocess.py:1070: ResourceWarning: subprocess 2427938 is still running
  _warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.10/subprocess.py:1070: ResourceWarning: subprocess 2427955 is still running
  _warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.10/subprocess.py:1070: ResourceWarning: subprocess 2427961 is still running
  _warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.10/subprocess.py:1070: ResourceWarning: subprocess 2427972 is still running
  _warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
[21:57:53] INFO     Iteration:       1, Coverage: 0.100062                                                                                                                                                   searchobserver.py:73
[21:57:54] INFO     Iteration:       1, Coverage: 0.114357                                                                                                                                                   searchobserver.py:73
⠼ Running Pynguin...[21:57:59] INFO     Start Pynguin Test Generation…                                                                                                                                                               generator.py:110
           INFO     Collecting static constants from module under test                                                                                                                                           generator.py:210
           INFO     No constants found                                                                                                                                                                           generator.py:213
           INFO     Setting up runtime collection of constants                                                                                                                                                   generator.py:222
⠦ Running Pynguin...[21:57:59] INFO     Start Pynguin Test Generation…                                                                                                                                                               generator.py:110
           INFO     Collecting static constants from module under test                                                                                                                                           generator.py:210
           INFO     No constants found                                                                                                                                                                           generator.py:213
           INFO     Setting up runtime collection of constants                                                                                                                                                   generator.py:222
[21:57:59] INFO     Stop Pynguin Test Generation…                                                                                                                                                                generator.py:113
[21:57:59] INFO     Stop Pynguin Test Generation…                                                                                                                                                                generator.py:113
[21:57:59] INFO     Stop Pynguin Test Generation…                                                                                                                                                                generator.py:113
[09:57:59 PM] INFO     Stop Pynguin Test Generation…                                                                                                                                                             generator.py:113
           INFO     Stop Pynguin Test Generation…                                                                                                                                                                generator.py:113
[09:57:59 PM] INFO     Stop Pynguin Test Generation…                                                                                                                                                             generator.py:113
           INFO     Stop Pynguin Test Generation…                                                                                                                                                                generator.py:113
[21:57:59] INFO     Stop Pynguin Test Generation…                                                                                                                                                                generator.py:113
[21:57:59] INFO     Stop Pynguin Test Generation…                                                                                                                                                                generator.py:113
[21:57:59] INFO     Stop Pynguin Test Generation…                                                                                                                                                                generator.py:113
[09:57:59 PM] INFO     Stop Pynguin Test Generation…

At which point I had begun interrupting the process.

Incidentally, as seen in the log above, any file that imports urlllib appears to run into two errors that do not halt the program. These are the only logging ERRORs that occur. It does not appear related, but I'll put it here in case:

DEBUG Analysing method module.py:1188
urllib.request.URLopener.open_ftp
ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call

...
DEBUG Analysing method module.py:1188
urllib.request.ftpwrapper.real_close
ERROR While getting the types of exceptions in syntaxtree.py:336
the handler, expected to find an ast.Name,
ast.Tuple, or ast.Attribute, but got ast.Call

Nothing else in the debug level log appears relevant, but I can provide it if wanted.

Unfortunately a bug in the bytecode module prevented it from handling EXTENDED_ARG NOPs in certain situations, which bottle generates. Backporting the fix was declined by the bytecode maintainers, so this means you will either have to:

A) Update Pynguin to be compatible with the latest bytecode version for which a PR fix I sent was accepted. I don't think this fix is released on PyPI yet, but could be easily installed from source.

B) Install a locally modified version of 0.13.0 with the fix shown here https://github.com/EmmaJaneBonestell/bytecode/blob/54a1af74f33dfa323d920540fc8bab8e18b7e64c/bytecode/concrete.py#L371 , or just install from my fork/branch directly: pip install git+https://github.com/EmmaJaneBonestell/bytecode.git@v0.13

After that, no special commands are required to reproduce. e.g.:

git clone https://github.com/bottlepy/bottle ./bottle
pynguin --project-path ./bottle --output-path ./pynguin_output --module-name bottle

This normally happens around 30 iterations. It may take fewer, as it did in the above log, or nearly 200. It has never managed to reach the 600 second default timeout/full coverage without occurring.

stephanlukasczyk commented 1 year ago

Thank you for reporting this.

I have updated bytecode to include the fix in a149848488abdd82956793a9149eefc3282b3bcd. However, I cannot make a release, because PyPI does not seem to allow me to push a release version that depends on a git version of a dependency. Nevertheless, for local testing, it should be fine if you directly install the Pynguin version from GitHub. As soon as there is a new bytecode release, I'll make a Pynguin release, too.

EmmaJaneBonestell commented 1 year ago

While this did fix the need to vendor bytecode, attempting to run Pynguin on the bottle module still results in multiple instances of Pynguin.

stephanlukasczyk commented 1 year ago

I agree, just checked this with latest Pynguin on bottle. However, while I am not entirely sure (and I do currently not have the time to debug this to the end), I doubt that the issue is easily fixable in Pynguin.

What I saw from running it is that there is a sub-process running Pynguin again being spawned from the initial Pynguin process. Pynguin does not have any code that spawns sub-processes on its own. It only uses threads to better control test-case execution.

Thus, I assumed bottle.py being the one that causes the sub process. And, at least, there is a line that could cause a sub-process being spawned: https://github.com/bottlepy/bottle/blob/99341ff3791b2e7e705d7373e71937e9018eb081/bottle.py#L3667 From what I see from Pynguin's logs is that the run function, where this code is part of, is executed. Hence, I assume that there is the chance of reaching this line with appropriate parameters that basically re-start Pynguin in a sub process.

Please note that there is a similar issue (#41) when trying to generate tests for a flask app, which might be a similar problem (since both flask and bottle are WSGI frameworks).

For now, I do not know about an easy and quick fix for the issue. Unfortunately, I am busy with other things, which prevents me from digging deeper in the problem.