PyCall on thread != main: process will not exit

[snickell]

If you use PyCall from only one thread , but that thread is NOT the main thread, the process will not exit when the main thread exits.

This is not the same issue as: "Is PyCall Thread Safe" #96, as we are only using PyCall from one thread:

1. PyCall is only called from one thread: "side_thread"
2. side_thread exits
3. Process still does not exit

Example:

#!/usr/bin/env ruby

def run(do_pycall_import:)
  if do_pycall_import
    puts "Running with do_pycall_import=true, process will not exit when main thread is done"
  else
    puts "Running with do_pycall_import=false, process will exit when main thread is done"
  end

  puts "Main thread ID: #{Thread.current.object_id}\n"

  def print_threads
    puts
    puts "Threads:"
    Thread.list.each do |thread|
      puts "\tThread ID: #{thread.object_id}, Status: #{thread.status}, Name: #{thread.name}"
    end
    puts
  end

  Thread.new do
    Thread.current.name = "side_thread"
    Thread.current.abort_on_exception = true

    # Demonstrate that PyCall has not been used:
    raise "Only load pycall in this thread" if defined?(PyCall)

    require 'pycall'

    if do_pycall_import
      puts "side_thread: import sys"
      # This will initialize libpython, if this happens, the process wil not exit:
      PyCall.import_module('sys')
    end

    sleep 2
    puts "side_thread: exiting"
  end

  sleep 1
  print_threads() #=> Two threads: main and side_thread

  sleep 4
  print_threads() #=> One thread: main
end

if __FILE__ == $0
  run(do_pycall_import: true) #=> Process does NOT exit after printing "End of main thread"
  # run(do_pycall_import: false) #=> Process exits after printing "End of main thread"
end

at_exit { puts "at_exit called"}
puts "End of main thread"

Output when do_pycall_import: true

zsh >> ./pycall_hangs_main_thread.rb
Running with do_pycall_import=true, process will not exit when main thread is done
Main thread ID: 60
side_thread: import sys

Threads:
        Thread ID: 60, Status: run, Name: 
        Thread ID: 80, Status: sleep, Name: side_thread

side_thread: exiting

Threads:
        Thread ID: 60, Status: run, Name: 

End of main thread
at_exit called

#=> control never returns to the shell

Output when do_pycall_import: false

zsh >> ./pycall_hangs_main_thread.rb
Running with do_pycall_import=false, process will exit when main thread is done
Main thread ID: 60

Threads:
        Thread ID: 60, Status: run, Name: 
        Thread ID: 80, Status: sleep, Name: side_thread

side_thread: exiting

Threads:
        Thread ID: 60, Status: run, Name: 

End of main thread
at_exit called

zsh >> # notice, process exited

[snickell]

I noticed we do not call Py_FinalizeEx(). If I call Py_FinalizeEx() at the end of my thread: the process exits! 🥳

Changes in my repro code:

  Thread.new do

    # CUT TO MAKE SHORT

    # I have modified pycall.c to add this method, which calls Py_API(Py_FinalizeEx)()
    PyCall.finalize
  end

With these changes, now the main process exits.

[snickell]

I have started a fix for this here: https://github.com/mrkn/pycall.rb/pull/187, but its not complete yet. It permits the process to exit, but sometimes it segfaults at exit.