Cannot automate window when running another program using GTK

[bensandland]

I am having trouble with interacting with my RAutomation::Window class when running a test using some testing software utilizing GTK. The test will execute as intended when I am not using the GUI, or if i hide the UI by setting the windows visibility to false.

I have tried using both GTK2 and GTK3 - the issue is present in both versions. Setting the adapter to ms_uia also does nothing, because it looks like it is the ffi calls that are causing the program to stall.

I am calling the 'exists?' function in RAutomation, which will try to get the hwnd using ffi - this is where the program will stall indefinitely.

  def start_program
      # calling exists? here, which causes the stall
      Kernel.system("cd \"#{@path}\" && start #{@program}") unless get_hwnd.exists?
   end

   def get_hwnd
      # getting window class using specified title here
      window = RAutomation::Window.new(title: @title, :adapter => :ms_uia)
      return window
   end

Using Ruby 2.5.1, RAutomation 1.1.0 and ffi 1.15.5

[jarmo]

Not sure why it stalls there and I guess you need to go deep into it with setting some breakpoints here and there (or add logging). However, I don't think that you can do much useful with RAutomation using GTK applications. Not sure though, but I have a suspicion that they don't work as "native" applications.

[bensandland]

I am not using RAutomation to interact with the GTK application. The GTK application is a UI for a test framework we use. This application will call the code using RAutomation, which then will start another process it is supposed to automate.

When this exists? (located in window.rb) function is called, the exists? within win_32/window.rb is called, since we're using ffi.

def exists?
   result = hwnd && Functions.window_exists(hwnd)
   !!result
end

This function calls hwnd to find the handle.

def hwnd
   @hwnd ||= Functions.window_hwnd(@locators)
end

window_hwnd then calls find_hwnd, which it seems like it never returns from

def find_hwnd(locators, window_hwnd = nil)
   found_hwnd = nil
   found_index = -1
   window_callback = FFI::Function.new(:bool, [:long, :pointer], {:convention => :stdcall}) do |hwnd, _|
     if yield(hwnd)
       found_index += 1

       if locators[:index]
         found_hwnd = hwnd if locators[:index] == found_index
       else
         found_hwnd = hwnd
       end
     end
     !found_hwnd
   end

   unless window_hwnd
     enum_windows(window_callback, nil)
   else
     enum_child_windows(window_hwnd, window_callback, nil)
   end

   found_hwnd # this return statement is never executed
end

When I run the same test without using the GTK GUI, the window handle is found without any issues.

I have also tried using the ms_uia adapter, but it looks like the exact same issue is present.

I hope you understand what I mean. Please let me know if there is any more information I can provide. I am unable to draw any meaningful conclusion from this.

[jarmo]

Did you manage to solve this issue?

Sorry that I didn't manage to help with this. Without having access to the application and not have been active with RAutomation for years, I just can't help you. I can recommend you to try StackOverflow, maybe somewhere there can help.

[bensandland]

No worries, I figured as much - it was a bit of long shot opening this issue :-)

I managed to find the cause of the issue, but still working on how to fix it. It is due to multi threading - our GTK application opens up a new thread which will call the code containing the RAutomation calls. If I move this code outside of the thread, it works out fine. Have to do a bit more digging to figure out a long term solution.