search

micro-manager / pycro-manager

Python control of micro-manager for customized data acquisition
https://pycro-manager.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
159 stars 50 forks source link

Errors on shutdown with multiple headless MM instances. #763

Open carlkesselman opened 2 months ago

carlkesselman commented 2 months ago

Hi,

I'm seeing an issue on shutdown when having multiple headless instances. I haven't had a chance to hunt this down further. Here is the exception:

Exception ignored in atexit callback: <function _JavaObjectShadow.__init__.<locals>.cleanup at 0x000001EF7101C040>
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\spim-data\Lib\site-packages\pyjavaz\bridge.py", line 560, in cleanup
    wr()._close()
  File "C:\ProgramData\Anaconda3\envs\spim-data\Lib\site-packages\pyjavaz\bridge.py", line 575, in _close
    self._send(message)
  File "C:\ProgramData\Anaconda3\envs\spim-data\Lib\site-packages\pyjavaz\bridge.py", line 593, in _send
    return self._get_bridge()._send(message)
           ^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\Anaconda3\envs\spim-data\Lib\site-packages\pyjavaz\bridge.py", line 619, in _get_bridge
    bridge_to_use = _Bridge.create_or_get_existing_bridge(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\Anaconda3\envs\spim-data\Lib\site-packages\pyjavaz\bridge.py", line 259, in create_or_get_existing_bridge
    b = _Bridge(port, convert_camel_case, debug, ip_address, timeout, iterate)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\Anaconda3\envs\spim-data\Lib\site-packages\pyjavaz\bridge.py", line 300, in __init__
    raise Exception(
Exception: Socket timed out after 3000 milliseconds
Exception ignored in: <function _Bridge.__del__ at 0x000001EF6A766520>
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\spim-data\Lib\site-packages\pyjavaz\bridge.py", line 323, in __del__
    del _Bridge._cached_bridges_by_port_and_thread[self._port_thread_id]
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
KeyError: (4908, 23660)

Code for reproduction

# your code here

Expected outcome

Actual outcome

henrypinkard commented 2 months ago

I could imagine that this was introduced when I fixed https://github.com/micro-manager/pycro-manager/issues/737.

Do you see this consistently? Could you make a minimal script for reproducing?

carlkesselman commented 2 months ago

Yes, I do.

Let me see if I can come up with a small stand-alone example of this.

Carl

Dr. Carl Kesselman William H. Keck Professor of Engineering Professor, Epstein Department of Industrial and Systems Engineering Director Informatics Systems Research Division, Information Sciences Institute Viterbi School of Engineering

Professor, Department of Population and Public Health Sciences, Keck School of Medicine Professor, Biomedical Sciences, Ostrow School of Dentistry

University of Southern California 4676 Admiralty Way, Suite 1001, Marina del Rey, CA 90292-6695 Phone: +1 (310) 448-9338 Email: @.*** Web: http://www.isi.edu/~carl

On Apr 22, 2024, at 9:10 AM, Henry Pinkard @.***> wrote:

I could imagine that this was introduced when I fixed #737https://urldefense.us/v2/url?u=https-3A__github.com_micro-2Dmanager_pycro-2Dmanager_issues_737&d=DwMFaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=sGCma2ufaUVT-N141kRIZQ&m=qAJ6nbFxaUeooZehndOfIG-ilSpYaG0QXXwhWLU5DzDbUWaFzxLYlbYN3mjqo7S_&s=JnqDxtSaThuNXOMIL31srCkJBduoEtOMo_2Owatp1tA&e=.

Do you see this consistently? Could you make a minimal script for reproducing?

— Reply to this email directly, view it on GitHubhttps://urldefense.us/v2/url?u=https-3A__github.com_micro-2Dmanager_pycro-2Dmanager_issues_763-23issuecomment-2D2070062502&d=DwMFaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=sGCma2ufaUVT-N141kRIZQ&m=qAJ6nbFxaUeooZehndOfIG-ilSpYaG0QXXwhWLU5DzDbUWaFzxLYlbYN3mjqo7S_&s=pdQCNj0ciiJOSoGZeIs92ms4nzwqQ7wwS6c_F7g5y7Y&e=, or unsubscribehttps://urldefense.us/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AA3OGXTYDJUN2ZUM6VO4OMLY6UY67AVCNFSM6AAAAABGPVK4E6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZQGA3DENJQGI&d=DwMFaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=sGCma2ufaUVT-N141kRIZQ&m=qAJ6nbFxaUeooZehndOfIG-ilSpYaG0QXXwhWLU5DzDbUWaFzxLYlbYN3mjqo7S_&s=aFShIFLTEAHbuO1RnW2o7qG-Faec-NqLiyRNFSuLSys&e=. You are receiving this because you authored the thread.Message ID: @.***>

carlkesselman commented 2 months ago

Ok,

I figured it out. I was calling stop_headless explicitly, and this caused the problem, as it was being called twice, once from my code and once from the onexit.

Carl

Dr. Carl Kesselman William H. Keck Professor of Engineering Professor, Epstein Department of Industrial and Systems Engineering Director Informatics Systems Research Division, Information Sciences Institute Viterbi School of Engineering

Professor, Department of Population and Public Health Sciences, Keck School of Medicine Professor, Biomedical Sciences, Ostrow School of Dentistry

University of Southern California 4676 Admiralty Way, Suite 1001, Marina del Rey, CA 90292-6695 Phone: +1 (310) 448-9338 Email: @.*** Web: http://www.isi.edu/~carl

On Apr 22, 2024, at 9:10 AM, Henry Pinkard @.***> wrote:

I could imagine that this was introduced when I fixed #737https://urldefense.us/v2/url?u=https-3A__github.com_micro-2Dmanager_pycro-2Dmanager_issues_737&d=DwMFaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=sGCma2ufaUVT-N141kRIZQ&m=qAJ6nbFxaUeooZehndOfIG-ilSpYaG0QXXwhWLU5DzDbUWaFzxLYlbYN3mjqo7S_&s=JnqDxtSaThuNXOMIL31srCkJBduoEtOMo_2Owatp1tA&e=.

Do you see this consistently? Could you make a minimal script for reproducing?

— Reply to this email directly, view it on GitHubhttps://urldefense.us/v2/url?u=https-3A__github.com_micro-2Dmanager_pycro-2Dmanager_issues_763-23issuecomment-2D2070062502&d=DwMFaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=sGCma2ufaUVT-N141kRIZQ&m=qAJ6nbFxaUeooZehndOfIG-ilSpYaG0QXXwhWLU5DzDbUWaFzxLYlbYN3mjqo7S_&s=pdQCNj0ciiJOSoGZeIs92ms4nzwqQ7wwS6c_F7g5y7Y&e=, or unsubscribehttps://urldefense.us/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AA3OGXTYDJUN2ZUM6VO4OMLY6UY67AVCNFSM6AAAAABGPVK4E6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZQGA3DENJQGI&d=DwMFaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=sGCma2ufaUVT-N141kRIZQ&m=qAJ6nbFxaUeooZehndOfIG-ilSpYaG0QXXwhWLU5DzDbUWaFzxLYlbYN3mjqo7S_&s=aFShIFLTEAHbuO1RnW2o7qG-Faec-NqLiyRNFSuLSys&e=. You are receiving this because you authored the thread.Message ID: @.***>

henrypinkard commented 2 months ago

Gotcha. Maybe this means there should be a check in that function to see if it has already been called and then just give a warning instead?

carlkesselman commented 2 months ago

Probably… or handle the case where the process you want to shut down has already been shut down.

I think that this used to work for me, so something somewhere must have changed.

Carl

Dr. Carl Kesselman William H. Keck Professor of Engineering Professor, Epstein Department of Industrial and Systems Engineering Director Informatics Systems Research Division, Information Sciences Institute Viterbi School of Engineering

Professor, Department of Population and Public Health Sciences, Keck School of Medicine Professor, Biomedical Sciences, Ostrow School of Dentistry

University of Southern California 4676 Admiralty Way, Suite 1001, Marina del Rey, CA 90292-6695 Phone: +1 (310) 448-9338 Email: @.*** Web: http://www.isi.edu/~carl

On Apr 24, 2024, at 10:46 AM, Henry Pinkard @.***> wrote:

Gotcha. Maybe this means there should be a check in that function to see if it has already been called and then just give a warning instead?

— Reply to this email directly, view it on GitHubhttps://urldefense.us/v2/url?u=https-3A__github.com_micro-2Dmanager_pycro-2Dmanager_issues_763-23issuecomment-2D2075505423&d=DwMFaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=sGCma2ufaUVT-N141kRIZQ&m=eeSkf7OwO98Ami2lA-a7LTAIoPhmjSnTrxp6K7mSi4d2RzF-W9Zl6BrpZzKx24_L&s=st3i1Ij_hmflVDf9ReDHXCL8r08Qng_Rw5qzGY1NlxQ&e=, or unsubscribehttps://urldefense.us/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AA3OGXRE5GKYM6IQ2RBA2BLY67VYBAVCNFSM6AAAAABGPVK4E6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZVGUYDKNBSGM&d=DwMFaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=sGCma2ufaUVT-N141kRIZQ&m=eeSkf7OwO98Ami2lA-a7LTAIoPhmjSnTrxp6K7mSi4d2RzF-W9Zl6BrpZzKx24_L&s=o9hj4VOUQXjZg7fNAAxp9CrSdsSZDHxEYQeZkL1Puqw&e=. You are receiving this because you authored the thread.Message ID: @.***>