MEDM not responsive after acquiring image in "continuous" mode and Mono12Packed Pixel format

[LeeYangLBLBCS]

issue moved from: https://github.com/epics-extensions/medm/issues/6

[MarkRivers]

Is this issue still happening? Can you describe the symptoms in more detail? Are there any error messages on the IOC console when this happens?

[LeeYangLBLBCS]

yes, this is still happening. I just recreated it with the "new" computer without any problems. There is no error on IOC terminal, except it does not respond to any command. The only way to exit is ctrl-C. It happens in continuous mode fairly quickly. It also happens in multiple image mode when the number of images is large enough, such as 1000 images. It did not happen in multiple image mode with up to a few hundred images. screen attached shows multiple image mode with 1000 images, where the "acquiring" is frozen at image number 993.

[MarkRivers]

There is no error on IOC terminal, except it does not respond to any command. The only way to exit is ctrl-C.

Do you mean that if you press on the keyboard you don't get a new iocsh prompt? If so, that it is a problem I have never seen, and it sounds like it could be a resource depletion issue.

Please post a screen shot of the Plugins/All screen when the system has collected about 900 images in Multiple mode, or when it has collected more than 1000 images in Continuous mode.

[MarkRivers]

@LeeYangLBLBCS how much memory does your computer have, and how many cores?

[LeeYangLBLBCS]

correction: the IOC responds to command input. I checked epicsMutexShowAll 1, the results show 0: epics> epicsMutexShowAll 1 ellCount(&mutexList) 37822 ellCount(&freeList) 0

However, I found that It also happens in mono16 pixel format.

More information: In Mono12Packed mode, if I stop acquisition before it reaches 1000 images, usually it will stop normally after a few more frames taken. If I click stop more than once, it will usually show "acquiring" permanently. At this time, I see two mutex in IOC terminal: epics> epicsMutexShowAll 1 ellCount(&mutexList) 37719 ellCount(&freeList) 0 epicsMutexId 0x55d7011c5970 source ../../asyn/asynPortDriver/asynPortDriver.cpp line 3700 epicsMutexId 0x55d7011c7c20 source ../../asyn/asynDriver/asynManager.c line 1975 epics>

[LeeYangLBLBCS]

I think it's 6 core, 64GB. cpu info attached.

On Thu, Aug 27, 2020 at 8:16 AM Mark Rivers notifications@github.com wrote:

@LeeYangLBLBCS https://github.com/LeeYangLBLBCS how much memory does your computer have, and how many cores?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/areaDetector/ADSpinnaker/issues/4#issuecomment-682013892, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSYGNGT3O7YRJSNG7H34GLSCZ2DBANCNFSM4PJ2ADJQ .

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 63 model name : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz stepping : 2 microcode : 0x43 cpu MHz : 1197.250 cache size : 15360 KB physical id : 0 siblings : 12 core id : 0 cpu cores : 6 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 15 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit bogomips : 6984.39 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management:

processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 63 model name : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz stepping : 2 microcode : 0x43 cpu MHz : 1197.352 cache size : 15360 KB physical id : 0 siblings : 12 core id : 1 cpu cores : 6 apicid : 2 initial apicid : 2 fpu : yes fpu_exception : yes cpuid level : 15 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit bogomips : 6984.39 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management:

processor : 2 vendor_id : GenuineIntel cpu family : 6 model : 63 model name : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz stepping : 2 microcode : 0x43 cpu MHz : 1197.438 cache size : 15360 KB physical id : 0 siblings : 12 core id : 2 cpu cores : 6 apicid : 4 initial apicid : 4 fpu : yes fpu_exception : yes cpuid level : 15 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit bogomips : 6984.39 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management:

processor : 3 vendor_id : GenuineIntel cpu family : 6 model : 63 model name : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz stepping : 2 microcode : 0x43 cpu MHz : 1197.555 cache size : 15360 KB physical id : 0 siblings : 12 core id : 3 cpu cores : 6 apicid : 6 initial apicid : 6 fpu : yes fpu_exception : yes cpuid level : 15 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit bogomips : 6984.39 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management:

processor : 4 vendor_id : GenuineIntel cpu family : 6 model : 63 model name : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz stepping : 2 microcode : 0x43 cpu MHz : 1197.343 cache size : 15360 KB physical id : 0 siblings : 12 core id : 4 cpu cores : 6 apicid : 8 initial apicid : 8 fpu : yes fpu_exception : yes cpuid level : 15 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit bogomips : 6984.39 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management:

processor : 5 vendor_id : GenuineIntel cpu family : 6 model : 63 model name : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz stepping : 2 microcode : 0x43 cpu MHz : 1197.355 cache size : 15360 KB physical id : 0 siblings : 12 core id : 5 cpu cores : 6 apicid : 10 initial apicid : 10 fpu : yes fpu_exception : yes cpuid level : 15 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit bogomips : 6984.39 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management:

processor : 6 vendor_id : GenuineIntel cpu family : 6 model : 63 model name : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz stepping : 2 microcode : 0x43 cpu MHz : 1197.349 cache size : 15360 KB physical id : 0 siblings : 12 core id : 0 cpu cores : 6 apicid : 1 initial apicid : 1 fpu : yes fpu_exception : yes cpuid level : 15 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit bogomips : 6984.39 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management:

processor : 7 vendor_id : GenuineIntel cpu family : 6 model : 63 model name : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz stepping : 2 microcode : 0x43 cpu MHz : 1197.899 cache size : 15360 KB physical id : 0 siblings : 12 core id : 1 cpu cores : 6 apicid : 3 initial apicid : 3 fpu : yes fpu_exception : yes cpuid level : 15 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit bogomips : 6984.39 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management:

processor : 8 vendor_id : GenuineIntel cpu family : 6 model : 63 model name : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz stepping : 2 microcode : 0x43 cpu MHz : 1197.506 cache size : 15360 KB physical id : 0 siblings : 12 core id : 2 cpu cores : 6 apicid : 5 initial apicid : 5 fpu : yes fpu_exception : yes cpuid level : 15 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit bogomips : 6984.39 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management:

processor : 9 vendor_id : GenuineIntel cpu family : 6 model : 63 model name : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz stepping : 2 microcode : 0x43 cpu MHz : 1198.277 cache size : 15360 KB physical id : 0 siblings : 12 core id : 3 cpu cores : 6 apicid : 7 initial apicid : 7 fpu : yes fpu_exception : yes cpuid level : 15 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit bogomips : 6984.39 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management:

processor : 10 vendor_id : GenuineIntel cpu family : 6 model : 63 model name : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz stepping : 2 microcode : 0x43 cpu MHz : 1197.553 cache size : 15360 KB physical id : 0 siblings : 12 core id : 4 cpu cores : 6 apicid : 9 initial apicid : 9 fpu : yes fpu_exception : yes cpuid level : 15 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit bogomips : 6984.39 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management:

processor : 11 vendor_id : GenuineIntel cpu family : 6 model : 63 model name : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz stepping : 2 microcode : 0x43 cpu MHz : 1197.566 cache size : 15360 KB physical id : 0 siblings : 12 core id : 5 cpu cores : 6 apicid : 11 initial apicid : 11 fpu : yes fpu_exception : yes cpuid level : 15 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit bogomips : 6984.39 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management:

[LeeYangLBLBCS]

All of the computers we use are from one vendor, Supermicro. https://www.supermicro.com/en/products/motherboard/X10SRA Maybe it could be the motherboard?

On Thu, Aug 27, 2020 at 8:52 AM Lee L. Yang llyang@lbl.gov wrote:

I think it's 6 core, 64GB. cpu info attached.

On Thu, Aug 27, 2020 at 8:16 AM Mark Rivers notifications@github.com wrote:

@LeeYangLBLBCS https://github.com/LeeYangLBLBCS how much memory does your computer have, and how many cores?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/areaDetector/ADSpinnaker/issues/4#issuecomment-682013892, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSYGNGT3O7YRJSNG7H34GLSCZ2DBANCNFSM4PJ2ADJQ .

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

[MarkRivers]

I want to be clear on exactly what symptoms you are seeing.

If you configure your system as I show below: GC_AdcBitDepth=Bit10 PixelFormat=Mono12Packed ConvertPixelFormat=Mono16 FrameRate=105 (actual 104.984) ImageMode=Multiple NumImages=1000

If you acquire in this mode does it ever fail to acquire 1000 images? It does not for me.

However, it you try to stop the acquisition before it completes normally, does it sometimes get into this state where it is stuck with Acquire=1 (says Collecting)?

That is the problem I see.

At that point epicsMutexShowAll 1 does show a deadlock:

epics> epicsMutexShowAll 1 ellCount(&mutexList) 38759 ellCount(&freeList) 0 epicsMutexId 0x556e97136c80 source ../../asyn/asynPortDriver/asynPortDriver.cpp line 3712 epicsMutexId 0x556e97138950 source ../../asyn/asynDriver/asynManager.c line 1986

Typing "exit" at the iocsh prompt does not work, I have to press ^C.

The same thing happens in Continuous mode. When I press Stop it often hangs.

[MarkRivers]

Maybe it could be the motherboard?

I think I am a seeing the same problem as you. It is a bug in my driver. I am working on it now. I will let you know when I think I have it fixed.

[LeeYangLBLBCS]

yes, I see exactly what you saw. Plus, it happens in Mono16 format also. It never can reach 1000 images. But if I can always stop it without problems, either before stuck acquiring at the end, or in the middle of acquiring.

On Thu, Aug 27, 2020 at 10:43 AM Mark Rivers notifications@github.com wrote:

Maybe it could be the motherboard?

I think I am a seeing the same problem as you. It is a bug in my driver. I am working on it now. I will let you know when I think I have it fixed.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/areaDetector/ADSpinnaker/issues/4#issuecomment-682094900, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSYGNBIKT7ZOVGE4QZ4ES3SC2LNLANCNFSM4PJ2ADJQ .

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

[MarkRivers]

yes, I see exactly what you saw. Plus, it happens in Mono16 format also. It never can reach 1000 images. But if I can always stop it without problems, either before stuck acquiring at the end, or in the middle of acquiring.

I am confused by what you are saying. In my case if I set Multiple mode and NumImages=1000 it always acquires 1000 images with no problem. You seem to be saying that it never collects all 1000 images?

My only problem is trying to stop it manually by pressing Stop, either in Multiple mode or Continuous mode.

[MarkRivers]

I just tried PixelFormat=Mono16 and ConvertPixelFormat=None. I see a few frames are dropped with these messages:

2020/08/27 13:31:56.391 ADSpinnaker::grabImage error GetImageStatus returned 9
2020/08/27 13:31:56.393 ADSpinnaker::grabImage error GetImageStatus returned 9

In this case 2 frames had missing packets, and so it only collected 998 frames, not 1000. That is "normal". When I pressed Stop it changed back to the Idle state and I could collect again.

[LeeYangLBLBCS]

I made a mistake when I ran mono16. The frame rate needs to be lowered to 70 fps. I left it at 105 which only works for mono12Pack. So, the mono16 has no problem. Only mono12Pack has the same problem in both of our computers. Also, I see message on IOC terminal in mono12Pack mode when I interrupted the acquiring: 2020/08/27 12:19:02.295 ADSpinnaker::grabImage pixel format conversion exception Spinnaker: Parameter is not initialized. Input image is NULL [-1009] 2020/08/27 12:19:02.295 ADSpinnaker:grabImage: unsupported pixel format=0xb It may be normal. Not tied to the permanent Acquiring state.

On Thu, Aug 27, 2020 at 11:37 AM Mark Rivers notifications@github.com wrote:

I just tried PixelFormat=Mono16 and ConvertPixelFormat=None. I see a few frames are dropped with these messages:

2020/08/27 13:31:56.391 ADSpinnaker::grabImage error GetImageStatus returned 9 2020/08/27 13:31:56.393 ADSpinnaker::grabImage error GetImageStatus returned 9

In this case 2 frames had missing packets, and so it only collected 998 frames, not 1000. That is "normal". When I pressed Stop it changed back to the Idle state and I could collect again.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/areaDetector/ADSpinnaker/issues/4#issuecomment-682121255, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSYGNF3M7JDTVN3JFCWYLDSC2RVDANCNFSM4PJ2ADJQ .

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

[MarkRivers]

Yes, for some reason the maximum frame rate it not updating as it should for Mono16 and Mono12Packed.

I have isolated the problem to what appears to be a bug in their SDK. When I call pCamera->EndAcquisition() when acquiring Mono12Packed and converting to Mono16 that call hangs forever. There are some comments in the code indicating that I have seen this before.

ADSpinnaker is currently using Spinnaker 1.20.0.14. That is quite old now, and I am going to try the latest version, 2.0.0.147 and see if we still have the problem.

[MarkRivers]

I have created a new SDK_2.0 branch. It uses the most recent Spinnaker SDK from FLIR (2.0.0.147).

It seems to fix the problem of stopping acquistion hanging the IOC with Acquire=1 when using PixelFormat=Mono12Packed and ConvertPixelFormat=Mono16. I can now stop acquistion in Multiple or Continuous mode with those parameters with no problem now, it never hangs.

It seems like it might drop a few more frames compared to the older SDK with the Oryx camera, but I need to test this more to be sure. @LeeYangLBLBCS I would be interested in you testing to see if this fixes the problem for you, and whether you see more dropped frames. By dropped frames I mean Multiple mode asking for 1000 frames and seeing it only collect 995, etc.

[LeeYangLBLBCS]

I'm getting ready to test SDK2.0. I'll let you know ASAP.

On Thu, Aug 27, 2020 at 3:16 PM Mark Rivers notifications@github.com wrote:

I have created a new SDK_2.0 branch. It uses the most recent Spinnaker SDK from FLIR (2.0.0.147).

It seems to fix the problem of stopping acquistion hanging the IOC with Acquire=1 when using PixelFormat=Mono12Packed and ConvertPixelFormat=Mono16. I can now stop acquistion in Multiple or Continuous mode with those parameters with no problem now, it never hangs.

It seems like it might drop a few more frames compared to the older SDK with the Oryx camera, but I need to test this more to be sure. @LeeYangLBLBCS https://github.com/LeeYangLBLBCS I would be interested in you testing to see if this fixes the problem for you, and whether you see more dropped frames. By dropped frames I mean Multiple mode asking for 1000 frames and seeing it only collect 995, etc.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/areaDetector/ADSpinnaker/issues/4#issuecomment-682218548, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSYGNDMZRDD6XROFKP7E3LSC3LNJANCNFSM4PJ2ADJQ .

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

[LeeYangLBLBCS]

I tested with SDK2.0. It works in mono16 pixel format, either continuous or multiple at 1000, fixed frame rate or not. It works in mono12pack format at fixed frame rate 105. It can't stop in mono12pack format with non fixed frame rate (which runs at 130 fps, much higher than 105).

On Thu, Aug 27, 2020 at 4:16 PM Lee L. Yang llyang@lbl.gov wrote:

I'm getting ready to test SDK2.0. I'll let you know ASAP.

On Thu, Aug 27, 2020 at 3:16 PM Mark Rivers notifications@github.com wrote:

I have created a new SDK_2.0 branch. It uses the most recent Spinnaker SDK from FLIR (2.0.0.147).

It seems to fix the problem of stopping acquistion hanging the IOC with Acquire=1 when using PixelFormat=Mono12Packed and ConvertPixelFormat=Mono16. I can now stop acquistion in Multiple or Continuous mode with those parameters with no problem now, it never hangs.

It seems like it might drop a few more frames compared to the older SDK with the Oryx camera, but I need to test this more to be sure. @LeeYangLBLBCS https://github.com/LeeYangLBLBCS I would be interested in you testing to see if this fixes the problem for you, and whether you see more dropped frames. By dropped frames I mean Multiple mode asking for 1000 frames and seeing it only collect 995, etc.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/areaDetector/ADSpinnaker/issues/4#issuecomment-682218548, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSYGNDMZRDD6XROFKP7E3LSC3LNJANCNFSM4PJ2ADJQ .

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

[MarkRivers]

If you use Multiple mode and 1000 images does it collect exactly 1000 images. Try 8, 12, 16 bit, fixed rate and free running.

[LeeYangLBLBCS]

All pixel formats, mono8/16/12pack/12p work correctly in "fixed frame" rate. All 1000 images collected and acquisition stopped any time.

For "non fixed frame rate", some pixel format can't be stopped. Though they all finish 1000 frames correctly. Among them,mono8/mono16 worked (ADC=8/10/12). mono12pack/monop (ADC =8/10), acquired 1000 images correctly but can't be stopped. momo12pack/monop in ADC=12bit worked fine in all cases.

Apparently the non fixed frame rates are programmed to use higher values that cause problems. Are they calculated in the SDK based on some kind of theoretical settings? I believe they are not going to be correct at all times for all systems.

We acquire images on a fixed interval triggered by motor output pulses at a steady rate, so this will not be the problem for our application. (We always have to calibrate the highest frame rate achievable right before the tomography run.)

[LeeYangLBLBCS]

I have been able to run external trigger TTL pulse train at 11 msec periodicity reliably with number of pulses at the max. possible value for NumImages of 65535. Pixel format is mono12pack. ADC at 10 bit. This is close to 100 Hz. I tried 10 msec interval but it failed to complete all images.

By the way, I saw that the NumImages is an int32, ADNumImages asynInt32 r/w Number of images to acquire in one acquisition sequence NIMAGES $(P)$(R)NumImages $(P)$(R)NumImages Which should allow more than 65535 number of images. Is this limit 65535 caused by MEDM?

On Thu, Aug 27, 2020 at 5:33 PM Mark Rivers notifications@github.com wrote:

If you use Multiple mode and 1000 images does it collect exactly 1000 images. Try 8, 12, 16 bit, fixed rate and free running.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/areaDetector/ADSpinnaker/issues/4#issuecomment-682258967, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSYGNGI47IBTSUWIJT4Q6TSC33MHANCNFSM4PJ2ADJQ .

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

[MarkRivers]

Which should allow more than 65535 number of images. Is this limit 65535 caused by MEDM?

No, this must be a limit enforced by the camera firmware. The GenICam specification (https://www.emva.org/wp-content/uploads/GenICam_SFNC_2_3.pdf page 166) says that AcquisitionFrameCount is an IInteger, which is actually a 64-bit integer. But vendors are free to limit the values to a specific range. In ADSpinnaker if I write 65536 to NumImages (which is the AcquisitionFrameCount feature) the value then read back from the camera is 65535, indicating it won't allow more than that.

[LeeYangLBLBCS]

you are correct. I tried to enter into spinview. It won't take anything higher than 65535 either. Once in a while we do TIMBIR scan (Time Interlaced Model-Based Iterative Reconstruction), which can get close to this number, though we are limited by our current motor controller first (~13k position array), at this time.

On Fri, Aug 28, 2020 at 10:21 AM Mark Rivers notifications@github.com wrote:

Which should allow more than 65535 number of images. Is this limit 65535 caused by MEDM?

No, this must be a limit enforced by the camera firmware. The GenICam specification ( https://www.emva.org/wp-content/uploads/GenICam_SFNC_2_3.pdf page 166) says that AcquisitionFrameCount is an IInteger, which is actually a 64-bit integer. But vendors are free to limit the values to a specific range. In ADSpinnaker if I write 65536 to NumImages (which is the AcquisitionFrameCount feature) the value then read back from the camera is 65535, indicating it won't allow more than that.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/areaDetector/ADSpinnaker/issues/4#issuecomment-682958960, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSYGNBYQV5EFWIMHQR6DLDSC7RRPANCNFSM4PJ2ADJQ .

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

[MarkRivers]

Interestingly I cannot reproduce your result of failing to stop. Here are some results I obtain when FrameRateEnable=No and AcquireTime=0.001.

Pixel format	ConvertPixelFormat	ADC bits	FLIR specified frame rate	Actual frame rate	# Images collected	Can stop?
Mono8	None	8	162	162	991-993	Yes
Mono12Packed	Mono16	8	122	120	978-983	Yes
Mono16	None	8	81	79	965-966	Yes
Mono8	None	10	144	144	996-997	Yes
Mono12Packed	Mono16	10	130	130	983-984	Yes
Mono16	None	10	85	84	978-979	Yes
Mono8	None	12	89	89	1000	Yes
Mono12Packed	Mono16	12	89	89	1000	Yes
Mono16	None	12	89	89	989-1000	Yes

In the Mono12Packed mode there is up to 2 second delay between pressing Stop and it actually stopping. I don't see the problem you do of not being able to stop it all. I think the problem is that the time to convert the pixel format from Mono12Packed to Mono16. Currently the mutex is locked during this operation, which is probably preventing the Stop command from getting enough time to execute. I will try changing that and see if it fixes your problem.

[MarkRivers]

Once in a while we do TIMBIR scan (Time Interlaced Model-Based Iterative Reconstruction), which can get close to this number, though we are limited by our current motor controller first (~13k position array), at this time.

I think you can work around that problem by using Continuous mode rather than Multiple. Once you have collected the correct number of images your software just stops the camera.

[MarkRivers]

We always have to calibrate the highest frame rate achievable right before the tomography run.

You should not need to do that. I also trigger from motor pulses, and I adjust the motor speed to generate the next trigger as close as possible to previous image readout being completed. I don't do that empirically for each run, I have a table for each camera, PixelFormat, ADC bits, etc. I recently converted my software from IDL to Python, and it is here:

Code: https://github.com/tomography/tomoscan

That code for the readout time is here: https://github.com/tomography/tomoscan/blob/d997a247407640b96d7ce9d124ab432f274b9254/tomoscan/tomoscan.py#L748-L790

Documentation: https://tomoscan.readthedocs.io/en/latest/

[LeeYangLBLBCS]

Is the network traffic stable enough to save files at constant speed? I tried to calibrate it once (differently from yours) but it only lasted a few months. I don't know what's causing the changes. It's too time consuming to calibrate again.

On Fri, Aug 28, 2020 at 11:51 AM Mark Rivers notifications@github.com wrote:

< We always have to calibrate the highest frame rate achievable right before the tomography run.

You should not need to do that. I also trigger from motor pulses, and I adjust the motor speed to generate the next trigger as close as possible to previous image readout being completed. I don't do that empirically for each run, I have a table for each camera, PixelFormat, ADC bits, etc. I recently converted my software from IDL to Python, and it is here:

Code: https://github.com/tomography/tomoscan

That code for the readout time is here:

https://github.com/tomography/tomoscan/blob/d997a247407640b96d7ce9d124ab432f274b9254/tomoscan/tomoscan.py#L748-L790

Documentation: https://tomoscan.readthedocs.io/en/latest/

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/areaDetector/ADSpinnaker/issues/4#issuecomment-683074742, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSYGNBTXTBP4B5LTSAOKGDSC74BJANCNFSM4PJ2ADJQ .

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

[MarkRivers]

There is no need to worry about the network traffic for file saving with areaDetector. If the file saving cannot keep up with the detector then you just set the queue size for the file plugin to be large, up to the number of images you are collecting. That way the detector runs at its maximum speed and the files to be written will be queued, but they will eventually all be written. That way the detector sets the acquisition speed (time resolution of measurement) while the disk/network speed sets the time between datasets.

[MarkRivers]

@LeeYangLBLBCS I have pushed a change that may fix the problem that you cannot stop acquisition with Mono12Packed. I am not sure it is safe, I did observe it to lock up once. It seems to reduce the delay for me between pressing Stop and when it actually stops.

My suspicion is that your system is a little less powerful and the mutex is never unlocked long enough to allow the Stop command to be processed. Please try the master branch.

[LeeYangLBLBCS]

thanks. I'll try that. Wouldn't that mean all future users will need to upgrade to ADSpinnaker 2.0 if it's in master branch?

On Fri, Aug 28, 2020 at 3:38 PM Mark Rivers notifications@github.com wrote:

@LeeYangLBLBCS https://github.com/LeeYangLBLBCS I have pushed a change that may fix the problem that you cannot stop acquisition with Mono12Packed. I am not sure it is safe, I did observe it to lock up once. It seems to reduce the delay for me between pressing Stop and when it actually stops.

My suspicion is that your system is a little less powerful and the mutex is never unlocked long enough to allow the Stop command to be processed. Please try the master branch.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/areaDetector/ADSpinnaker/issues/4#issuecomment-683175651, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSYGNBO5QQVNUGWYM2JDSLSDAWU7ANCNFSM4PJ2ADJQ .

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

[MarkRivers]

Sorry, I did not mean master branch, I meant the new version of the SDK_2.0 branch.

What version of Linux are you running?

[LeeYangLBLBCS]

I'm using Ubuntu 18.04.

On Fri, Aug 28, 2020 at 3:49 PM Mark Rivers notifications@github.com wrote:

Sorry, I did not mean master branch, I meant the new version of the SDK_2.0 branch.

What version of Linux are you running?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/areaDetector/ADSpinnaker/issues/4#issuecomment-683178745, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSYGNCM4NKNXCAGK4HV5QTSDAYAFANCNFSM4PJ2ADJQ .

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

[MarkRivers]

I am running that version as well.

[LeeYangLBLBCS]

I ran various tests with the bug fix on SDK2.0. It doesn't work too well. Most of the runs are slower or can't succeed than without the fix. In particular, the pixel format mono16 can't finish even down to 20 fps. I attached screen shot showing the result in "LibreOffice.Calc" sheet. I also attached the file in zip if it goes through.

spinnaker sdk 2.0 bug fix test result.zip

[MarkRivers]

Your results in terms of not collecting all of the images is similar to what I am seeing. But I don't think that is due to the small change I made in that fix. This is the fix I made:

diff --git a/spinnakerApp/src/ADSpinnaker.cpp b/spinnakerApp/src/ADSpinnaker.cpp
index 4331fa2..abaa1a6 100755
--- a/spinnakerApp/src/ADSpinnaker.cpp
+++ b/spinnakerApp/src/ADSpinnaker.cpp
@@ -477,6 +477,7 @@ asynStatus ADSpinnaker::grabImage()
             }

             pixelFormat = pImage->GetPixelFormat();
+            unlock();
             try {
                 pImage  = pImage->Convert(convertedFormat);
                 imageConverted = true;
@@ -486,6 +487,7 @@ asynStatus ADSpinnaker::grabImage()
                      "%s::%s pixel format conversion exception %s\n",
                  driverName, functionName, e.what());
             }
+            lock();
         }

         pixelFormat = pImage->GetPixelFormat();

So I just added calls to unlock() and lock(). For me that change did not affect the number of frames collected successfully.

But it seems that for you that minor change did fix the problem of mono12Packed not stopping when you pressed Stop. Is that true?

Please try commenting out those lines and run your tests again. I suspect you will still have dropped frames. If you set the Status rate in the upper right of ADSpinnaker.adl you should see the number of FailedPacket increase each time there are dropped frames. I see that, do you?

[LeeYangLBLBCS]

I don't see drop frame count on the top right corner. Maybe I don't know where to look? The screen I attached is showing no dropped frames in "status" area, while the acquiring is stuck at frame 4784 out of 5000 frames.

[MarkRivers]

I repeated your measurement of 8-bit, Mono16, 20 FPS. You measured 4632/5000 frames, which is a lot of missing frames. I measured 4996 and 5000.

[MarkRivers]

The screen I attached is showing no dropped frames in "status" area,

You need to change the StatusRate there from Passive to "1 second" or some other non-zero value. The values that I see increase when frames are dropped is the one labeled "Failed packet".

[LeeYangLBLBCS]

OK, I see failed packets number increasing. It's something like 2232 packets dropped for each frame. And yes, the acquisition stop is almost fixed, though I've still seen stop failed once or twice in the entire test run. I'll remove the "unlock/lock" change and retest again.

On Mon, Aug 31, 2020 at 2:06 PM Mark Rivers notifications@github.com wrote:

The screen I attached is showing no dropped frames in "status" area,

You need to change the StatusRate there from Passive to "1 second" or some other non-zero value. The values that I see increase when frames are dropped is the one labeled "Failed packet".

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/areaDetector/ADSpinnaker/issues/4#issuecomment-684040095, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSYGNCQK3IKEGHSL3DT5ZTSDQGFNANCNFSM4PJ2ADJQ .

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

[LeeYangLBLBCS]

One other thing I noticed is that IOC terminal seems to show a different error when frames are dropped (used to be a message with number 9): 2020/08/31 14:12:26.761 ADSpinnaker::grabImage error GetImageStatus returned 3 2020/08/31 14:12:27.361 ADSpinnaker::grabImage error GetImageStatus returned 3 2020/08/31 14:12:27.411 ADSpinnaker::grabImage error GetImageStatus returned 3 2020/08/31 14:12:27.911 ADSpinnaker::grabImage error GetImageStatus returned 3 2020/08/31 14:12:28.261 ADSpinnaker::grabImage error GetImageStatus returned 3 2020/08/31 14:12:28.961 ADSpinnaker::grabImage error GetImageStatus returned 3 2020/08/31 14:12:30.711 ADSpinnaker::grabImage error GetImageStatus returned 3 2020/08/31 14:12:32.161 ADSpinnaker::grabImage error GetImageStatus returned 3 2020/08/31 14:12:33.161 ADSpinnaker::grabImage error GetImageStatus returned 3 2020/08/31 14:12:34.711 ADSpinnaker::grabImage error GetImageStatus returned 3 2020/08/31 14:12:34.761 ADSpinnaker::grabImage error GetImageStatus returned 3 2020/08/31 14:12:34.811 ADSpinnaker::grabImage error GetImageStatus returned 3

On Mon, Aug 31, 2020 at 2:19 PM Lee L. Yang llyang@lbl.gov wrote:

OK, I see failed packets number increasing. It's something like 2232 packets dropped for each frame. And yes, the acquisition stop is almost fixed, though I've still seen stop failed once or twice in the entire test run. I'll remove the "unlock/lock" change and retest again.

On Mon, Aug 31, 2020 at 2:06 PM Mark Rivers notifications@github.com wrote:

The screen I attached is showing no dropped frames in "status" area,

You need to change the StatusRate there from Passive to "1 second" or some other non-zero value. The values that I see increase when frames are dropped is the one labeled "Failed packet".

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/areaDetector/ADSpinnaker/issues/4#issuecomment-684040095, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSYGNCQK3IKEGHSL3DT5ZTSDQGFNANCNFSM4PJ2ADJQ .

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

[MarkRivers]

2020/08/31 14:12:27.361 ADSpinnaker::grabImage error GetImageStatus returned 3

Yes, I noticed that too.

It looks like 3 is probably more accurate than 9:

        IMAGE_MISSING_PACKETS = 3,  /**< Image has missing packets */
...
        IMAGE_DATA_INCOMPLETE = 9,                  /**< Image data is incomplete. */

[LeeYangLBLBCS]

I recompiled the code with two lines removed, it worked similarly. I also retested the original version I downloaded without this bug fix. The result is also similar. There are some difference between these tests in terms of the number of lost frames, but I think it's just statistical variations from one run to another. Though it's clear from all the tests that the only mono12Packed pixel format could complete all 5000 images reliably. mono16 and mono8 almost always lose frames. It's also likely even some cases with mono12Packed could result in frame loss with NumImages=65535.

Is this the best we can do with this camera? Did other users also have this problem? Is my computer brand (Supermicro) inferior to others? (which I converted to linux from Windows 10 machine).

On Mon, Aug 31, 2020 at 4:19 PM Mark Rivers notifications@github.com wrote:

2020/08/31 14:12:27.361 ADSpinnaker::grabImage error GetImageStatus returned 3

Yes, I noticed that too.

It looks like 3 is probably more accurate than 9:

    IMAGE_MISSING_PACKETS = 3,  /**< Image has missing packets */

... IMAGE_DATA_INCOMPLETE = 9, /*< Image data is incomplete. /

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/areaDetector/ADSpinnaker/issues/4#issuecomment-684092381, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSYGNHEK5GI74A44K3EIYLSDQVYVANCNFSM4PJ2ADJQ .

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

[MarkRivers]

Is this the best we can do with this camera? Did other users also have this problem?

I have not started to use this camera in production, but Francesco at APS 2-BM has. I am sure he would have told me if he had this problem.

We need to determine 2 things:

• Does the problem occur with FLIR software, or only with EPICS? Can SpinView be run on Linux? I have not tried it. If not, FLIR does provide small C++ test programs. If we can generate the problem with their programs we can send the issue to FLIR

• Does the problem occur with the older 1.20.0.14 SDK, i.e. the master branch? If not then we can contact FLIR to ask why the problem appears with the 2.0.0 SDK.

[LeeYangLBLBCS]

I will revert back to V1.20.0.14 SDK and retest these exact cases. I don't remember I've seen frequent loss of frames, especially for mono16.

On Tue, Sep 1, 2020 at 8:33 AM Mark Rivers notifications@github.com wrote:

Is this the best we can do with this camera? Did other users also have this problem?

I have not started to use this camera in production, but Francesco at APS 2-BM has. I am sure he would have told me if he had this problem.

We need to determine 2 things:

-

Does the problem occur with FLIR software, or only with EPICS? Can SpinView be run on Linux? I have not tried it. If not, FLIR does provide small C++ test programs. If we can generate the problem with their programs we can send the issue to FLIR

Does the problem occur with the older 1.20.0.14 SDK, i.e. the master branch? If not then we can contact FLIR to ask why the problem appears with the 2.0.0 SDK.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/areaDetector/ADSpinnaker/issues/4#issuecomment-684941848, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSYGNDWOUBKZTRNDO4CX5TSDUH3LANCNFSM4PJ2ADJQ .

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

[LeeYangLBLBCS]

I tested it with SDK V1.20.0.14. The problem with ADC 10 bit has the same problem, only the number of missing frames is marginally smaller. But even at 20 fps the acquisition can't complete. Similarly, mono12Packed and momo8 worked better in both SDK versions. I'm able to run spinview 2.0 but not 1.20. I'll see if I can recreate the same problem with 2.0.

On Tue, Sep 1, 2020 at 8:38 AM Lee L. Yang llyang@lbl.gov wrote:

I will revert back to V1.20.0.14 SDK and retest these exact cases. I don't remember I've seen frequent loss of frames, especially for mono16.

On Tue, Sep 1, 2020 at 8:33 AM Mark Rivers notifications@github.com wrote:

Is this the best we can do with this camera? Did other users also have this problem?

I have not started to use this camera in production, but Francesco at APS 2-BM has. I am sure he would have told me if he had this problem.

We need to determine 2 things:

-

Does the problem occur with FLIR software, or only with EPICS? Can SpinView be run on Linux? I have not tried it. If not, FLIR does provide small C++ test programs. If we can generate the problem with their programs we can send the issue to FLIR

Does the problem occur with the older 1.20.0.14 SDK, i.e. the master branch? If not then we can contact FLIR to ask why the problem appears with the 2.0.0 SDK.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/areaDetector/ADSpinnaker/issues/4#issuecomment-684941848, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSYGNDWOUBKZTRNDO4CX5TSDUH3LANCNFSM4PJ2ADJQ .

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

-- Lee Yang Lawrence Berkeley National Lab 1 Cyclotron Road, MS 7H210 Berkeley, California 97320 office:(510)486-7320 fax:(510) 486-4633

[MarkRivers]

Can you provide some quantitative information with SDK 1.20.0.14. Like your previous table but don't need all the entries if that takes too long. I particularly want the numbers for the 20 fps, 70 fps with 8-bit/Mono8 and 10-bit/Mono12Packed.

[LeeYangLBLBCS]

I don't see drop frame count on the top right corner. Maybe I don't know where to look? The screen I attached is showing no dropped frames in "status" area, while the acquiring is stuck at frame 4784 out of 5000 frames.

[MarkRivers]

You need to change the "Status rate" in that box from Passive to "1 second" or some other non-Passive value.

[LeeYangLBLBCS]

The status doesn't work even with non passive mode, except with the version you put in the additional bug fix (lock/unlock) in ADSpinnakerApp.cpp file.

I've uploaded a screen shot image with test results for V1.20. I also tried to include the spreed sheet (in zip format) that contains four sets of tests with similar parameters. The names of the sheets are:

SDK1.20: older SDK with mono12packed stop lock up SDK2.0_first version: first version I installed SDK2.0-bugfix: additional bug fix with lock/unlock

SDK2.0-bugfix-reverted: reverted the bug fix from above to check if it's the same.

spinnaker sdk 2.0 bug fix test result.zip

[LeeYangLBLBCS]

the screen shot maybe the wrong sheet. try again: spinnaker sdk 2.0 bug fix test result.zip

[MarkRivers]

The status doesn't work even with non passive mode, except with the version you put in the additional bug fix (lock/unlock) in ADSpinnakerApp.cpp file.

That indicates to me that your system is completely CPU-bound, so the status thread is never getting any time to run. Did you try setting a non-passive mode at 20 fps? That should not be CPU bound. If you see the following at 20 fps:

• Number of frames is < 5000
• Failed packets = 0

then something is wrong. In that case please take a screen shot of another terminal on the IOC computer running this command:

top -H

That will show the CPU time for each thread.