search

byte-physics / igortest

Igor Pro Universal Testing Framework
https://docs.byte-physics.de/igor-unit-testing-framework/
BSD 3-Clause "New" or "Revised" License
7 stars 2 forks source link

Feature: Check for free wave leaks in test cases #297

Closed Garados007 closed 1 year ago

Garados007 commented 2 years ago

It is possible to write igor code that won't release free waves after they are no longer used. This can conclude to big wave leaks and therefore useless memory consumption.

With this commit the ability to check for these free wave leaks is introduced. To enable this it is only required to set waveTrackingMode in RunTest to UTF_WAVE_TRACKING_FREE, UTF_WAVE_TRACKING_LOCAL or UTF_WAVE_TRACKING_ALL. This will track the number of created and released waves during a test case run. If there is a difference this will produce a CHECK error.

If the user wants to opt-out a single test case from this wave tracking the test case as to be tagged with UTF_NO_WAVE_TRACKING.

Close #200.

Dependency

Garados007 commented 2 years ago

I have resolved the merge conflict.

t-b commented 2 years ago

@MichaelHuth I've assigned you instead due to workload issues on my side.

t-b commented 2 years ago

Before this is merged I would like to start a CI run with MIES to see if it is working as intended. I can think of data generators being a problem or user hooks.

Garados007 commented 2 years ago

This PR requires PR #306 to be merged first.

MichaelHuth commented 2 years ago

@t-b I have played a bit around already and alone in the MIES SF and Utils tests there are a lot of tests with leak issues.

Garados007 commented 2 years ago

I was forced to do a rebase to let the CI pipeline rerun. You need to approve this PR again.

t-b commented 2 years ago

Michael and me played around with this PR and MIES. And found some bugs in IP. I've reported them as #4280/#4281. Once these are fixed I'll resume testing.

MichaelHuth commented 2 years ago

@Garados007 Resulting testing with Thomas, I have moved the wavetrack evaluation to a function that is called after the user hook code. This is required as we need in tests the user hook to cleanup created waves to reach the same state as on test case entry.

t-b commented 2 years ago

@Garados007 Can we have a new PR with only the introduction of TestCaseFail? And also maybe reuse the new ReportError/ReportErrorAndAbort functions? And avoid code duplication with printFailinfo? And also use TestCaseFail in other cases (I've found at least AfterTestCase). Thx.

t-b commented 2 years ago

Note to self: Reported another issue to WM regarding correct tracking.

Garados007 commented 2 years ago

Can we have a new PR with only the introduction of TestCaseFail?

See PR #310. The commit is still included in this PR (for compatibility issues). I remove it after #310 is merged to main.

And also maybe reuse the new ReportError/ReportErrorAndAbort functions?

Done.

And avoid code duplication with printFailinfo? And also use TestCaseFail in other cases (I've found at least AfterTestCase).

Let's move this to PR #310.

Garados007 commented 2 years ago

I have included the new TestCaseFail method into this PR.

t-b commented 2 years ago

After some more playing with the MIES Basic.pxp tests, I found https://gitlab.byte-physics.de:4001/internal/json-xop/-/issues/125. This is very likely reponsible for the vast majority of SF wave leaks.

The JSON-XOP testsuite does not wave any wave leaks though.

t-b commented 2 years ago

MIES wave leaks:

Basic.pxp

  Test finished with 38 errors
    ▶ Local wave leak detected (leaked waves: 2) in "AsynTest#TASYNC_AddParam_Wave"
    ▶ Local wave leak detected (leaked waves: 2) in "AsynTest#TASYNC_AddParam_FreeWave"
    ▶ Local wave leak detected (leaked waves: 1) in "AsynTest#TASYNC_TestFetch"
    ▶ Local wave leak detected (leaked waves: 6) in "AsynTest#TASYNC_Execute_Valid"
    ▶ Free wave leak detected (leaked waves: -1) in "AsynTest#TASYNC_Execute_Valid"
    ▶ Local wave leak detected (leaked waves: 5) in "AsynTest#TASYNC_WorkerRealDF"
    ▶ Free wave leak detected (leaked waves: -1) in "AsynTest#TASYNC_WorkerRealDF"
    ▶ Local wave leak detected (leaked waves: 6) in "AsynTest#TASYNC_StopForAssert"
    ▶ Free wave leak detected (leaked waves: -1) in "AsynTest#TASYNC_StopForAssert"
    ▶ Local wave leak detected (leaked waves: 4) in "AsynTest#TASYNC_WorkerNoDF"
    ▶ Assertion "WARN(0)" failed in TASYNC_RunOrderless (UTF_AsynFrameworkTest.ipf, line 741)
    ▶ Local wave leak detected (leaked waves: 1200) in "AsynTest#TASYNC_RunOrderless"
    ▶ Free wave leak detected (leaked waves: -200) in "AsynTest#TASYNC_RunOrderless"
    ▶ Local wave leak detected (leaked waves: 1200) in "AsynTest#TASYNC_RunInOrder"
    ▶ Free wave leak detected (leaked waves: -200) in "AsynTest#TASYNC_RunInOrder"
    ▶ Local wave leak detected (leaked waves: 2400) in "AsynTest#TASYNC_InOrderDiffWL"
    ▶ Free wave leak detected (leaked waves: -400) in "AsynTest#TASYNC_InOrderDiffWL"
    ▶ Local wave leak detected (leaked waves: 48) in "AsynTest#TASYNC_RunErrorWorker"
    ▶ Local wave leak detected (leaked waves: 6) in "AsynTest#TASYNC_RunErrorReadOut"
    ▶ Free wave leak detected (leaked waves: -1) in "AsynTest#TASYNC_RunErrorReadOut"
    ▶ Assertion "WARN(0)" failed in TASYNC_OrderlessDirectStop (UTF_AsynFrameworkTest.ipf, line 972)
    ▶ Local wave leak detected (leaked waves: 1200) in "AsynTest#TASYNC_OrderlessDirectStop"
    ▶ Free wave leak detected (leaked waves: -200) in "AsynTest#TASYNC_OrderlessDirectStop"
    ▶ Local wave leak detected (leaked waves: 988) in "AsynTest#TASYNC_StopTimeOut"
    ▶ Free wave leak detected (leaked waves: -12) in "AsynTest#TASYNC_StopTimeOut"
    ▶ Local wave leak detected (leaked waves: 588) in "AsynTest#TASYNC_StopTimeOutForce"
    ▶ Local wave leak detected (leaked waves: 2400) in "AsynTest#TASYNC_IODiffWLDirectStop"
    ▶ Free wave leak detected (leaked waves: -400) in "AsynTest#TASYNC_IODiffWLDirectStop"
    ▶ Local wave leak detected (leaked waves: 1200) in "AsynTest#TASYNC_RunClassSingle"
    ▶ Free wave leak detected (leaked waves: -200) in "AsynTest#TASYNC_RunClassSingle"
    ▶ Local wave leak detected (leaked waves: 2400) in "AsynTest#TASYNC_RunClassDouble"
    ▶ Free wave leak detected (leaked waves: -400) in "AsynTest#TASYNC_RunClassDouble"
    ▶ Local wave leak detected (leaked waves: 4) in "AsynTest#TASYNC_RunClassMixedOrder"
    ▶ Local wave leak detected (leaked waves: 16) in "AsynTest#TASYNC_RunClassChangeOrder"
    ▶ Free wave leak detected (leaked waves: -2) in "AsynTest#TASYNC_RunClassChangeOrder"
    ▶ Local wave leak detected (leaked waves: 4) in "UTF_SweepFormula#TPWithModelCell"
    ▶ Local wave leak detected (leaked waves: 2) in "UtilsTest#GetListOfObjectsWorksWithFreeDF"
    ▶ Local wave leak detected (leaked waves: 1) in "DFED_FailsRegular3"

PAPlot.pxp: No leaks 🎉

HardwareTesting.pxp: 

  Test finished with 239 errors
    ▶ Assertion "CHECK_EQUAL_VAR(CHI_CheckInstallation(), 0)" failed in CheckInstallation (UTF_VeryBasicHardwareTests.ipf, line 7)
    ▶ Free wave leak detected (leaked waves: 929) in "TrackSweepCounts#MD0_RA0_I0_L0_BKG0:ITC"
    ▶ Free wave leak detected (leaked waves: 33) in "TrackSweepCounts#MD1_RA0_I0_L0_BKG1:ITC"
    ▶ Free wave leak detected (leaked waves: 2926) in "TrackSweepCounts#MD0_RA1_I0_L0_BKG0:ITC"
    ▶ Free wave leak detected (leaked waves: 95) in "TrackSweepCounts#MD1_RA1_I0_L0_BKG1:ITC"
    ▶ Free wave leak detected (leaked waves: 157) in "TrackSweepCounts#MD1_RA1_I1_L0_BKG1:ITC"
    ▶ Free wave leak detected (leaked waves: 4828) in "TrackSweepCounts#MD0_RA1_I1_L0_BKG0:ITC"
    ▶ Free wave leak detected (leaked waves: 185) in "TrackSweepCounts#MD1_RA1_I1_L1_BKG1:ITC"
    ▶ Free wave leak detected (leaked waves: 5846) in "TrackSweepCounts#MD0_RA1_I1_L1_BKG0:ITC"
    ▶ Free wave leak detected (leaked waves: 191) in "TrackSweepCounts#RepeatSets_1:ITC"
    ▶ Free wave leak detected (leaked waves: 309) in "TrackSweepCounts#RepeatSets_2:ITC"
    ▶ Free wave leak detected (leaked waves: 372) in "TrackSweepCounts#RepeatSets_3:ITC"
    ▶ Free wave leak detected (leaked waves: 373) in "TrackSweepCounts#RepeatSets_4:ITC"
    ▶ Free wave leak detected (leaked waves: 311) in "TrackSweepCounts#RepeatSets_5:ITC"
    ▶ Free wave leak detected (leaked waves: 217) in "TrackSweepCounts#RepeatSets_6:ITC"
    ▶ Free wave leak detected (leaked waves: 32) in "TrackSweepCounts#CheckIZeroClampMode:ITC"
    ▶ Free wave leak detected (leaked waves: 97) in "TrackSweepCounts#RepeatSets_7:ITC"
    ▶ Assertion "AllTests(t, str)" failed in RepeatSets_8_REENTRY➔AllTests (UTF_TrackSweepCounts.ipf, line 1432➔133)
    ▶ Assertion "AllTests(t, str)" failed in RepeatSets_8_REENTRY➔AllTests (UTF_TrackSweepCounts.ipf, line 1432➔133)
    ▶ Assertion "AllTests(t, str)" failed in RepeatSets_8_REENTRY➔AllTests (UTF_TrackSweepCounts.ipf, line 1432➔133)
    ▶ Assertion "AllTests(t, str)" failed in RepeatSets_8_REENTRY➔AllTests (UTF_TrackSweepCounts.ipf, line 1432➔133)
    ▶ Local wave leak detected (leaked waves: 808) in "TrackSweepCounts#RepeatSets_8:ITC"
    ▶ Free wave leak detected (leaked waves: 196) in "TrackSweepCounts#RepeatSets_8:ITC"
    ▶ Assertion "AllTests(t, str)" failed in RepeatSets_9_REENTRY➔AllTests (UTF_TrackSweepCounts.ipf, line 1479➔133)
    ▶ Assertion "AllTests(t, str)" failed in RepeatSets_9_REENTRY➔AllTests (UTF_TrackSweepCounts.ipf, line 1479➔133)
    ▶ Assertion "AllTests(t, str)" failed in RepeatSets_9_REENTRY➔AllTests (UTF_TrackSweepCounts.ipf, line 1479➔133)
    ▶ Assertion "AllTests(t, str)" failed in RepeatSets_9_REENTRY➔AllTests (UTF_TrackSweepCounts.ipf, line 1479➔133)
    ▶ Local wave leak detected (leaked waves: 784) in "TrackSweepCounts#RepeatSets_9:ITC"
    ▶ Free wave leak detected (leaked waves: 192) in "TrackSweepCounts#RepeatSets_9:ITC"
    ▶ Free wave leak detected (leaked waves: 154) in "BasicHardwareTests#CheckActiveSetCount:UnlockedIndexing:ITC"
    ▶ Free wave leak detected (leaked waves: 154) in "BasicHardwareTests#CheckActiveSetCount:LockedIndexing:ITC"
    ▶ Local wave leak detected (leaked waves: 371) in "BasicHardwareTests#Abort_ITI_TP:SingleDevice:ITC"
    ▶ Free wave leak detected (leaked waves: 142) in "BasicHardwareTests#Abort_ITI_TP:SingleDevice:ITC"
    ▶ Local wave leak detected (leaked waves: 413) in "BasicHardwareTests#Abort_ITI_TP:MultiDevice:ITC"
    ▶ Free wave leak detected (leaked waves: 167) in "BasicHardwareTests#Abort_ITI_TP:MultiDevice:ITC"
    ▶ Local wave leak detected (leaked waves: 364) in "BasicHardwareTests#Abort_ITI_TP_A_TP:SingleDevice:ITC"
    ▶ Free wave leak detected (leaked waves: 143) in "BasicHardwareTests#Abort_ITI_TP_A_TP:SingleDevice:ITC"
    ▶ Local wave leak detected (leaked waves: 434) in "BasicHardwareTests#Abort_ITI_TP_A_TP:MultiDevice:ITC"
    ▶ Free wave leak detected (leaked waves: 174) in "BasicHardwareTests#Abort_ITI_TP_A_TP:MultiDevice:ITC"
    ▶ Local wave leak detected (leaked waves: 326) in "BasicHardwareTests#AbortTP:SingleDevice:ITC"
    ▶ Free wave leak detected (leaked waves: 108) in "BasicHardwareTests#AbortTP:SingleDevice:ITC"
    ▶ Local wave leak detected (leaked waves: 427) in "BasicHardwareTests#AbortTP:MultiDevice:ITC"
    ▶ Free wave leak detected (leaked waves: 131) in "BasicHardwareTests#AbortTP:MultiDevice:ITC"
    ▶ Local wave leak detected (leaked waves: 696) in "BasicHardwareTests#StartDAQDuringTP:SingleDevice:ITC"
    ▶ Free wave leak detected (leaked waves: 223) in "BasicHardwareTests#StartDAQDuringTP:SingleDevice:ITC"
    ▶ Local wave leak detected (leaked waves: 840) in "BasicHardwareTests#StartDAQDuringTP:MultiDevice:ITC"
    ▶ Free wave leak detected (leaked waves: 280) in "BasicHardwareTests#StartDAQDuringTP:MultiDevice:ITC"
    ▶ Local wave leak detected (leaked waves: 7) in "BasicHardwareTests#Abort_ITI_PressAcq:SingleDevice:ITC"
    ▶ Free wave leak detected (leaked waves: 31) in "BasicHardwareTests#Abort_ITI_PressAcq:SingleDevice:ITC"
    ▶ Local wave leak detected (leaked waves: 7) in "BasicHardwareTests#Abort_ITI_PressAcq:MultiDevice:ITC"
    ▶ Free wave leak detected (leaked waves: 38) in "BasicHardwareTests#Abort_ITI_PressAcq:MultiDevice:ITC"
    ▶ Local wave leak detected (leaked waves: 371) in "BasicHardwareTests#Abort_ITI_TP_A_PressAcq:SingleDevice:ITC"
    ▶ Free wave leak detected (leaked waves: 144) in "BasicHardwareTests#Abort_ITI_TP_A_PressAcq:SingleDevice:ITC"
    ▶ Local wave leak detected (leaked waves: 434) in "BasicHardwareTests#Abort_ITI_TP_A_PressAcq:MultiDevice:ITC"
    ▶ Free wave leak detected (leaked waves: 172) in "BasicHardwareTests#Abort_ITI_TP_A_PressAcq:MultiDevice:ITC"
    ▶ Free wave leak detected (leaked waves: 31) in "BasicHardwareTests#ChangeToOtherDeviceDAQ:SingleDevice:ITC"
    ▶ Free wave leak detected (leaked waves: 22) in "BasicHardwareTests#ChangeToOtherDeviceDAQ:MultiDevice:ITC"
    ▶ Local wave leak detected (leaked waves: 896) in "BasicHardwareTests#ChangeStimSetDuringDAQ:ITC"
    ▶ Free wave leak detected (leaked waves: 215) in "BasicHardwareTests#ChangeStimSetDuringDAQ:ITC"
    ▶ Free wave leak detected (leaked waves: 31) in "BasicHardwareTests#UnassociatedChannelsAndTTLs:ITC"
    ▶ Free wave leak detected (leaked waves: 32) in "BasicHardwareTests#CheckSamplingInterval1:ITC"
    ▶ Free wave leak detected (leaked waves: 31) in "BasicHardwareTests#CheckSamplingInterval2:ITC"
    ▶ Free wave leak detected (leaked waves: 32) in "BasicHardwareTests#CheckSamplingInterval3:ITC"
    ▶ Free wave leak detected (leaked waves: 96) in "BasicHardwareTests#ChangeCMDuringSweep:ITC"
    ▶ Free wave leak detected (leaked waves: 90) in "BasicHardwareTests#ChangeCMDuringSweepWMS:ITC"
    ▶ Free wave leak detected (leaked waves: 31) in "BasicHardwareTests#ChangeCMDuringSweepNoRA:ITC"
    ▶ Free wave leak detected (leaked waves: 97) in "BasicHardwareTests#ChangeCMDuringITI:ITC"
    ▶ Local wave leak detected (leaked waves: 1246) in "BasicHardwareTests#ChangeCMDuringITIWithTP:ITC"
    ▶ Free wave leak detected (leaked waves: 289) in "BasicHardwareTests#ChangeCMDuringITIWithTP:ITC"
    ▶ Free wave leak detected (leaked waves: 10) in "BasicHardwareTests#AutoPipetteOffsetIgnoresApplyOnModeSwitch:ITC"
    ▶ Free wave leak detected (leaked waves: 12) in "BasicHardwareTests#HasNaNAsDefaultWhenAborted:ITC"
    ▶ Free wave leak detected (leaked waves: 36) in "BasicHardwareTests#UnassocChannelsDuplicatedEntry:ITC"
    ▶ Local wave leak detected (leaked waves: 540) in "BasicHardwareTests#RestoreDAEphysPanel:ITC"
    ▶ Free wave leak detected (leaked waves: 108) in "BasicHardwareTests#RestoreDAEphysPanel:ITC"
    ▶ Free wave leak detected (leaked waves: 34) in "BasicHardwareTests#LabnotebookEntriesCanBeQueried:ITC"
    ▶ Free wave leak detected (leaked waves: 31) in "BasicHardwareTests#DataBrowserCreatesBackupsByDefault:ITC"
    ▶ Free wave leak detected (leaked waves: 27) in "BasicHardwareTests#IncrementalLabnotebookCacheUpdate:ITC"
    ▶ Free wave leak detected (leaked waves: 185) in "BasicHardwareTests#TestSweepRollback:ITC"
    ▶ Free wave leak detected (leaked waves: 190) in "BasicHardwareTests#TestAcquiringNewDataOnOldData:ITC"
    ▶ Free wave leak detected (leaked waves: 31) in "BasicHardwareTests#AsyncAcquisitionLBN:ITC"
    ▶ Free wave leak detected (leaked waves: 1) in "BasicHardwareTests#CheckSettingsFails:ITC"
    ▶ Local wave leak detected (leaked waves: 266) in "BasicHardwareTests#CheckAcquisitionStates:SingleDevice:ITC"
    ▶ Free wave leak detected (leaked waves: 130) in "BasicHardwareTests#CheckAcquisitionStates:SingleDevice:ITC"
    ▶ Local wave leak detected (leaked waves: 322) in "BasicHardwareTests#CheckAcquisitionStates:MultiDevice:ITC"
    ▶ Free wave leak detected (leaked waves: 162) in "BasicHardwareTests#CheckAcquisitionStates:MultiDevice:ITC"
    ▶ Free wave leak detected (leaked waves: 6) in "BasicHardwareTests#ConfigureFails:ITC"
    ▶ Local wave leak detected (leaked waves: 14) in "BasicHardwareTests#StopDAQDueToUnlocking:ITC"
    ▶ Free wave leak detected (leaked waves: 41) in "BasicHardwareTests#StopDAQDueToUnlocking:ITC"
    ▶ Local wave leak detected (leaked waves: -10) in "BasicHardwareTests#StopDAQDueToUncompiled:ITC"
    ▶ Free wave leak detected (leaked waves: 44) in "BasicHardwareTests#StopDAQDueToUncompiled:ITC"
    ▶ Local wave leak detected (leaked waves: 2) in "BasicHardwareTests#ExportStimsetsAndRoundtripThem:v1"
    ▶ Local wave leak detected (leaked waves: 4) in "BasicHardwareTests#ExportStimsetsAndRoundtripThem:v2"
    ▶ Assertion "CHECK_PROPER_STR(stimsets)" failed in ExportIntoNWBSweepBySweep_REENTRY (UTF_BasicHardwareTests.ipf, line 1911)
    ▶ Assertion "CHECK_PROPER_STR(acquisition)" failed in ExportIntoNWBSweepBySweep_REENTRY (UTF_BasicHardwareTests.ipf, line 1914)
    ▶ Assertion "CHECK_PROPER_STR(stimulus)" failed in ExportIntoNWBSweepBySweep_REENTRY (UTF_BasicHardwareTests.ipf, line 1917)
    ▶ Assertion "CHECK_EQUAL_VAR(acqState, AS_INACTIVE)" failed in TEST_CASE_END_OVERRIDE (UTF_HardwareHelperFunctions.ipf, line 144)
    ▶ Local wave leak detected (leaked waves: 5) in "BasicHardwareTests#ExportIntoNWBSweepBySweep:ITC"
    ▶ Free wave leak detected (leaked waves: 2) in "BasicHardwareTests#ExportIntoNWBSweepBySweep:ITC"
    ▶ Local wave leak detected (leaked waves: 32) in "BasicHardwareTests#ExportOnlyCommentsIntoNWB:ITC"
    ▶ Free wave leak detected (leaked waves: 143) in "BasicHardwareTests#CheckPulseInfoGathering:ITC"
    ▶ Free wave leak detected (leaked waves: 21) in "BasicHardwareTests#RepeatedAcquisitionWithOneSweep:SingleDevice:ITC"
    ▶ Free wave leak detected (leaked waves: 32) in "BasicHardwareTests#RepeatedAcquisitionWithOneSweep:MultiDevice:ITC"
    ▶ Free wave leak detected (leaked waves: 31) in "BasicHardwareTests#EnableIndexingInPostDAQ:ITC"
    ▶ Free wave leak detected (leaked waves: 192) in "BasicHardwareTests#ScaleZeroWithCycling:ITC"
    ▶ Free wave leak detected (leaked waves: 33) in "BasicHardwareTests#AcquireWithoutAmplifier:ITC"
    ▶ Free wave leak detected (leaked waves: 45) in "HandlesFIFOTimeoutProperly:ITC"
    ▶ Free wave leak detected (leaked waves: 73) in "HandlesStuckFIFOProperly:ITC"
    ▶ Free wave leak detected (leaked waves: 31) in "SweepSkipping#SkipAhead:ITC"
    ▶ Free wave leak detected (leaked waves: 122) in "SweepSkipping#SweepSkipping:ITC"
    ▶ Free wave leak detected (leaked waves: 119) in "SweepSkipping#SweepSkippingAdvanced:ITC"
    ▶ Local wave leak detected (leaked waves: 556) in "SweepSkipping#SkipSweepsDuringITI:SingleDevice:ITC"
    ▶ Free wave leak detected (leaked waves: 105) in "SweepSkipping#SkipSweepsDuringITI:SingleDevice:ITC"
    ▶ Local wave leak detected (leaked waves: 616) in "SweepSkipping#SkipSweepsDuringITI:MultiDevice:ITC"
    ▶ Free wave leak detected (leaked waves: 126) in "SweepSkipping#SkipSweepsDuringITI:MultiDevice:ITC"
    ▶ Local wave leak detected (leaked waves: 980) in "SweepSkipping#SkipSweepsBackDuringITI:ITC"
    ▶ Free wave leak detected (leaked waves: 425) in "SweepSkipping#SkipSweepsBackDuringITI:ITC"
    ▶ Free wave leak detected (leaked waves: 32) in "TestPulseAndTPDuringDAQ#CheckCalculatedTPEntries:ITC"
    ▶ Local wave leak detected (leaked waves: 105) in "TestPulseAndTPDuringDAQ#CheckTPBaseline:BL_25:ITC"
    ▶ Free wave leak detected (leaked waves: 39) in "TestPulseAndTPDuringDAQ#CheckTPBaseline:BL_25:ITC"
    ▶ Local wave leak detected (leaked waves: 105) in "TestPulseAndTPDuringDAQ#CheckTPBaseline:BL_35:ITC"
    ▶ Free wave leak detected (leaked waves: 39) in "TestPulseAndTPDuringDAQ#CheckTPBaseline:BL_35:ITC"
    ▶ Local wave leak detected (leaked waves: 70) in "TestPulseAndTPDuringDAQ#CheckTPBaseline:BL_45:ITC"
    ▶ Free wave leak detected (leaked waves: 29) in "TestPulseAndTPDuringDAQ#CheckTPBaseline:BL_45:ITC"
    ▶ Local wave leak detected (leaked waves: 490) in "TestPulseAndTPDuringDAQ#CheckTPEntriesFromLBN:ITC"
    ▶ Free wave leak detected (leaked waves: 181) in "TestPulseAndTPDuringDAQ#CheckTPEntriesFromLBN:ITC"
    ▶ Local wave leak detected (leaked waves: 1638) in "TestPulseAndTPDuringDAQ#TPCachingWorks:ITC"
    ▶ Free wave leak detected (leaked waves: 340) in "TestPulseAndTPDuringDAQ#TPCachingWorks:ITC"
    ▶ Local wave leak detected (leaked waves: 980) in "TestPulseAndTPDuringDAQ#CheckTPStorage1:ITC"
    ▶ Free wave leak detected (leaked waves: 149) in "TestPulseAndTPDuringDAQ#CheckTPStorage1:ITC"
    ▶ Local wave leak detected (leaked waves: 1064) in "TestPulseAndTPDuringDAQ#CheckTPStorage2:ITC"
    ▶ Free wave leak detected (leaked waves: 161) in "TestPulseAndTPDuringDAQ#CheckTPStorage2:ITC"
    ▶ Local wave leak detected (leaked waves: 1050) in "TestPulseAndTPDuringDAQ#CheckTPStorage3:ITC"
    ▶ Free wave leak detected (leaked waves: 159) in "TestPulseAndTPDuringDAQ#CheckTPStorage3:ITC"
    ▶ Local wave leak detected (leaked waves: 210) in "TestPulseAndTPDuringDAQ#TPDuringDAQOnlyTP:ITC"
    ▶ Free wave leak detected (leaked waves: 49) in "TestPulseAndTPDuringDAQ#TPDuringDAQOnlyTP:ITC"
    ▶ Local wave leak detected (leaked waves: 630) in "TestPulseAndTPDuringDAQ#TPDuringDAQOnlyTPAndIndexing:UnlockedIndexing:ITC"
    ▶ Free wave leak detected (leaked waves: 153) in "TestPulseAndTPDuringDAQ#TPDuringDAQOnlyTPAndIndexing:UnlockedIndexing:ITC"
    ▶ Local wave leak detected (leaked waves: 630) in "TestPulseAndTPDuringDAQ#TPDuringDAQOnlyTPAndIndexing:LockedIndexing:ITC"
    ▶ Free wave leak detected (leaked waves: 145) in "TestPulseAndTPDuringDAQ#TPDuringDAQOnlyTPAndIndexing:LockedIndexing:ITC"
    ▶ Local wave leak detected (leaked waves: 210) in "TestPulseAndTPDuringDAQ#TPDuringDAQTPAndAssoc:ITC"
    ▶ Free wave leak detected (leaked waves: 49) in "TestPulseAndTPDuringDAQ#TPDuringDAQTPAndAssoc:ITC"
    ▶ Local wave leak detected (leaked waves: 217) in "TestPulseAndTPDuringDAQ#TPDuringDAQ:ITC"
    ▶ Free wave leak detected (leaked waves: 54) in "TestPulseAndTPDuringDAQ#TPDuringDAQ:ITC"
    ▶ Local wave leak detected (leaked waves: 196) in "TestPulseAndTPDuringDAQ#TPDuringDAQWithoodDAQ:ITC"
    ▶ Free wave leak detected (leaked waves: 38) in "TestPulseAndTPDuringDAQ#TPDuringDAQWithoodDAQ:ITC"
    ▶ Local wave leak detected (leaked waves: 420) in "TestPulseAndTPDuringDAQ#TPDuringDAQOnlyWithoodDAQ:ITC"
    ▶ Free wave leak detected (leaked waves: 53) in "TestPulseAndTPDuringDAQ#TPDuringDAQOnlyWithoodDAQ:ITC"
    ▶ Local wave leak detected (leaked waves: 742) in "TestPulseAndTPDuringDAQ#TPDuringDAQTPStoreCheck:ITC"
    ▶ Free wave leak detected (leaked waves: 64) in "TestPulseAndTPDuringDAQ#TPDuringDAQTPStoreCheck:ITC"
    ▶ Local wave leak detected (leaked waves: 826) in "TestPulseAndTPDuringDAQ#CheckThatTPsCanBeFound:ITC"
    ▶ Free wave leak detected (leaked waves: 127) in "TestPulseAndTPDuringDAQ#CheckThatTPsCanBeFound:ITC"
    ▶ Local wave leak detected (leaked waves: 357) in "TestPulseAndTPDuringDAQ#TPDuringDAQWithTTL:ITC"
    ▶ Free wave leak detected (leaked waves: 44) in "TestPulseAndTPDuringDAQ#TPDuringDAQWithTTL:ITC"
    ▶ Local wave leak detected (leaked waves: 812) in "TestPulseAndTPDuringDAQ#RunPowerSpectrum:ITC"
    ▶ Free wave leak detected (leaked waves: 125) in "TestPulseAndTPDuringDAQ#RunPowerSpectrum:ITC"
    ▶ Local wave leak detected (leaked waves: 588) in "TestPulseAndTPDuringDAQ#TestPulseCachingWorks:ITC"
    ▶ Free wave leak detected (leaked waves: 196) in "TestPulseAndTPDuringDAQ#TestPulseCachingWorks:ITC"
    ▶ Local wave leak detected (leaked waves: 816) in "TestPulseAndTPDuringDAQ#ExportIntoNWB:ITC"
    ▶ Free wave leak detected (leaked waves: 127) in "TestPulseAndTPDuringDAQ#ExportIntoNWB:ITC"
    ▶ Local wave leak detected (leaked waves: 270) in "CheckIfConfigurationRestoresMCCFilterGain:ITC"
    ▶ Free wave leak detected (leaked waves: 9) in "CheckIfConfigurationRestoresMCCFilterGain:ITC"
    ▶ Free wave leak detected (leaked waves: 134) in "DashboardTests#DAB_Indexing:ITC"
    ▶ Free wave leak detected (leaked waves: 28) in "DashboardTests#DAB_Skipping:ITC"
    ▶ Free wave leak detected (leaked waves: 24) in "Epochs#EP_EpochTest1:ITC"
    ▶ Free wave leak detected (leaked waves: 23) in "Epochs#EP_EpochTest2:ITC"
    ▶ Free wave leak detected (leaked waves: 28) in "Epochs#EP_EpochTest3:ITC"
    ▶ Free wave leak detected (leaked waves: 43) in "Epochs#EP_EpochTest4:ITC"
    ▶ Free wave leak detected (leaked waves: 38) in "Epochs#EP_EpochTest5:ITC"
    ▶ Free wave leak detected (leaked waves: 36) in "Epochs#EP_EpochTest6:ITC"
    ▶ Free wave leak detected (leaked waves: 14) in "Epochs#EP_EpochTest7:ITC"
    ▶ Free wave leak detected (leaked waves: 18) in "Epochs#EP_EpochTest8:ITC"
    ▶ Free wave leak detected (leaked waves: 15) in "Epochs#EP_EpochTest9:ITC"
    ▶ Local wave leak detected (leaked waves: 196) in "Epochs#EP_EpochTest10:ITC"
    ▶ Free wave leak detected (leaked waves: 42) in "Epochs#EP_EpochTest10:ITC"
    ▶ Free wave leak detected (leaked waves: 30) in "Epochs#EP_EpochTest11:ITC"
    ▶ Free wave leak detected (leaked waves: 13) in "Epochs#EP_EpochTest12:ITC"
    ▶ Free wave leak detected (leaked waves: 32) in "Epochs#EP_EpochTest13:ITC"
    ▶ Free wave leak detected (leaked waves: 90) in "Epochs#EP_TestUserEpochs:ITC"
    ▶ Free wave leak detected (leaked waves: 73) in "Epochs#EP_EpochTest14:ITC"
    ▶ Local wave leak detected (leaked waves: 136) in "SweepFormulaHardware#SF_TPTest:ITC"
    ▶ Free wave leak detected (leaked waves: 67) in "SweepFormulaHardware#SF_TPTest:ITC"
    ▶ Free wave leak detected (leaked waves: 67) in "SweepFormulaHardware#SF_ButtonTest:ITC"
    ▶ Free wave leak detected (leaked waves: 89) in "SweepFormulaHardware#TestSweepFormulaCodeResults:ITC"
    ▶ Local wave leak detected (leaked waves: 1052) in "SweepFormulaHardware#SF_InsertedTPVersusTP:ITC"
    ▶ Free wave leak detected (leaked waves: 218) in "SweepFormulaHardware#SF_InsertedTPVersusTP:ITC"
    ▶ Free wave leak detected (leaked waves: 24) in "AnalysisFunctionTesting#AnalysisParamsMustHaveSameOptionality"
    ▶ Free wave leak detected (leaked waves: 24) in "AnalysisFunctionTesting#GenerateAnalysisFunctionTable"
    ▶ Free wave leak detected (leaked waves: 14) in "AnalysisFunctionTesting#AFT3:ITC"
    ▶ Free wave leak detected (leaked waves: 261) in "AnalysisFunctionTesting#AFT4:ITC"
    ▶ Free wave leak detected (leaked waves: 20) in "AnalysisFunctionTesting#AFT5:ITC"
    ▶ Free wave leak detected (leaked waves: 373) in "AnalysisFunctionTesting#AFT6:ITC"
    ▶ Free wave leak detected (leaked waves: 261) in "AnalysisFunctionTesting#AFT6a:ITC"
    ▶ Free wave leak detected (leaked waves: 259) in "AnalysisFunctionTesting#AFT6b:ITC"
    ▶ Free wave leak detected (leaked waves: 260) in "AnalysisFunctionTesting#AFT7:ITC"
    ▶ Free wave leak detected (leaked waves: 90) in "AnalysisFunctionTesting#AFT8:ITC"
    ▶ Free wave leak detected (leaked waves: 13) in "AnalysisFunctionTesting#AFT9:ITC"
    ▶ Free wave leak detected (leaked waves: 14) in "AnalysisFunctionTesting#AFT10:ITC"
    ▶ Free wave leak detected (leaked waves: 262) in "AnalysisFunctionTesting#AFT11:ITC"
    ▶ Free wave leak detected (leaked waves: 13) in "AnalysisFunctionTesting#AFT13:ITC"
    ▶ Free wave leak detected (leaked waves: 15) in "AnalysisFunctionTesting#AFT14:ITC"
    ▶ Free wave leak detected (leaked waves: 15) in "AnalysisFunctionTesting#AFT14a:ITC"
    ▶ Free wave leak detected (leaked waves: 14) in "AnalysisFunctionTesting#AFT14f:ITC"
    ▶ Free wave leak detected (leaked waves: 15) in "AnalysisFunctionTesting#AFT14g:ITC"
    ▶ Free wave leak detected (leaked waves: 14) in "AnalysisFunctionTesting#AFT14h:ITC"
    ▶ Free wave leak detected (leaked waves: 14) in "AnalysisFunctionTesting#AFT15:ITC"
    ▶ Free wave leak detected (leaked waves: 12) in "AnalysisFunctionTesting#AFT16:ITC"
    ▶ Free wave leak detected (leaked waves: 56) in "AnalysisFunctionTesting#AFT18:ITC"
    ▶ Free wave leak detected (leaked waves: 2) in "AnalysisFunctionTesting#AFT19:ITC"
    ▶ Free wave leak detected (leaked waves: 2) in "AnalysisFunctionTesting#AFT19a:ITC"
    ▶ Free wave leak detected (leaked waves: 15) in "AnalysisFunctionTesting#AFT20:ITC"
    ▶ Free wave leak detected (leaked waves: 14) in "AnalysisFunctionTesting#AFT21:ITC"
    ▶ Free wave leak detected (leaked waves: 261) in "AnalysisFunctionTesting#AFT22:ITC"
    ▶ Free wave leak detected (leaked waves: 27) in "AnalysisFunctionTesting#CanModifyStimsetInPreSweepConfig:ITC"
    ▶ Local wave leak detected (leaked waves: 56) in "AutoTP#AutoTP_OptimumValues:ITC"
    ▶ Free wave leak detected (leaked waves: 35) in "AutoTP#AutoTP_OptimumValues:ITC"
    ▶ Local wave leak detected (leaked waves: 84) in "AutoTP#AutoTP_BadValues:ITC"
    ▶ Free wave leak detected (leaked waves: 93) in "AutoTP#AutoTP_BadValues:ITC"
    ▶ Local wave leak detected (leaked waves: 84) in "AutoTP#AutoTP_MixedOptimumBadValues:ITC"
    ▶ Free wave leak detected (leaked waves: 66) in "AutoTP#AutoTP_MixedOptimumBadValues:ITC"
    ▶ Local wave leak detected (leaked waves: 84) in "AutoTP#AutoTP_SpecialCases:ITC"
    ▶ Free wave leak detected (leaked waves: 48) in "AutoTP#AutoTP_SpecialCases:ITC"
    ▶ Free wave leak detected (leaked waves: 13) in "CheckSweepSavingCompatible:ITC"

HardwareTestingWithAnalysisFunctions.pxp: TBA

t-b commented 1 year ago

I thought about this PR some more, and think we should get it merged now. We can always come back to it later if the need arises. But the first tests posted here have shown that the wave tracking works. And where it does not work, it is not the fault of the UTF.

I found a couple of minor things still left to do:

waveTrackingMode = ParamIsDefault(waveTrackingMode) ? 0 : waveTrackingMode
[...]
variable/G dfr:waveTrackingMode = 0

Both cases should use a constant.

variable/G dfr:waveTrackingMode = waveTrackingMode

Here you don't check that the user provided something sensible.

Thanks.

t-b commented 1 year ago

Oh and I don't think you need to return NaN after ReportErrorAndAbort. At least in other places you also skip the return.