Run all multivariate capable `aeon` classifiers on 40 MTSC datasets

[TonyBagnall]

we have 40 datasets now, so it would be good to get reference results with all capable classifiers. I will just track problems here and gather results on the beast. Some are large so may need contracting. First wave: sktime (version 0.12) run using estimatro_evaluation package, run on ADA. All classifiers with 16gb, no time limit

1. HC2 components, including train files
2. HC2 proper, no train files
3. Compose HC2, verify same results as (2)
4. Other MTSC capable classifiers (form list and post here)
5. Deep learning MTSC classifiers

The 40 data

        "ArticularyWordRecognition",
        "AsphaltObstaclesCoordinatesEq",
        "AsphaltPavementTypeCoordinatesEq",
        "AsphaltRegularityCoordinatesEq",
        "AtrialFibrillation",
        "BasicMotions",
        "Blink",
        "CharacterTrajectoriesEq",
        "Cricket",
        "DuckDuckGeese",
        "EigenWorms",
        "EMOPain",
        "Epilepsy",
        "EthanolConcentration",
        "ERing",
        "EyesOpenShut",
        "FaceDetection",
        "FingerMovements",
        "HandMovementDirection",
        "Handwriting",
        "Heartbeat",
        "InsectWingbeatEq",
        "JapaneseVowelsEq",
        "Libras",
        "LSST",
        "MindReading",
        "MotionSenseHAR",
        "MotorImagery",
        "NATOPS",
        "PenDigits",
        "PEMS-SF",
        "PhonemeSpectra",
        "RacketSports",
        "SelfRegulationSCP1",
        "SelfRegulationSCP2",
        "Siemens",
        "SpokenArabicDigitsEq",
        "Tiselac",
        "StandWalkJump",
        "UWaveGestureLibrary"

[TonyBagnall]

First run through problems:

DrCIF crashes on one of the new MTSC datasets, EmoPain, that does not occur with the other three base classifiers. It looks like an interval is generating a NaN somewhere

[TonyBagnall]

all (STC, DrCIF, STC, TDE) run out of memory on InsectWingbeatEq, re run with 64GB

[TonyBagnall]

do not have equal length JapaneseVowelEq problem

[TonyBagnall]

TDE didnt run on PenDigits for some reason

[TonyBagnall]

code not correctly ignoring existing results

[TonyBagnall]

TDE Eigenworms missing some folds

[TonyBagnall]

Tiselac is a pig for all, will probably need a contract

[TonyBagnall]

TDE PhonemeSpectra missing some folds

[TonyBagnall]

15/9/22, four HC2 components: 32 complete DATASETS removed (8): [PhonemeSpectra, PenDigits, InsectWingbeatEq, FaceDetection, EigenWorms, EMOPain, Tiselac, JapaneseVowels]

[TonyBagnall]

16/9/22 Arsenal: Tiselac, JapaneseVowelsEq STC: Tiselac, JapaneseVowelsEq DrCIF: InsectWingbeatEq, FaceDetection, EMOPain, Tiselac, JapaneseVowelsEq TDE: PhonemeSpectra, PenDigits, JapaneseVowelsEq, InsectWingbeatEq, FaceDetection, EigenWorms, Tiselac

running atm: Tiselac: STC, DrCIF, TDE InsectWingbeatEq: TDE, DrCIF FaceDetection: TDE

not running, maybe crashed: DrCIF: FaceDetection, EMOPain TDE: EigenWorms, PenDigits (?), PhonemeSpectra 27 out of 30, others may have crashed

TDE: PenDigits TDE, series too short: Error in TemporalDictionaryEnsemble, min_window =10 is bigger than max_window =8. Try set min_window to be smaller than series length in the constructor, but the classifier may not work at all with very short series

---

EigenWorms: out of memory

[TonyBagnall]

17/9/22: running ROCKET Mini-ROCKET Multi-ROCKET FreshPRINCE MUSE over long weekend

[TonyBagnall]

Time out on cluster TDETiselac (all on Tiselac?) TDEFaceDetection TDEInsectWingbeatEq CIFTiselac STCTiselac CIFInsectWingbeatEq

[TonyBagnall]

Tunning Tiselac on the kraken, need to set up to safeguard against windows box. Full list on MTSC capable classifiers

multivariate_classifiers = [ "Arsenal", "CNNClassifier", "CanonicalIntervalForest", "Catch22Classifier", "ColumnEnsembleClassifier", "DrCIF", "FreshPRINCE", "HIVECOTEV2", "IndividualTDE", "KNeighborsTimeSeriesClassifier", "MUSE", "ProbabilityThresholdEarlyClassifier", "RandomIntervalClassifier", "RocketClassifier", "ShapeletTransformClassifier", "SignatureClassifier", "SummaryClassifier", "TSFreshClassifier", "TemporalDictionaryEnsemble", "WeightedEnsembleClassifier", ]

KNN is the new version which needs testing and debugging.

[TonyBagnall]

20 classifiers, not running ColumnEnsembleClassifier, ProbabilityThresholdEarlyClassifier or WeightedEnsembleClassifier since they have no default. Delay running CNNClassifier and SingatureClassifier, they require soft deps Run three versions of KNeighborsTimeSeriesClassifier: 1NN-ED, 1NN-DTW and 1NN-MSM (all with defaults). Leaves 18 classifiers, Signatures and CC (and the other deep learning algorithms) to do

[TonyBagnall]

static String[] referenceClassifiers = {
        "Arsenal",
        "CIF",
        "Catch22",
        "DrCIF",
        "FreshPRINCE",
        "HC2",
        "IndividualTDE",
        "1NN-ED",
        "1NN-MSM",
        "1NN-DTW",
        "MUSE",
        "RandomInterval",
        "ROCKET",
        "STC",
        "Signature",
        "Summary",
        "TSFresh",
        "TDE",
};

[TonyBagnall]

Updating for redux Original 26 MiniROCKET wont run on PenDigits, length < 8 FreshPRINCE missing on DuckDuckGeese: running10/3 and FaceDetection memory with 100gb TDE missing on FaceDetection: running 8/3 will probably time out HC2 missing on FaceDetection: (TDE time out) PenDigits (arsenal crash), PhonemeSpectra (Matthew build from file) InceptionTime missing on EigenWorms (crashed on memory )

           "ArticularyWordRecognition", //Index 0
           "AtrialFibrillation",
           "BasicMotions",
           "Cricket",
           "DuckDuckGeese",
           "EigenWorms",
           "Epilepsy",
           "EthanolConcentration",
           "ERing",
           "FaceDetection",
           "FingerMovements",
           "HandMovementDirection",
           "Handwriting",
           "Heartbeat",
           "Libras",
           "LSST",
           "MotorImagery",
           "NATOPS",
           "PenDigits",
           "PEMS-SF",
           "PhonemeSpectra",
           "RacketSports",
           "SelfRegulationSCP1",
           "SelfRegulationSCP2",
           "StandWalkJump",
           "UWaveGestureLibrary"
   };

[TonyBagnall]

Original Equalised Length complete public static String[] equalised={ "AsphaltObstaclesCoordinatesEq", "AsphaltPavementTypeCoordinatesEq", "AsphaltRegularityCoordinatesEq", "CharacterTrajectoriesEq", "JapaneseVowelsEq", };

[TonyBagnall]

New problems, TISELAC too big, others compete

public static String[] newMTSC={ "EMOPain", "EyesOpenShut", "MindReading", "MotionSenseHAR", // "Tiselac", };

[TonyBagnall]

converted problems: AustraliaRainfall: none of them, ignore BeijingPM25Quality-no-missing: TDE/HC2 Timeout BenzeneConcentration-no-missing: TDE/HC2 Timeout BIDMC3x3: 1nn time out, TDE/HC2 memory IEEEPPG: TDE/HC2 out of memory: running 10/3 with 100gb LiveFuelMoistureContent: out of memory running 10/3 with 100gb Newsx2: FP,1NN,DrCIF,TDE,HC2, MultiR PGDalia: FP, 1NN, DrCIF, TDE, HC2, MR