search

clamsproject / app-swt-detection

CLAMS app for detecting scenes with text from video input
Apache License 2.0
1 stars 0 forks source link

Change the MMIF output using TimePoints and TimeFrames #62

Closed marcverhagen closed 9 months ago

marcverhagen commented 9 months ago

Because

The MMIF output now is a list of TimeFrames:

{
    "@type": "http://mmif.clams.ai/vocabulary/TimeFrame/v1",
    "properties": {
        "start": 2000,
        "end": 4000,
        "frameType": "chyron",
        "score": 0.996342992782,
        "scores": [0.995216727256, 0.9942985177, 0.996800303459],
        "points": [2000, 3000, 4000],
        "representatives": [4000],
        "id": "tf_1"
   }
}

After some discussion with @keighrim we are now thinking:

{
    "@type": "http://mmif.clams.ai/vocabulary/TimeFrame/v1",
    "properties": {
        "targets": ["tp2", "tp3", "tp4"]
        "frameType": "chyron",
        "score": 0.996342992782,
         "representatives": ["tp4"],
         "id": "tf_1"
   }
}

The targets would be TimePoints with scores for all labels, for example:

{ 
    "@type": "http://mmif.clams.ai/vocabulary/TimePoint/v1",
    "properties": { 
        "id": "tp1", 
        "timePoint": 1000, 
        "label": "slate",
        "classification": {"slate": 0.71, "chyron": 0.3, "credit": 0.4}
    }
}

See also #41 .

Done when

No response

Additional context

No response

marcverhagen commented 9 months ago

Did much of this in https://github.com/clamsproject/app-swt-detection/commit/eaa65220234120eecc1011979e8c03253494d4fb. But due to limitations imposed by mmif-python (see https://github.com/clamsproject/mmif-python/issues/252), the classification for timepoints looks like

"classification": [
    "slate:4.1978837543865666e-05",
    "chyron:0.9895544052124023",
    "credit:0.0007274810341186821",
    "NEG:0.009676150046288967"
]

Not sure what is worse, the above or

"labels": ["slate", "chyron", "credit", "NEG"],
"scores": [4.1978837543865666e-05, 0.9895544052124023, 0.0007274810341186821,  0.009676150046288967]
keighrim commented 9 months ago

closing as duplicate to #41