Fixing #4

[carueda]

Hi @hohonuuli ... for now, just a draft PR to facilitate discussion.

[carueda]

Brian, I think the change has been completed AFAICT now. I've done basic testing running the GUI, pausing, drawing a rectangle, replaying and checking (visually) that the duration (now indicated in millis) of the drawn rectangle is as expected (that is, about twice as long per the value in the Settings dialog:

(5000 millis in this test just to facilitate the "visual" verification).

Of course, please take a look and let me know what you think.

[hohonuuli]

@carueda Thanks for the PR. I ran a sanity check with the window set to 100ms. With this setting, Cthulhu is skipping over and not drawing bounding boxes during playback. I'm thinking that that the the way localizations are selected to be drawn inside cthulhu is not fast enough to keep up with the playback. I've posted a video of this issue at https://youtu.be/wrB0GMTFmN0

[carueda]

OK, I'll continue reviewing the code to see where relevant adjustments are needed. BTW, I would appreciate a live session as I'm not sure how to "integrate" that other GUI shown in your video.

[hohonuuli]

There's a few pieces to running a live session. Here's the steps.

Spin up M3 microservices pointing at MBARI's development databases

1. git clone git@bitbucket.org:mbari/m3-microservices.git
2. cd m3-microservices
3. dev_mbari.sh up

Run VARS Annotation from source code (gives access to debugging)

1. git clone git@github.com:mbari-media-management/vars-annotation.git
2. cd vars-annotation
3. gradlew run --info

[hohonuuli]

dev_mbari.sh up <- this shell script passes commands through to docker compose. So you can use dev_mbari.sh up -d, dev_mbari.sh down, etc.

[hohonuuli]

Also, you should be running Java 11 when running VARS annotation. It uses java modules and changes to modules after Java 11 break things. When Java 17 comes out, I'll update VARS to use that version.

[carueda]

Just some quick misc notes while playing with the annotations:

(no replies requested for now )

• per the logging, seems like "durationMillis" is always zero when "Publishing to 'localization'":

  DEBUG o.m.v.sharktopoda.client.localization.IO | Publishing to 'localization': {"action":"add","localizations":[{"concept":"Concept","elapsedTimeMillis":5156,"durationMillis":0,"videoReferenceUuid":"44cf6e6a-3844-4502-a94f-dbbd579ea326","localizationUuid":"23541f3f-e2ca-478a-ace5-78089a01f741","x":584,"y":414,"width":630,"height":432}]}

• not sure yet about the interplay of this project and vcr4j, but the comment in vcr4j for that field says:

   * This is the amount of time that the localization applies to the annotation.
   * It is optional and if not present should be considtered to be 0. At a
   * minimum the bounding box should be drawn from `elapsedTime` to
   * `elapsedTime + duration`
   @SerializedName("durationMillis")
   private Duration duration;

• there are no comments for the startTime and endTime fields in the Annotation class. But one of the constructors indicates:

  The end time will be set to the same value as the start time (this is OK as when annotations are rendered there is a padding "time window" that will extend any such zero duration.

  Yes, that's what I have already gathered but still trying to find where such "padding" takes place.

[carueda]

Just captured this video with those adjustments f98c5ba: https://youtu.be/Pu2pc6pPbm0

Note, I'm still just running cathulhu alone (./gradlew -t run), not with any other "integration" yet.

This change does seem to have improved the handling a bit, although, occasionally there are still some missing rectangles when replaying..

As indicated in the comments, seems like libvlc_media_player_get_time does not report with enough time resolution. I changed the timer period to 100ms (previously 500ms), but still the delta of the status time is always > ~250ms, more often ~ 400-500ms.

Is there a way to instruct libvlc to use a finer resolution?

[hohonuuli]

re: duration: In VARS, a user may set a duration or, when annotations are imported from ML. they may be a duration set. Basically it says, we saw "this thing" for "this long'. The Cthulhu player never sets the duration value though; the duration is always set by another application.

I think Cthulhu has some handling for duration internally, so how long a bounding box is drawn includes the duration. I think Cthulhu should ignore the duration or maybe we can toggle it on/off. I can discuss the details of why we might ignore duration in our next VAA meeting.

[carueda]

@hohonuuli

Since mediaPlayer.status().time() does not report with the desired time precision, I did the following:

• during playing, our 100ms timer always calls the handler at every tick
• the handler is called with the following time depending on whether mediaPlayer.status().time() has reported a new time or not:
  • if reported time is new, then that's the one used for the call
  • otherwise, the time is the previously reported time plus the difference in system time wrt to when that last time was reported. This also takes into account the "direction of time" based on reported mediaPlayer.status().time() values, which should help better reflect the changes when the user drags the time control back and forth during playing.

This strategy seems to have improved this overall handling, short demo here: https://youtu.be/V3Ym595yuHI . Please have a look at these changes.

Next, I'll try to "integrate" the VARS Annotation build and see how this all behaves.

---

EDIT: btw, i tried the MediaPlayerEventListener approach, but the listener is also called not more frequently than every 250ms. (no surprise here, as the same basic mechanism is used elsewhere in the code.)

[hohonuuli]

Wow! Your video shows a HUGE improvement. Would mind posting that into the vars-annotation-assistance slack channel to share with the team? That looks great.