Possibility to show videos from different perspectives

[uklomp]

Hi @Jetske and @Woseseltops We talked about the possibility to upload multiple videos for signs and annotated sentences and to show this sign/sentence from multiple perspectives. We record the signs, and a new batch of annotated sentences (that are not from the corpus), with three cameras: one from the middle, like we upload now, one from the front-left and one from the front-right. We would like the following things:

1. A symbol in the video frame that users can click to see one of the other perspectives.
2. You asked before if this functionality of multiple perspectives would only be for single signs or also for annotated sentences --> we indeed also want this possibility for sentences, if it's not too difficult. --> Related to 2: If it's indeed possible for annotated sentences, the functionality regarding linked glosses should still work the same as it's working now, also when a video with a different perspective is shown.
3. We discussed what to do with search results, because then a small video from the front is shown: this will stay the same, we don't need the other perspectives here.
4. You asked what to do with deleting, moving, overwriting and you suggested to build two options: upload three videos at the same time, and upload/change only the middle video. We agree with this suggestion. In the second option, potential older videos from the left and right perspective remain, while the middle video is updated.

[Woseseltops]

To make this task a bit easier to digest, let's split it up into separate phases:

1. Upload 'other perspective video' to a gloss
2. Display 'other perspective videos', and make it possible to switch between them
3. Delete/overwrite extra videos
4. The relation to annotated sentences

[Jetske]

How would you prefer to upload the 3 files?

• As a zip file that contains 1-3 videos given some name that distinguishes which video is which perspective?
• One by one in three file upload boxes?

[susanodd]

Just a request, can this be on a different page rather than on Gloss Detail?

(Gloss Detail has some problems with too much code.)

[uklomp]

How would you prefer to upload the 3 files?

• As a zip file that contains 1-3 videos given some name that distinguishes which video is which perspective?
• One by one in three file upload boxes?

I would say the first option, but it might be that @rem0g has different wishes for uploading things automatically?

[uklomp]

Just a request, can this be on a different page rather than on Gloss Detail?

(Gloss Detail has some problems with too much code.)

Preferably not, especially not for the sentences. For the single gloss videos, I can imagine a pop-out window for the videos, if that would make things better. Going to a completely different page is not really an option.

[susanodd]

Just a request, can this be on a different page rather than on Gloss Detail? (Gloss Detail has some problems with too much code.)

Preferably not, especially not for the sentences. For the single gloss videos, I can imagine a pop-out window for the videos, if that would make things better. Going to a completely different page is not really an option.

Some of the issues that refer to Gloss View need to be fixed. There is a coding problem because there is not enough space at the moment. (E.g., the issues where the OK buttons appear off the screen to the right. And where menus overlap each other.)

[susanodd]

[CODE / DEVELOPMENT / RELEVANT TO @Woseseltops ] @Woseseltops I can add some dummy prep code where needed. Because Gloss Detail is so crowded, @Jetske made the Annotated Sentence Edit go to a completely different page. The same could be done here. The annotated sentences are only visible on the production server because there are no videos or eaf files on the development servers. That made it difficult to debug the code. Kindly look at the recent updates to the "video" models. I added inheritance for the NME videos. They also do not import to the same location, since it had to work with the API as well as Gloss Edit and the names had to become different. The newest updates for Searching on annotated sentences/glosses also were worked on by @Jetske and @susanodd and to be merged, needed to be merged and put on master already. (Those are live, but you did not review the code, because they needed to be merged as interdependent. The scrollbar and query view are also upgraded to work with these.) That is related to this because it involves Gloss Edit view and links between it and other pages. The pages need to return to where the links were clicked on, but to different caller views.

[susanodd]

There are multiple issues with new kinds of upload data. Plus the API for the example sentences still needs to be implemented. (#1207) Some test data is needed.

[susanodd]

I put this on signbank-susan

You can see it as a new panel right above the NME videos.

Not all the bullets are implemented yet, just the upload.

[susanodd]

4. You asked what to do with deleting, moving, overwriting and you suggested to build two options: upload three videos at the same time, and upload/change only the middle video. We agree with this suggestion. In the second option, potential older videos from the left and right perspective remain, while the middle video is updated.

I implemented a separate upload for the perspective videos. I added a choice list where you can choose either left or right. If there already is a left or right, the old video is effectively replaced.

I made the choices "left" and "right" hard-coded. Because this ends up in the file name itself, it would be dodgy to actually allow language translations here. Just as for the NME videos, "nme" appears in the file names for those. The uploaded perspective videos are stored in the same (sub)folder as the other gloss videos. As (4) all three can be uploaded independently of each other. If additional perspectives are needed, more choices can be added, such as "top" or "headcam". :)

[uklomp]

I put this on signbank-susan

You can see it as a new panel right above the NME videos.

Not all the bullets are implemented yet, just the upload.

Hi Susan, it shouldn't be a new panel at all. We want the option to show the different perspectives within the video frames that are already there. So: we now have video's from the middle as the neutral gloss video. Here, in this video frame, we want the option to click and see different perspectives. I showed this to @Woseseltops and @Jetske so please ask them if you're unsure what we want.

[susanodd]

I put this on signbank-susan You can see it as a new panel right above the NME videos. Not all the bullets are implemented yet, just the upload.

Hi Susan, it shouldn't be a new panel at all. We want the option to show the different perspectives within the video frames that are already there. So: we now have video's from the middle as the neutral gloss video. Here, in this video frame, we want the option to click and see different perspectives. I showed this to @Woseseltops and @Jetske so please ask them if you're unsure what we want.

Yes, I understand exactly what you want. I'm trying to implement it. (That's when I came across a bug in a different issue about the sentence videos, some of the javascript was not being done because of a duplicate id in the template. I can't upload any normal videos at all because of the bug, so it doesn't work to check out the perspective videos.)

The videos still need to be uploaded some place in the interface. I just put them there. Where do you want the upload to be, if not there?

It can't be in the same area as the normal gloss videos. That is because of some hard-coded identifiers for normal videos. Those are also "backed up" and more things happen for those files.

A different "upload" button is needed for the perspective videos. And the "left" or "right" indication is also needed in combination with this button.

I can move it to just be underneath all the other stuff in the left panel. But the "citation image" ought to remain below the normal video because it is supposed to be generated at the same time as the normal video.

My opinion is that the left column is too crowded once it's open in Edit mode.

[susanodd]

I moved the panel to the left column.

Question, the example perspective videos are quite large (very wide angle). Will these be cropped to remove the cameras/lighting equipment at the sides?

[uklomp]

For manual upload, I would want to upload it in the regular spot, so the option to upload multiple videos. @rem0g how do you want to upload multiple videos from multiple perspectives from signcollect to signbank? As for cropping: the official videos will not show cameras or lights anymore, indeed. With some videos, there will still be a lot of space on the side. We now turned the cameras in portrait position so that this will become less in the future, but we already made many recordings in the horizontal way.

[susanodd]

For manual upload, I would want to upload it in the regular spot, so the option to upload multiple videos. @rem0g how do you want to upload multiple videos from multiple perspectives from signcollect to signbank? As for cropping: the official videos will not show cameras or lights anymore, indeed. With some videos, there will still be a lot of space on the side. We now turned the cameras in portrait position so that this will become less in the future, but we already made many recordings in the horizontal way.

We're having a bit of trouble with videos in the API. @Jetske is working on that now for sentences. She said @rem0g has had some problems with videos as well.

I will ask @Woseseltops about this. The problem is that the "video" needs to be put into the file system. (This involves multiple steps....)

[CODE TRANSACTIONS] But the database is also "locked" while creating objects in which the video is stored. So what has been happening is that the "video" isn't finished yet being uploaded in the file system and the database remains locked. Or if it's unlocked, the video hasn't been put into storage yet. (The operation of putting a file into the file system takes more time than just creating objects, so there is some time mismatch and this causes stuff to fail.)

We need to consider this if multiple videos are uploaded at the same time. It needs to be "all or nothing" and the database "should not be locked for an excessive amout of time".

At the moment, only one perspective video is uploaded at a time.

[CREATED NEW ISSUE FOR THIS DESCRIBED BELOW] This is causing problems on my Mac computer. I needed to remove some code that "re-loads a video if a new one is uploaded" (that was another issue that was solved in the past. But now there are many video (elements) on the Gloss page and this "reload video" code is kind of going berserk.) (My videos disappeared from my server -- iCloud server -- and the citation image also disappeared.) So there are some bugs in the upload code.

(I don't know if that has anything to do with "atomic transaction blocks". If a transction -- ie. uploading a video -- fails for some reason, then the "transaction" would be rolled back by the database. That could cause things to be deleted in spite of the user not having deleted them. It is a side effect of a transaction failure.)

[susanodd]

For uploading a bunch of videos by the user, I made the Manage Media page that accepts a zip file. Then it's only ONE file that is uploaded.

I did that to avoid transaction problems.

But on the Manage Media page, only about 15 videos are "processed" -- stored in the respective glosses -- at a time. Otherwise, the file system can't catch up and it starts to fail. We don't have any concurrent writing to the file system.

[susanodd]

In summary, I don't know what to do.

The underlying operations still need to be done sequentially. (The perspective videos need to be saved to the file system one after the other.)

[susanodd]

I thought of something else. Django removed the "multi-select" for files with Django 4.2 for security reasons.

We had to remove this from uploading of EAF files for the Corpus. (In the Manage Dataset, there used to be functionality for uploading multiple EAF files at the same time.) This feature was removed from Signbank.

So it won't be possible to upload multiple perspective video files at the same time.

[susanodd]

I moved the code discussion to the Pull request. There are screenshots of the perspective panel there. The goal now is to swap out the videos without reloading video files in the browser.

[susanodd]

@uklomp Here's the first gloss with perspective videos showing

https://signbank-susan.cls.ru.nl/dictionary/gloss/36430/

It's looping at the moment. The play and pause buttons don't work yet.

It looks weird to switch them during play unless they are cropped to keep the signer in the middle. The videos are also quite short and switching by the time you click the video has already finished.

The upload for the perspective videos is a panel that opens. (This can be moved below or between other sections, but not really. The videos need to be "visible" after uploading. They can't show "behind" or "block" the normal video (which is linked to a drag and drop as well as the citation image code. They can't show in the target display area right after uploading. You also need direct access in the layout to be able to delete them.)

At the moment, perspective videos cannot be uploaded at the same time as a normal video because of how videos are stored. @Woseseltops should not have suggested this as an option. This needs to wait for #1332.

[uklomp]

I can't see it, I get this message: The gloss you are trying to view is not in your selected datasets. I'm logged in btw.

As for uploading and the videos being visible: let's discuss this on Thursday with @Woseseltops together.

[susanodd]

I can't see it, I get this message: The gloss you are trying to view is not in your selected datasets. I'm logged in btw.

As for uploading and the videos being visible: let's discuss this on Thursday with @Woseseltops together.

It's in the test dataset tstMH (Test Dataset Micha). You need to select it.

This is an example where the error reporting is insufficient. It ought to tell the user which dataset the gloss is in. (I'll add that to #1331.)

[uklomp]

Ok, I saw it now, looks good!

[Woseseltops]

To do:

• Button order: left center right
• Small icons instead of buttons
• Overlap the video instead of above and under

[susanodd]

I modified the collapsible panel heading to be "upload perspective videos"

The lower buttons (play/pause) are now in front of the videos. (ON UBUNTU FIREFOX)

But the trick does not work for the top buttons. (It ends up hiding them or moving the entire video up or cropping off part of the videos)

ON MACOS SAFARI THIS DOES NOT WORK The buttons end up behind the videos

@Woseseltops the solution we came up with yesterday does not work on Safari. The buttons are behind the videos, not in front of them. This won't work.

What to do? Don't let users use Apple?

[susanodd]

Regarding use glyphs instead of words for the "perspective" video buttons.....

@uklomp do you guys already have appropriate icons to use for switching perspective?

I used some hands pointing, but that looks weird. There aren't any free glyphicons with silhouettes.

I looked up "head looking left" but the symbols may have "different thinking perspective" semantics attached to the symbols. (like a rainbow isn't a rainbow, a head with arrows pointing might mean something else, like a red baseball cap isn't the Rosmalen tennis tournament)

[susanodd]

[Details]

@uklomp here I try to describe how things work with the three videos and why they work the way they do.

The three perspective videos are each inside their own html element. These are next to each other during parsing. Then only one is made visible. But underneath there are still three elements. The controls need to apply to all three videos. So if a button is pushed all three play at the same time (@Woseseltops), but still only one is visible. So if you press pause, all three videos are paused and you can switch to a different perspective, also paused at the same point.

If we use the "native" video controls for each video, then the play / pause will only work on that one video. So the "pause" all thing won't work. But the controls would be on top of each video.

The buttons to "swich" are "outside" of the videos because they need to "show" and "hide" the other videos. (There is a html structure and it is nested this way.) If you end up with the buttons behind the video in some browsers, then you can't click them. (This is browser-dependent where it puts buttons behind or in front. Or whether it just shoves stuff around and crops the other things to make things fit in the space. You can't click on things that are not visible. On Firefox it cropped the vertical dimension of the video to show the l-m-r buttons. The browser "makes stuff fit". The specific perspective video code is not in a vacuum, so the browser also needs to keep showing the rest of the page.)

The controls if they are the native controls are dependent on the browser.

So we can "put the controls on top of the videos" by using the "native controls". But then the "pause" won't work for all the videos.

At the moment, the "normal" video has hard-coded element tags and hard-coded javascript. This means that the code that works for the normal video is tied to that video. Not for the new videos. Because there is a hierarchy of the elements and the ids have scopes I have not "messed with" the "normal" video code.

[susanodd]

The "fontawesome" icons require a Pro license for the non-basic icons.

We could user arrows for the two perspectives icons. (But for the middle one?)

left-long-solid.svg

I tried some other icons for normal/middle instead of the "user" icon, but it just looks confusing.

[susanodd]

I removed the button outline. Here's what they look like in Safari (MacOS) in dark mode.

[susanodd]

Opinion:

The left middle right buttons are potentially confusing. As opposed to video games, the left button gives the camera on the left, so the signer is actually looking right, not left.

I can swap the pointing icons around so they point to the middle. (?)

(The sample files are named as such with left and right as such, matching the camera position, not the signer.) [NOTE these are the file names of the Signbank stored videos (..._left.mp4, ..._right.mp4). With the current implementation, the user can choose any file they want. No need to put anything in the filename on the user's side.]

[uklomp]

we were thinking of icons like this

[susanodd]

Okay, so the camera icon. That's much more clear. Thanks.

Are those icons a pop-up then? Is it not on top of the video, but to the right?

If the icons are small, the user needs to move the cursor / mouse around to be on top of it.

Depending on the background of the video that the icons might not show up. I think those buttons as shown with a gradient for the controls are the browser?

[susanodd]

The icons have been made the rotated camera icons.This is on signbank-susan.

Putting the buttons "in front of the video" does not work on Safari. So they are still around the video.

[susanodd]

This has been deployed

[uklomp]

I think this design is good. I asked my colleagues if they like it as well. Hope to report back later this week.

[uklomp]

• as for the standard options to play videos: we are now missing the full screen button, and one colleague also mentioned the progress bar when a video plays.
• as for the buttons that are proposed for multiple perspectives: we like it. We would like to see which perspective is currently shown. Maybe by underlining the current perspective symbol?

[susanodd]

[CODE]

BUG related to videos being deleted. There are too many submit buttons in the LEFT column where videos are uploaded. All the "video forms" SHARE one "form implementation" internally.

https://github.com/Signbank/Global-signbank/blob/9c35b16e31e8179c07c58deba58da9fc6590319c/signbank/dictionary/templates/dictionary/gloss_detail.html#L839

This code is causing problems on Safari. It is "hidden" but why is this here? This function is inside the "record from webcam code" and this code is not available on Safari, so this should not be generated at all in the template (because the code is unreachable for Safari.)

The uploaded Perspective videos disappear on Safari after a while. I suspect it's because they share some of the "input" fields in the (Django) forms for videos. Can this line of code be removed, please, @Jetske ? (It's out of scope since it's in an external javascript file that is not used on Safari, since the upload does not work and is disabled in the code. So this causes a javascript error for the template.)

[susanodd]

This is completed and in use. I'll make a new issue for the "full screen" button.