Mac bugs + feature request: audio/subtitle track selection

[zackbatist]

Hi, thanks for developing this fantastic program. I am still exploring its kinks, trying to figure out if it's worth using for my thesis. I'm just writing to report some issues that I've encountered so far. I'm using MacOS 10.14.5 Mojave on a MacBook Pro (Retina, 15 inch, mid-2015, 2.5 GHz Intel Core i7, 16 GB 1600 MHz DDR3), so perhaps you can count this as MacOS testing (though it's not really that systematic, admittedly).

First thing, I'm dealing with numerous large video files, and each time I open the project a backup is created, which copies all of the project's files. It might be a good idea to give the option of referring to files stored in an external folder, via symlink or whatever other mechanism, rather than bringing over the new files to the project folder. Alternatively, perhaps there could be an option in the backup settings to only backup certain files, or certain kinds of files.

Another thing is that there is no ability to manage files or view video files when the project is stored on an external drive. The drive I tested on is formatted as Mac OS Extended (Journaled) and is encrypted. The encryption might be tripping it up, so I'll have to test on another drive to see if this replicates.

Additionally, the open file dialog (where I select the files from the finder) stays at the front of my screen, in front of all other apps, which is a major nuisance, especially when dealing with large files. This also happens when opening a project, which involves duplicating all of the files before the process can be completed and the window can be closed.

Finally, I have a feature request. Would it be possible to allow the user to enable/disable different audio or subtitle tracks on the fly? This is a basic feature of VLC, so it might be feasible.

[ccbogel]

Thanks for the interest in QualCoder. I used to have a Mac mini which now runs Linux Mint, so I cant test on Mac products myself. Yes I can enact some of those changes fairly easily - e.g. back up only some file types, I think would be fairly easy to do. Re external project on encrypted drive - hopefully only encryption is the problem, let me know. Re open file dialog staying open too long - yes i can see how this happens when opening a large project - i will see if i can implement a workaround (e.g. qt process events). When/where specifically does this occur at other times? Re subtitle tracks - I will look into it - is there a sample video with tracks that I could experiment with.

[ccbogel]

@zackbatist Regards numerous large video files, and each time I open the project a backup is created, which copies all of the project's files. I have made changes to the settings options: (1) an option to NOT automatically back up the project folder when a project is opened. (2) an option duringthe auto backup, to backup all files excluding audio/video files (this uses pattern matching for *.mp4 and several other file extensions). Note: I have also slightly changed the format for how the user settings are stored, so you might need to review these in the Settings dialog - e.g. your coder name might be reset to 'default'

Additionally, the open file dialog (where I select the files from the finder) stays at the front of my screen, in front of all other apps, which is a major nuisance, especially when dealing with large files. Is the open file dialog occurring when you are trying to Import many large files using the Manage Files dialog? If yes, then I have added a process events line in manage_files.py which hopefully will minimise this issue.

[ccbogel]

@zackbatist OK regards an external drive. I tested using a project on a USB with Linux Mint. I also had problems viewing video from the USB. Using the Manage Files dialog I could add a file to the project on the USB, but I afterwards could not delete it from the USB. So, I'm going to have to recommend using the project folder on the main machine and not on external drives. I'm guessing this might be an issue that is too difficult for me to fix.

[zackbatist]

@ccbogel sorry for the delay getting back to you.

Re subtitle tracks - I will look into it - is there a sample video with tracks that I could experiment with.

Here is a file with two audio tracks: https://we.tl/t-Kpwm1FCxhA. In VLC (on a mac at least), expand the 'Audio Track' menu from the menu bar item called 'Audio' and you should be able to select Track 1, Track 2 and Disable (the first track doesn't have sound for around the first 5 seconds). For subtitles, VLC typically detects .srt files (time-coded transcriptions) in the same directory and gives you an option to select which one to read.

Is the open file dialog occurring when you are trying to Import many large files using the Manage Files dialog? If yes, then I have added a process events line in manage_files.py which hopefully will minimise this issue.

This happens whenever I open the file manager menu (i.e. importing a file, opening a project) The dialog stays open until the processes of locating and managing all the files are completed.

So, I'm going to have to recommend using the project folder on the main machine and not on external drives. I'm guessing this might be an issue that is too difficult for me to fix.

This seems reasonable, though I will try testing in a more systematic way with different encryption settings, disk formats, etc. I will report back, and hopefully after identifying the problem we might think of a potential solution.

Unfortunately I will not be able to test things out (including fixes you just made) before the weekend, as my schedule is pretty crammed right now.

[ccbogel]

@zackbatist Hi, I have been able to enact choosing between multiple audio tracks in an MP4. It is slightly quirky - as you need to briefly play then pause the video first. This is needed so that QualCoder can grab the audio track information from vlc.

[zackbatist]

Hi, sorry again for the delay. I managed to try things out by creating test projects both on my device and on a fresh microSD card (exFAT, non-encrypted). It works great, aside from the slight quirk that you mentioned, but that is not a major issue and seems to be related to the overall mechanism of track selection, not your own implementation of it. However, after stopping a video and re-starting it, the audio track resets to the first one as a default, but the selector still reads the second, which means the user must go back to 1 before switching to 2. Again, not a huge issue, but still worth noting.

I also picked up another bug, which may or may not be related. If you close the video screen, the audio still plays. With the video screen closed, if you hit pause the audio stops, and then pressing play again plays the audio again (without the video). Pressing stop crashes python. This happens with media containing both single and multiple audio tracks.

[ccbogel]

@zackbatist (1) I have improved the handling of the display of tracks (1 ..n) in the combobox. No need to pause to get the track information to fill the combobox. (2) When a video is stopped - the combobox will default to the first track - and display this to the user too. (3) When the video screen is closed. OK I took liberties here and have disabled the close button on the dialog that displays vlc. So this should now only close when closing the controls dialog. (The only time this won't work is if there's a programming error occurs in the control dialog).

These changes should be tested in the coding dialog as well via the manage_files dialog. To me - both seem ok now.

Are you also converting audio to text, e.g. manually and/or using something like https://otter.ai/login and having transcription files for coding? I am asking because I want to understand the interpretive aspect of coding video segments. There might be additional features I need to develop in this space, as I have primarily only coded textual data.

p.s. above subtitle tracks were mentioned, but I have so far only looked at multiple audio tracks. Again if possible, is there a short video containing subtitles? As I guess this could be extracted as a text transcription of the video for qual coding.

[zackbatist]

Are you also converting audio to text, e.g. manually and/or using something like https://otter.ai/login and having transcription files for coding? I am asking because I want to understand the interpretive aspect of coding video segments. There might be additional features I need to develop in this space, as I have primarily only coded textual data.

I'm doing manual transcription. A few of my interviews were transcribed using MaxQDA's built in transcription features, but I'm shifting away towards F5 due to the fact that it is cumbersome to export these transcripts with timestamps as plain text. For interviews I tend to code and memo the text while also playing back the video, and I code the video when there are gestures, pauses or other visual cues that I want to note. I code the video for my observation-style data, but when notable conversational exchanges occur I transcribe those, along with timestamps to indicate where they fit in. So I guess it might be handy to be able to detect standard timestamp formats (i.e. #HH:MM:SS.sss#, etc) and use them to jump to the specified times in videos.

Also, things get messy when trying to integrate codes from these different 'streams' (text vs video), so I've toyed with the idea of including tags (which tend to be manifested as codes, when working with MaxQDA) to specify which codings within a common interview or case are derived from text or video. I guess this might overlap with file and case attributes to some degree, but it is handy to have that info visible within the text to describe the kind of media that specific codings are drawn from.

Regarding subtitles, I would be exporting the transcripts from F5 as srt files, which essentially determines durations between timestamps, and displays text situated between timestamps for such durations. Example:

1
09:33:04,100 --> 09:33:09,600
Blah blah  

2
09:33:10,000 --> 09:33:13,900
Bloop bloop  

3
09:33:14,300 --> 09:33:18,300
Meep moop  

I could definitely see this being integrated as part of QualCoder itself, but it's probably not necessary to reinvent the wheel, especially since VLC is already capable of reading from srt files. I will put together a sample video with subtitles, and post it here shortly.

To be clear, I still haven't used QualCoder for substantial coding yet. I plan to try it out within the next month or so for a project whose data is finally being edited/assembled, and it looks like it will be totally feasible given the amazing changes that you have made.

Edit: here's a zip file containing a 20 second clip and 2 srt files, one in english and the other in french: https://we.tl/t-8jzUOp2gv1. VLC does a good job detecting compatibility (by file name and location, I presumed), and subtitle tracks can be selected or manually assigned on the Subtitle menu, just like with selecting audio tracks.

[ccbogel]

OK thanks for the input. I have now improved integration of video/audio and transcribed text. You can now code both the a/v directly via time segments and also the associated transcription file at the same time.

One other issue might be that I have not so far added a scaling factor to the canvas that shows coded segments in the video. With short videos it is currently OK, but with long videos, short coded segments would look like grains of sand.

Thanks for the video with srt files, I did not realise they were separate rather than included within the mp4 folder itself. I will incorporate this when I have time, but this might be a lesser goal for now.

[ccbogel]

OK I have added detection within the transcribed text for the following timestamp formats: [00:34:12] [45:33] [01.23.45] [02.34] #00:12:34.567# 09:33:04,100 --> 09:33:09,600

In the Manage_Files Dialog when a video is played, and the scroll transcript checkBox is checked, the text window transcription will scroll along based on the timestamps. If you have to edit the text however, the checkbox need to be unchecked (as the text cursor insertion point may not be where you expect, so I made it read only). This is similar in the coding dialog - when checked you cannot code transcribed text, but the text will scroll based on timestamps in sync with the playing video.

In the av_coding dialog: Right clicking on a timestamp in the transcribed text will give another context menu option - to move to the relevant video section.

TODOs: scaler for code segments

[zackbatist]

Sorry for the delay getting back to you. Not sure how this is working on linux and windows, but on MacOS the automatic scrolling does not seem to function, and there does not seem to be a menu for jumping to the appropriate time in the video after right clicking a timestamp. However I will continue to test things out with additional settings toggled on or off and with alternative timestamp formats. I agree that scaling might be an issue for longer videos, but it is not such a major issue, I think, since this program does not rely on visual drag and drop or range-selector interfaces for identifying the beginnings and ends of segments (compared with MaxQDA, which prioritizes such interaction, which makes it easier to make mistakes by doing incomplete or misaligned swipes on a touchpad, and which is compounded by terrible and inconsistent undo functions).

Not sure if I should create new issues for these, but two additional features that would be fantastic are (1) the ability to import and export code segments with their timestamps/line numbers, segment memos, info pertaining to the media they are coded for, etc, and (2) support for unusual video aspect ratios (some of my footage comprises composites of three horizontally stacked 1280x720 videos, resulting in singular 3840x720 videos).

[ccbogel]

Interesting. In the Media Coding dialog when you right click on a time stamp it should give a context menu option of moving the video to the timestamp (see pic below). If this option is not shown in the context menu, then it is not picking up the text for the timestamp - based on the limited regex expressions that I added. So I would need a copy of the exact timestamp text to check, and maybe add additional formats in the regex expressions - even characters within the timestamp format like an extra space, or variations of the dash character, would make it fail. Note: this would affect scrolling if it could not identify timestamps. Note2: this option is only available from within the Media Coding dialog, not the Manage files dialog, when viewing videos.

I have added an extra line of code to ensure the text cursor is visible - this might make scrolling work on the Mac.

Regards additional features: I think feature (2) unusual aspect ratios may be beyond my abilities, unless the vlc display does this automatically already. I could test this with a video snippet if you want to send me one.

Feature (1) In the Reports Dialog you can export code segments with timestamps and memos for the segment - I think the media info (memo) could be added afterwards in a text or ODT or word file. Hope I am understanding your request correctly here! You can export in html (an html file with an associated html media folder), this will give timestamps etc plus the media will be positioned to the correct time in the html video tag.

Import code segments, OK, I am curious under what circumstances this could occur. It is possible, I could add an option to import from a text file that contained details of the code name, coder name, time positions and segment memos.

[zackbatist]

Re: timestamp detection, I needed to have the correct number of digits after the decimal. Perhaps the regex can count the characters leftwards from the close bracket, rather than rightward from the decimal marker. Here's a snippet of what doesn't work:

 #00:00:46.3# Participant1: Umm, to an extent blah blah....

Regarding aspect ratios, I believe video_get_aspect_ratio and video_set_aspect_ratio, which are included in the vlc python wrapper, can be used here. I can take a look at this and submit a pull request shortly.

Thanks for pointing out export features I didn't know about. I've been memoing a bit in a spreadsheet for the videos with weird aspect ratios, but I guess I could just manually add those in.

[ccbogel]

OK thanks for the feedback. The regex I have now used and tested from the timestamp you gave:

[0-9][0-9]:[0-9][0-9]:[0-9][0-9].[0-9]{1,3}

This detects 1 to 3 characters in the msecs position after the period.

[ccbogel]

One logic bug I detected when code deleting or merging, is that the relevant cid is not deleted or merged in other coding tables. There are 3 coding tables: code_text, code_av and code_image. This is now fixed with an update to the relevant modules. You can check if there are any orphan code cids that are in the code_av table which do not have a matching code in the code_name table. Run the following sqls in the sql dialog to find them: select cid from code_av where cid not in (select cid from code_name) select cid from code_image where cid not in (select cid from code_name)