search

pietrop / digital-paper-edit-electron

Work in progress - digital paper edit project - Electron, Cross Platform Desktop app - Mac, Windows, Linux
https://pietropassarelli.net/autoedit
Other
103 stars 17 forks source link

Batch mode launches simultaneous ffmpeg transcodes for all files instead of processing them sequentially #73

Open motioncircus opened 3 years ago

motioncircus commented 3 years ago

Describe the bug A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

  1. Go to New Batch Transcripts
  2. Click on the above
  3. Add multiple source files
  4. Got to Task Manager and look at the amount of ffmpeg instances competing for your CPU and RAM

Expected behavior Sequential processing of ONE or A FEW files at a time, not 35 like I saw yesterday

AutoEdit 3-1.4.0

Desktop (please complete the following information):

pietrop commented 3 years ago

Hi @motioncircus, Thanks for flagging this.

The batch import it's somewhat of an experimental feature. The blocker I have with this one, is that I need to figure out a way to stagger or queue the requests in javascript/node/electron.

For context, It was initially introduced here https://github.com/pietrop/digital-paper-edit-client/pull/16

Looking at the code. As it is now, the problem is mostly on the client. where it takes the list of files and pietrop/digital-paper-edit-client/src/Components/Transcripts/BatchTranscriptForm.js#L98 and call pietrop/digital-paper-edit-client/src/Components/Transcripts/BatchTranscriptForm.js#L110 using a forEach loop.

So I guess if I could find a good way to add a queue, I could add it at that stage.

One solution could be to use a for loop and async await as described here on stackoverflow - Using async/await with a forEach loop

motioncircus commented 3 years ago

Hi Pietro,

It's not a huge deal. I can add several files at a time using the single transcribe button. That way I can limit the current ffmpeg processes to half a dozen.

This project I'm currently starting has hundreds of interviews so I'm facing a new set of issues.

Here's a feature request though:

Batch download transcription files for a whole 'project', instead of having to click on each transcription and choose 'export as'

Cheers

Nigel Haslam | Director

Motion Circus Pty Ltd. Byron Bay

pietrop commented 3 years ago

ah, interesting, what file format are you using for the transcription? (docx, txt etc..)

motioncircus commented 3 years ago

I've been going for .docx with titles and timecodes myself.

N PS. Am I right in thinking there's a way to search across multiple transcriptions? I need to compile a teaser with as many examples of a specific phrase as I can find in these hundreds of interviews.

I could do it by merging (or copying and pasting), all the text files in Google but it would take half a day

N

Nigel Haslam | Director

Motion Circus Pty Ltd. Byron Bay t: + 61 2 8007 7338 |m: + 61 403 020 126 w: www.motioncircus.com http://www.motioncircus.com/ link to website http://motioncircus.com

On Mon, May 3, 2021 at 10:05 AM Pietro @.***> wrote:

ah, interesting, what file format are you using for the transcription? (docx, txt etc..)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pietrop/digital-paper-edit-electron/issues/73#issuecomment-830932981, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCDOE5LXZNZV5YJUYB5TSLTLXSERANCNFSM4364XS4A .

pietrop commented 3 years ago

Ok, yeah so this is what I was wondering, do you use the paper-editing view?

See this section of the user manual for how to search across transcripts in a project

And this for more broadly about the paper editing functionalities section

motioncircus commented 3 years ago

I do, of course, it's your genius child!

I just forgot where it was.

Hey did you notice that REVS are beta trialling an online paper edit tool?

I gave it a look over a couple of months ago. It's not as good as AutoEdit but it looks almost identical. I wouldn't be surprised if they are using your code.

Nigel

Nigel Haslam | Director

Motion Circus Pty Ltd. Byron Bay t: + 61 2 8007 7338 |m: + 61 403 020 126 w: www.motioncircus.com http://www.motioncircus.com/ link to website http://motioncircus.com

On Mon, May 3, 2021 at 10:25 AM Pietro @.***> wrote:

Ok, yeah so this is what I was wondering, do you use the paper-editing view?

See this section of the user manual for how to search across transcripts in a project https://autoedit.gitbook.io/autoedit-3-user-manual/paperediting/search/search-across-transcripts-in-a-project

And this for more broadly about the paper editing functionalities section https://autoedit.gitbook.io/autoedit-3-user-manual/paperediting

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pietrop/digital-paper-edit-electron/issues/73#issuecomment-830965167, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCDOE7RV37DFUB6GZ72XU3TLXUPTANCNFSM4364XS4A .

pietrop commented 3 years ago

ah, no, I had not seen that, do you have a link? I looked through rev.com but can't find it

motioncircus commented 3 years ago

Here's the email they sent me and I signed up for the webinar,

I can't find a link to the actual app.. maybe I dismissed it during the webinar. Sorry, I didn't think at the time.

[image: product images] https://enterprise.rev.com/e/756353/inar-utm-content-webinar-promo/f61zb/335555273?h=D_MhUIEZFdbHYapW_om8qu_YQUlacUu13Ig2A8Oit6g

Rev Will Help You Tell Better Stories Faster

Webinar Tuesday, May 11, 2021 | 1 p.m. CT

Editing films, tv shows, and videos is often a clunky, time-consuming process. Editors spend hours sifting through footage just to find the best takes and soundbites. Sound familiar? If it does, you’re definitely not alone.

To avoid those headaches and speed up the editing process, many post-production teams first convert their video footage into text. Then, they take advantage of script-based editing workflows to cut down editing time and boost collaboration.

Here’s the good news: Rev can help you harness the power of script-based editing with new features that we’ll be covering during our webinar on May 11, 2021 at 1 p.m. CT. Join us for an in-depth discussion where we’ll cover:

There will be plenty of time for questions after the session, so reserve your seat today. This is one you don’t want to miss! REGISTER NOW https://enterprise.rev.com/e/756353/inar-utm-content-webinar-promo/f61zb/335555273?h=D_MhUIEZFdbHYapW_om8qu_YQUlacUu13Ig2A8Oit6g

Call Us: 888-369-0701 Email Us: @.** Send Us Pizza: 1717 W. 6th St. Suite 310, Austin, TX 78703 Wanna go? Hey, totally cool! Unsubscribe by clicking here. https://enterprise.rev.com/unsubscribe/u/756353/92f9ec0b648f3e8fc45bdee3b7f27591f799f20f1504c919b227330829fcd630/335555273* [image: Twitter] https://enterprise.rev.com/e/756353/rev/f61zd/335555273?h=D_MhUIEZFdbHYapW_om8qu_YQUlacUu13Ig2A8Oit6g [image: Facebook] https://enterprise.rev.com/e/756353/rev/f61zg/335555273?h=D_MhUIEZFdbHYapW_om8qu_YQUlacUu13Ig2A8Oit6g [image: LinkedIn] https://enterprise.rev.com/e/756353/company-rev-com-/f61zj/335555273?h=D_MhUIEZFdbHYapW_om8qu_YQUlacUu13Ig2A8Oit6g [image: YouTube] https://enterprise.rev.com/e/756353/c-Revdotcom/f61zl/335555273?h=D_MhUIEZFdbHYapW_om8qu_YQUlacUu13Ig2A8Oit6g

To ensure delivery to your inbox, add @.*** to your address book.

Nigel Haslam | Director

Motion Circus Pty Ltd. Byron Bay t: + 61 2 8007 7338 |m: + 61 403 020 126 w: www.motioncircus.com http://www.motioncircus.com/ link to website http://motioncircus.com

On Mon, May 3, 2021 at 11:30 AM Pietro @.***> wrote:

ah, no, I had not seen that, do you have a link? I looked through rev.com but can't find it

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pietrop/digital-paper-edit-electron/issues/73#issuecomment-830981100, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCDOE3FAROGOOAGHXDP2LLTLX4EPANCNFSM4364XS4A .

motioncircus commented 3 years ago

Hey

I googled up a link

https://www.rev.com/blog/rev-transcript-editor-guide

Nigel Haslam | Director

Motion Circus Pty Ltd. Byron Bay t: + 61 2 8007 7338 |m: + 61 403 020 126 w: www.motioncircus.com http://www.motioncircus.com/ link to website http://motioncircus.com

On Mon, May 3, 2021 at 1:49 PM Nigel Haslam @.***> wrote:

Here's the email they sent me and I signed up for the webinar,

I can't find a link to the actual app.. maybe I dismissed it during the webinar. Sorry, I didn't think at the time.

[image: product images] https://enterprise.rev.com/e/756353/inar-utm-content-webinar-promo/f61zb/335555273?h=D_MhUIEZFdbHYapW_om8qu_YQUlacUu13Ig2A8Oit6g

Rev Will Help You Tell Better Stories Faster

Webinar Tuesday, May 11, 2021 | 1 p.m. CT

Editing films, tv shows, and videos is often a clunky, time-consuming process. Editors spend hours sifting through footage just to find the best takes and soundbites. Sound familiar? If it does, you’re definitely not alone.

To avoid those headaches and speed up the editing process, many post-production teams first convert their video footage into text. Then, they take advantage of script-based editing workflows to cut down editing time and boost collaboration.

Here’s the good news: Rev can help you harness the power of script-based editing with new features that we’ll be covering during our webinar on May 11, 2021 at 1 p.m. CT. Join us for an in-depth discussion where we’ll cover:

  • Different use cases for script-based editing tools
  • How you can save time using transcripts to develop your storyline
  • Ways to bring your storytellers into the editing room

There will be plenty of time for questions after the session, so reserve your seat today. This is one you don’t want to miss! REGISTER NOW https://enterprise.rev.com/e/756353/inar-utm-content-webinar-promo/f61zb/335555273?h=D_MhUIEZFdbHYapW_om8qu_YQUlacUu13Ig2A8Oit6g

Call Us: 888-369-0701 Email Us: @.** Send Us Pizza: 1717 W. 6th St. Suite 310, Austin, TX 78703 Wanna go? Hey, totally cool! Unsubscribe by clicking here. https://enterprise.rev.com/unsubscribe/u/756353/92f9ec0b648f3e8fc45bdee3b7f27591f799f20f1504c919b227330829fcd630/335555273* [image: Twitter] https://enterprise.rev.com/e/756353/rev/f61zd/335555273?h=D_MhUIEZFdbHYapW_om8qu_YQUlacUu13Ig2A8Oit6g [image: Facebook] https://enterprise.rev.com/e/756353/rev/f61zg/335555273?h=D_MhUIEZFdbHYapW_om8qu_YQUlacUu13Ig2A8Oit6g [image: LinkedIn] https://enterprise.rev.com/e/756353/company-rev-com-/f61zj/335555273?h=D_MhUIEZFdbHYapW_om8qu_YQUlacUu13Ig2A8Oit6g [image: YouTube] https://enterprise.rev.com/e/756353/c-Revdotcom/f61zl/335555273?h=D_MhUIEZFdbHYapW_om8qu_YQUlacUu13Ig2A8Oit6g

To ensure delivery to your inbox, add @.*** to your address book.

Nigel Haslam | Director

Motion Circus Pty Ltd. Byron Bay t: + 61 2 8007 7338 |m: + 61 403 020 126 w: www.motioncircus.com http://www.motioncircus.com/ link to website http://motioncircus.com

On Mon, May 3, 2021 at 11:30 AM Pietro @.***> wrote:

ah, no, I had not seen that, do you have a link? I looked through rev.com but can't find it

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pietrop/digital-paper-edit-electron/issues/73#issuecomment-830981100, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCDOE3FAROGOOAGHXDP2LLTLX4EPANCNFSM4364XS4A .

motioncircus commented 3 years ago

Hi Pietro,

I now have transcribed a hundred of the three hundred interviews that I need to edit, but I'm finding the interface quite laggy and the video playback unreliable and the sound strangely reverberant, as if more than one file is playing back simultaneously but ever so slightly out of sync.

I'm running the WIN 10 version at the moment but thinking I might get better results if I migrate to the OS that you use yourself. I have access to a Macbook pro with Parallels on it so could dual boot into whatever you suggest,

Cheers Nigel

Nigel Haslam | Director

Motion Circus Pty Ltd. Byron Bay t: + 61 2 8007 7338 |m: + 61 403 020 126 w: www.motioncircus.com http://www.motioncircus.com/ link to website http://motioncircus.com

On Mon, May 3, 2021 at 1:51 PM Nigel Haslam @.***> wrote:

Hey

I googled up a link

https://www.rev.com/blog/rev-transcript-editor-guide

Nigel Haslam | Director

Motion Circus Pty Ltd. Byron Bay t: + 61 2 8007 7338 |m: + 61 403 020 126 w: www.motioncircus.com http://www.motioncircus.com/ link to website http://motioncircus.com

On Mon, May 3, 2021 at 1:49 PM Nigel Haslam @.***> wrote:

Here's the email they sent me and I signed up for the webinar,

I can't find a link to the actual app.. maybe I dismissed it during the webinar. Sorry, I didn't think at the time.

[image: product images] https://enterprise.rev.com/e/756353/inar-utm-content-webinar-promo/f61zb/335555273?h=D_MhUIEZFdbHYapW_om8qu_YQUlacUu13Ig2A8Oit6g

Rev Will Help You Tell Better Stories Faster

Webinar Tuesday, May 11, 2021 | 1 p.m. CT

Editing films, tv shows, and videos is often a clunky, time-consuming process. Editors spend hours sifting through footage just to find the best takes and soundbites. Sound familiar? If it does, you’re definitely not alone.

To avoid those headaches and speed up the editing process, many post-production teams first convert their video footage into text. Then, they take advantage of script-based editing workflows to cut down editing time and boost collaboration.

Here’s the good news: Rev can help you harness the power of script-based editing with new features that we’ll be covering during our webinar on May 11, 2021 at 1 p.m. CT. Join us for an in-depth discussion where we’ll cover:

  • Different use cases for script-based editing tools
  • How you can save time using transcripts to develop your storyline
  • Ways to bring your storytellers into the editing room

There will be plenty of time for questions after the session, so reserve your seat today. This is one you don’t want to miss! REGISTER NOW https://enterprise.rev.com/e/756353/inar-utm-content-webinar-promo/f61zb/335555273?h=D_MhUIEZFdbHYapW_om8qu_YQUlacUu13Ig2A8Oit6g

Call Us: 888-369-0701 Email Us: @.** Send Us Pizza: 1717 W. 6th St. Suite 310, Austin, TX 78703 Wanna go? Hey, totally cool! Unsubscribe by clicking here. https://enterprise.rev.com/unsubscribe/u/756353/92f9ec0b648f3e8fc45bdee3b7f27591f799f20f1504c919b227330829fcd630/335555273* [image: Twitter] https://enterprise.rev.com/e/756353/rev/f61zd/335555273?h=D_MhUIEZFdbHYapW_om8qu_YQUlacUu13Ig2A8Oit6g [image: Facebook] https://enterprise.rev.com/e/756353/rev/f61zg/335555273?h=D_MhUIEZFdbHYapW_om8qu_YQUlacUu13Ig2A8Oit6g [image: LinkedIn] https://enterprise.rev.com/e/756353/company-rev-com-/f61zj/335555273?h=D_MhUIEZFdbHYapW_om8qu_YQUlacUu13Ig2A8Oit6g [image: YouTube] https://enterprise.rev.com/e/756353/c-Revdotcom/f61zl/335555273?h=D_MhUIEZFdbHYapW_om8qu_YQUlacUu13Ig2A8Oit6g

To ensure delivery to your inbox, add @.*** to your address book.

Nigel Haslam | Director

Motion Circus Pty Ltd. Byron Bay t: + 61 2 8007 7338 |m: + 61 403 020 126 w: www.motioncircus.com http://www.motioncircus.com/ link to website http://motioncircus.com

On Mon, May 3, 2021 at 11:30 AM Pietro @.***> wrote:

ah, no, I had not seen that, do you have a link? I looked through rev.com but can't find it

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pietrop/digital-paper-edit-electron/issues/73#issuecomment-830981100, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCDOE3FAROGOOAGHXDP2LLTLX4EPANCNFSM4364XS4A .

pietrop commented 3 years ago

Hi Nigel, Sorry to hear that.

If you have a mac, then you could just use the mac version of the app (which is the one I use/primarily develop for)

There's a 1.6.0 release coming up shortly in the release section

(More info on installing the Mac OSX version in the user manual, just in case)

This issue tho

the interface quite laggy and the video playback unreliable and the sound strangely reverberant, as if more than one file is playing back simultaneously but ever so slightly out of sync.

sounds like an issue with the video playback part of the programme script in the paper edit I raised it as a separate issue to look into it separately. https://github.com/pietrop/digital-paper-edit-client/issues/57 I suspect it's a problem with the underlying BBC Video context component which unfortunately is not very actively maintained at the moment. So I'll have to have a think about what could be an effective solution for this.

For now, as far as you could tell, was there a sequence of steps that seemed to be mostly responsible for when this issue of the reverberation comes up?

motioncircus commented 3 years ago

Hey Pietro,

I've discovered that the mac version 1.6.0 works SO much better. It starts immediately and batch encodes ten files at a time with ease as well as playing back seamlessly. I admit that I've adopted a Handbrake proxy workflow now, because hundreds of 4K source interviews were taking too much space on my C drive, which I normally only use for applications.

It would be great if one could, optionally, choose one's own media folder for AutoEdit's proxy files. I have two 500G Samsung M.2 SSD's that I use for fast playback of video files and I'd love to be able to use those with AutoEdit.

For now, I'm just going to try and edit on Mac for this job.

N

Nigel Haslam | Director

Motion Circus Pty Ltd. Byron Bay t: + 61 2 8007 7338 |m: + 61 403 020 126 w: www.motioncircus.com http://www.motioncircus.com/ link to website http://motioncircus.com

On Thu, May 20, 2021 at 5:51 AM Pietro @.***> wrote:

Hi Nigel, Sorry to hear that.

If you have a mac, then you could just use the mac version of the app (which is the one I use/primarily develop for)

There's a 1.6.0 release coming up shortly in the release section https://github.com/pietrop/digital-paper-edit-electron/releases

(More info on installing the Mac OSX version in the user manual, just in case https://autoedit.gitbook.io/autoedit-3-user-manual/installing/installing-on-mac-osx )

This issue tho

the interface quite laggy and the video playback unreliable and the sound strangely reverberant, as if more than one file is playing back simultaneously but ever so slightly out of sync.

sounds like an issue with the video playback part of the programme script in the paper edit https://autoedit.gitbook.io/autoedit-3-user-manual/paperediting/preview I raised it as a separate issue to look into it separately. pietrop/digital-paper-edit-client#57 https://github.com/pietrop/digital-paper-edit-client/issues/57 I suspect it's a problem with the underlying BBC Video context component https://github.com/bbc/VideoContext which unfortunately is not very actively maintained at the moment. So I'll have to have a think about what could be an effective solution for this.

For now, as far as you could tell, was there a sequence of steps that seemed to be mostly responsible for when this issue of the reverberation comes up?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pietrop/digital-paper-edit-electron/issues/73#issuecomment-844420202, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCDOE2ZNBY5UKPJNTF6FJDTOQJEPANCNFSM4364XS4A .