fatal error when lots of ffmpeg process doing

naffan2014 commented 3 years ago

i do lots of cut work and concat work and transcode work. every work source dir and filepath is different codes catchs lots of error. "java.io.IOException: Cannot run program "/data0/www/cache/jave/ffmpeg-amd64-2.7.3": error=26, Text file busy" why is text file busy? how can i solve it?

a-schild commented 3 years ago

What OS are you using?
What JRE are you using?
Does it fails always, only at random 10% or at random 90% of the time?
Are you running multiple concurrent encodings?

Perhaps something among this line? https://bugs.openjdk.java.net/browse/JDK-8068370

naffan2014 commented 3 years ago

docker : centos x86_64
jre : java 1.8.0_232
not always, it ouccrs 10% but when thread processing at a time, the num will increase to 30%
of course yes. i control concat mission in 3 corepollsize and 8 queueCapacity. and same as cut . so theoretically ffmpeg thread will run 6 thread at one time.

i check the articel you give me . i will change open() param FD_CLOEXEC to O_CLOEXEC. and i will observe it .

a-schild commented 3 years ago

Are all the threads running in the same JVM, or are they running in different JVM's?

If the all run in the same JVM, then there is a race condition (Missing exclusive section) between the moment we check if the executable is available in the temp location and when it's completely written to disk and flagged as executable.
If they run in different JVM's, then you need to put the TEMP location to different folders to prevent this problem

naffan2014 commented 3 years ago

Are all the threads running in the same JVM, or are they running in different JVM's?

If the all run in the same JVM, then there is a race condition (Missing exclusive section) between the moment we check if the executable is available in the temp location and when it's completely written to disk and flagged as executable.

If they run in different JVM's, then you need to put the TEMP location to different folders to prevent this problem

in one jvm

could u describe more about why it occurs race condition? thank you.

naffan2014 commented 3 years ago

this is my database persist ffmpeg working result. the column detail tells me exception reason.

<!DOCTYPE html>
<html>
<head>
<title></title>
</head>
<body>

create_time | detail | state | log_task_id
-- | -- | -- | --
2021-09-03 17:31:00 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | EXCEPTION | 39
2021-09-03 17:31:01 |   | PROCESSING | 39
2021-09-03 17:31:01 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | EXCEPTION | 39
2021-09-03 17:31:01 |   | PROCESSING | 39
2021-09-03 17:31:01 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | EXCEPTION | 39
2021-09-03 17:31:04 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | EXCEPTION | 39
2021-09-03 17:31:04 |   | PROCESSING | 39
2021-09-03 17:32:32 |   | PROCESSING | 39
2021-09-03 17:32:32 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | FAILURE | 39
2021-09-03 17:32:32 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | FAILURE | 39
2021-09-03 17:32:33 |   | PROCESSING | 39
2021-09-03 17:32:33 |   | PROCESSING | 39
2021-09-03 17:32:33 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | FAILURE | 39
2021-09-03 17:32:33 |   | PROCESSING | 39
2021-09-03 17:32:33 | ws.schild.jave.EncoderException\|Exit code of ffmpeg encoding run is 1 | FAILURE | 39
2021-09-03 17:35:46 | Invalid data found when processing input | EXCEPTION | 45
2021-09-03 17:35:46 |   | PROCESSING | 45
2021-09-03 17:35:48 |   | PROCESSING | 45
2021-09-03 17:36:27 | Invalid data found when processing input | EXCEPTION | 45
2021-09-03 17:39:03 |   | PROCESSING | 45
2021-09-03 17:39:03 | Invalid data found when processing input | FAILURE | 45
2021-09-03 17:39:04 |   | CREATED | 46
2021-09-03 17:50:32 |   | DONE | 46
2021-09-03 17:50:32 |   | DONE | 46
2021-09-03 17:50:32 |   | PROCESSING | 46
2021-09-03 17:50:32 |   | PROCESSING | 46
2021-09-03 17:50:56 |   | CREATED | 47
2021-09-03 17:54:59 |   | CREATED | 48
2021-09-03 17:58:01 |   | PROCESSING | 46
2021-09-03 17:58:01 | java.io.IOException: Cannot run program "/data0/www/cache/jave/ffmpeg-amd64-2.7.3": error=26, Text file busy | EXCEPTION | 46
2021-09-03 17:58:01 |   | PROCESSING | 46
2021-09-03 17:58:01 | java.io.IOException: Cannot run program "/data0/www/cache/jave/ffmpeg-amd64-2.7.3": error=26, Text file busy | EXCEPTION | 46

</body>
</html>

there are three main reason in this sheet.

ws.schild.jave.EncoderException|Exit code of ffmpeg encoding run is 1
Invalid data found when processing input
java.io.IOException: Cannot run program "/data0/www/cache/jave/ffmpeg-amd64-2.7.3": error=26, Text file busy

naffan2014 commented 3 years ago

i see another issue https://github.com/a-schild/jave2/issues/157. u suggest control thread through attrs.setEncodingThreads(x)。is it control ffmpeg thread number ?

a-schild commented 3 years ago

Could you please try again with the 3.2.0-SNAPSHOT release?

I did put in place a fix for the race condition. https://github.com/a-schild/jave2/commit/5cec8b1faf85885a5d19dfda2825c3db98025b61

The ffmpeg executables are packed inside the jar files for the platforms. When jave starts, it checks if the ffmpeg executable is found in the temp folder, if not, then it copies it from the jar file to the temp location. If you now have two threads doing the check if the file exists in the same millisecond, then both threads will write a copy of the executable to the same location. The first one to start the executable will then probably try to start the still "copy in progress" file of the second thread, leading to the erro 26, file busy

a-schild commented 3 years ago

Issue #157 tells ffmpeg to only use a certain number of threads/cores for the encoding process, but this has nothing to do with running multiple java threads which spawn the executable. (Beside using cores/cpus)

a-schild commented 3 years ago

The exit code 1 however is something else, most of the time some corrupt input file or some unsupported encoding/transcoding formats/settings To see the real cause, you have to enable logging and collect the logs

naffan2014 commented 3 years ago

Could you please try again with the 3.2.0-SNAPSHOT release?

I did put in place a fix for the race condition. 5cec8b1

The ffmpeg executables are packed inside the jar files for the platforms. When jave starts, it checks if the ffmpeg executable is found in the temp folder, if not, then it copies it from the jar file to the temp location. If you now have two threads doing the check if the file exists in the same millisecond, then both threads will write a copy of the executable to the same location. The first one to start the executable will then probably try to start the still "copy in progress" file of the second thread, leading to the erro 26, file busy

because some reason i cannot use above 2.7.3. i really want to use 3.x but i can't. mayby later i will make somebody to fix out maven repository bug ~

naffan2014 commented 3 years ago

i fixed the maven bug. so i can use 3.1.1 now .

let me first rebuild my code to run jave 3.1.1 . and i will send my result here or close this issue

naffan2014 commented 3 years ago

i can get 3.1.1 but i can't get 3.2.0-SNAPSHOT :(

a-schild commented 3 years ago

Add this in your pom.xml

<repositories>
    <repository>
      <id>oss.sonatype.org-snapshot</id>
      <url>https://oss.sonatype.org/content/repositories/snapshots</url>
      <releases>
        <enabled>false</enabled>
      </releases>
      <snapshots>
        <enabled>true</enabled>
      </snapshots>
    </repository>
  </repositories>

naffan2014 commented 3 years ago

Add this in your pom.xml

<repositories>
   <repository>
     <id>oss.sonatype.org-snapshot</id>
     <url>https://oss.sonatype.org/content/repositories/snapshots</url>
     <releases>
       <enabled>false</enabled>
     </releases>
     <snapshots>
       <enabled>true</enabled>
     </snapshots>
   </repository>
 </repositories>

sorry

Add this in your pom.xml

<repositories>
   <repository>
     <id>oss.sonatype.org-snapshot</id>
     <url>https://oss.sonatype.org/content/repositories/snapshots</url>
     <releases>
       <enabled>false</enabled>
     </releases>
     <snapshots>
       <enabled>true</enabled>
     </snapshots>
   </repository>
 </repositories>

i use gradle ,not maven~ do u know how to add these in gradle ?

a-schild commented 3 years ago

Not sure, but probably something like this:

https://docs.gradle.org/current/userguide/declaring_repositories.html#sec:repository-content-filtering ?

naffan2014 commented 3 years ago

Not sure, but probably something like this:

https://docs.gradle.org/current/userguide/declaring_repositories.html#sec:repository-content-filtering ?

thank u . i can get 3.2.0-SNAPSHOT now

naffan2014 commented 3 years ago

when i use 3.2.0-snapshot . i can;t do the concat mp4 work again. issue is #166

a-schild commented 3 years ago

@naffan2014 Can we close this issue and continue on #166 ?

naffan2014 commented 3 years ago

@naffan2014 Can we close this issue and continue on #166 ?

yes。i close this issue at once.

a-schild / jave2

fatal error when lots of ffmpeg process doing #163