x265 Chunk encoding ignores PIPE setting AND follow up of seek implementation in direct input modes

[ghost]

Context: Staxrip v2.1.7.0 Encoding with x265.

Describe the bug If you set:

• PIPE = NONE and
• CHUNKS = 2 or more

then the encoding process will use avs2pipemod (or vspipe if using VS).

Expected behavior

The encoding, in this case, must not use any pipe!

[Dendraspis]

I could reproduce it and fixed it. Would be nice if you could test it in the next beta. For me it worked.

[Dendraspis]

Damn, after having a deeper look at the result I recognized that all chunks encoded the whole file. The trimming is made by the pipe using parameters, so there is no quick fix.

[ghost]

Yes indeed, you're right, the trim is part of the pipe :( I forgot about that. The proper way to do that without the pipe is to use --seek and --frames. Well it'll take some serious new code and time... Careful, the --chunk-start and --chunk-end switches are a completely different concept. But I guess you remember that.

[ghost]

Example with pipe:

• 1st chunk: avs2pipemod64.exe -dll="path_to\avisynth.dll" -trim=0,1258 -y4mp file.avs | x265.exe --output chunk1.hevc
• 2nd chunk: avs2pipemod64.exe -dll="path_to\avisynth.dll" -trim=1259,2517 -y4mp file.avs | x265.exe --output chunk2.hevc

Same example with no-pipe

• 1st chunk: x265.exe --seek 0 --frames 1259 --output chunk1.hevc
• 2nd chunk: x265.exe --seek 1259 --frames 1259 --output chunk2.hevc

[Dendraspis]

Careful, the --chunk-start and --chunk-end switches are a completely different concept. But I guess you remember that.

Yes, I remember, but good you're bringing that back to mind. 😊

[stax76]

But do the avs and vpy reader of the x265 mod really support --seek and --frames?

[Dendraspis]

These are x265 internal params, so I don't think there should be a problem with them as long as they deliver what x265 wants to have. Of course a test wouldn't be wrong. 😁

[ghost]

Somebody's called me ?

[ghost]

SEEK is not working with avs input :-((( tried both Patman86 and Yuuki-Asuna

--seek "anything you put here" will begin at 0 :(((((

is there something wrong with my command line ? "C:\Program Files\StaxRip\Apps\Encoders\x265\x265.exe" --crf 24 --output-depth 10 --seek 200 --frames 980 --output "C:\Temp_StaxRip\Timecode sample - 25fps_temp\Chunk2_out.hevc" "C:\Temp_StaxRip\Timecode sample - 25fps_temp\Timecode sample - 25fps.avs"

The command info output thinks it's begining at 200 (as shown below)

avs+ [info]: AviSynth+ 3.6.2 (r3341, master, x86_64)
avs+ [info]: Video colorspace: YUV420 (YV12)
avs+ [info]: Video depth: 8
avs+ [info]: Video resolution: 856x480
avs+ [info]: Video framerate: 25/1
avs+ [info]: Video framecount: 1960
avs+ [info]: 856x480 fps 25/1 i420p8 frames 200 - 1179 of 1960
raw  [info]: output file: C:\Temp\_StaxRip\Timecode sample - 25fps_temp\Chunk2_out.hevc
x265 [info]: HEVC encoder version x265M 3.4+28-419182243
x265 [info]: build info [Windows][GCC 10.2.0][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main 10 profile, Level-3 (Main tier)
x265 [info]: Thread pool created using 8 threads
x265 [info]: Slices                              : 1
x265 [info]: frame threads / pool features       : 3 / wpp(8 rows)
x265 [warning]: Source height < 720p; disabling lookahead-slices
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge         : hex / 57 / 2 / 3
x265 [info]: Keyframe min / max / scenecut / bias  : 25 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt        : 20 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb       : 1 / 1 / 0
x265 [info]: References / ref-limit  cu / depth  : 3 / off / on
x265 [info]: AQ: mode / str / qg-size / cu-tree  : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress            : CRF-24.0 / 0.60
x265 [info]: tools: rd=3 psy-rd=2.00 early-skip rskip mode=1 signhide tmvp
x265 [info]: tools: b-intra strong-intra-smoothing deblock sao
x265 [info]: frame I:      4, Avg QP:22.39  kb/s: 1197.05                        351.98 KB
x265 [info]: frame P:    198, Avg QP:27.64  kb/s: 129.82
x265 [info]: frame B:    778, Avg QP:31.24  kb/s: 51.67
x265 [info]: Weighted P-Frames: Y:0.0% UV:0.0%
x265 [info]: consecutive B-frames: 1.5% 2.0% 1.5% 0.0% 95.0%

encoded 980 frames in 8.00s (122.44 fps), 72.13 kb/s, Avg QP:30.48

But the resulting video begins at real frame 0 (this is a video showing timecode, specifically for this kinds of tests)

😢 😭

Same with VPY:

D:\Desktop>"C:\Program Files\StaxRip\Apps\Encoders\x265\x265.exe" --crf 24 --output-depth 10 --seek 700 --frames 980 --output "C:\Temp\_StaxRip\TimecodeSample25FpsWithChapts_temp\Chunk2_out.hevc" "C:\Temp\_StaxRip\TimecodeSample25FpsWithChapts_temp\TimecodeSample25FpsWithChapts.vpy"
vpy  [info]: using VapourSynth Video Processing Library Core R52
vpy  [info]: Video colorspace: YUV420 (YV12)
vpy  [info]: Video depth: 8
vpy  [info]: Video resolution: 856x480
vpy  [info]: Video framerate: 25/1
vpy  [info]: Video framecount: 1960
vpy  [info]: 856x480 fps 25/1 i420p8 sar 320:321 frames 700 - 1679 of 1960
vpy  [info]: using 8 parallel requests
raw  [info]: output file: C:\Temp\_StaxRip\TimecodeSample25FpsWithChapts_temp\Chunk2_out.hevc
x265 [info]: HEVC encoder version x265M 3.4+28-419182243
x265 [info]: build info [Windows][GCC 10.2.0][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main 10 profile, Level-3 (Main tier)
x265 [info]: Thread pool created using 8 threads
x265 [info]: Slices                              : 1
x265 [info]: frame threads / pool features       : 3 / wpp(8 rows)
x265 [warning]: Source height < 720p; disabling lookahead-slices
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge         : hex / 57 / 2 / 3
x265 [info]: Keyframe min / max / scenecut / bias  : 25 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt        : 20 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb       : 1 / 1 / 0
x265 [info]: References / ref-limit  cu / depth  : 3 / off / on
x265 [info]: AQ: mode / str / qg-size / cu-tree  : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress            : CRF-24.0 / 0.60
x265 [info]: tools: rd=3 psy-rd=2.00 early-skip rskip mode=1 signhide tmvp
x265 [info]: tools: b-intra strong-intra-smoothing deblock sao
x265 [info]: frame I:      4, Avg QP:21.99  kb/s: 1179.85                        339.07 KB
x265 [info]: frame P:    199, Avg QP:27.64  kb/s: 123.23
x265 [info]: frame B:    777, Avg QP:31.24  kb/s: 50.09
x265 [info]: Weighted P-Frames: Y:0.0% UV:0.0%
x265 [info]: consecutive B-frames: 2.0% 2.0% 1.5% 0.5% 94.1%

encoded 980 frames in 7.79s (125.83 fps), 69.55 kb/s, Avg QP:30.47

The info shows that the encoder acknowledges first frame is 700, however, in the resulting video, the sequence begins at real frame 0.

CONCLUSION: The avs and vpy readers of x265 do not support --seek, but they support --frames.

[Dendraspis]

@44vince44 A very good test! 🤔 But I think that your conclusion isn't nailing it right. --seek is a x265 parameter, so it should also work when you pipe a whole movie and let x265 make the selection. 🤔

I think there is a problem with the transfer of the frame numbers, indexing or so, so that x265 thinks it starts at the right frame which is not.

But you pointed out, that we can't implement Direct Input with chunks. I'm happy you did the test before we started changing the code. 😊

[ghost]

Yes you're right @Dendraspis my wording wasn't exact. x265 takes the --seek position correctly, but something translates wrong when it is passed through avs/vpy. Yes I guess the pipe will be a requirement for chunk operations. That means that avs2pipemod64 must be kept always.

In the options UI of x265, Input/Output section. is it possible to specify on the side of the Chunk number selection, something like that "Setting Chunks 2 or more will force use of AVS2Pipe" This way, Chunk users won't think it's a bug.

[ghost]

I will ask the question to msg7086 and to @Patman86. Well actually Patman86 will read it here ;-) Any thoughts Patman86? The issue is --seek nnn that specifies to x265 at what frame it should begin encoding is not passed correctly through the avs/vpy file process.

[ghost]

@Dendraspis I guess should do nothing. Maybe soon this will be adressed. Please read here: https://github.com/msg7086/x265-Yuuki-Asuna/issues/12

[ghost]

@Dendraspis as you already know, DJATOM has implemented seeking in VPY. I have tested it an it worked.

@Patman86, DJATOM has made an important improvement in VPY input, it allows now seeking Please check his commits at https://github.com/DJATOM/x265-aMod/commits/aMod-3.4-new This should be important.

[Dendraspis]

@44vince44
Indeed - and I'm pretty sure you've tested it before I saw his post. 🤣 When x265 is updated next time by @stax76 one of us will surely adjust the code. ☺️

[Patman86]

here is my version

x265M-test_seek_avs_and_vpy.zip

[ghost]

Testing right now (Patman86's version) !!

AVS: no problems found (but found accidentally another silly issue with muxing in staxrip, reported in another thread). doing more tests with AVS, using seeking inside of ranges (trim)

[Patman86]

Testing right now (Patman86's version) !!

AVS: no problems found (but found accidentally another silly issue with muxing in staxrip, reported in another thread). doing more tests with AVS, using seeking inside of ranges (trim)

And don't forget to run the same tests with vpy seek implementation 🤣

[ghost]

Of course... how can i forget such arousing idea !!!

OK about AVS, I tested seeking on a full video, and on a few multi-ranges of videos checking exactly what happens at ranges edges. It worked accurately. AVS implementation is OK!

Now looking at VPY. @Patman86 Did you take DJATOM's implementation, or did you do it from scratch ?

[Patman86]

Did you take DJATOM's implementation, or did you do it from scratch ?

Based on the implementation of @DJATOM. There is no need to reinvent this implementation, it is included in the y4m.cpp and yuv.cpp files of x265.

[ghost]

I see! Anyway, VPY is working perfectly also. I did the same tests, using multi-range, testing especially what happens when seeking at edges. VPY: ALL IS OK!

So it is safe to begin implementing the direct input chunks. @Dendraspis, feel like do it ?

It is easy: where the AVSpipe uses syntax "trim = a,b" ; the direct AVS input will use "--seek a --frames (b-a+1)" And a and b are already calculated in the currently existing procedure. Examples: avs2pipemod64.exe -trim=0,1499 translates to x265.exe --seek 0 --frames 1500 avs2pipemod64.exe -trim=1200,1300 translates to x265.exe --seek 1200 --frames 101

AND BTW: currently, the --frames switch is already handed to the x265.exe options. So you ONLY need to translate "trim=a,b" to "-- seek a".

For VS: Where vspipe uses "--start a --end b" ; the direct VS input will uses "--seek a --frames (b-a+1)" And --frames (b-a+1) is currently, already handed to x265 parameters so only need to specify --seek a.

It should not be very hard... right ?

[JJKylee]

Good to know that we are finally equipped with a very versatile modded binary of x265. Kudos to @Patman86 and @DJATOM. 👏

[JJKylee]

BTW, there seems to be a typo in the title of this issue.

...follow up of seek implementation is direct input modes => you meant in, right? 😄

[Dendraspis]

Good to know that we are finally equipped with a very versatile modded binary of x265. Kudos to @Patman86 and @DJATOM. 👏

...and @msg7086 😉

[Dendraspis]

This one is fixed, too. 😎 I've tested AVS and VPY with 2 different sources and it worked without frameloss.

@44vince44 I expect a lot of testing to be sure there is no frame lost. 😜 Another 2 or 3 files should be more than enough. 😄

[Patman86]

Chunks works great, but when combined with --seek and --frames as parameters, it doesn't work properly.

[Dendraspis]

@Patman86 You mean via x265 settings or custom params ?

[Patman86]

NO, not custom params. The option seek and frames from x265 options (task input/output). I can still use the function, for example I have a file with 20000 frames, set seek to 5000 and frames 10000. And I want to divide this area into 6 chunks ...

[Dendraspis]

Ok ... will be fixed this night.

[JJKylee]

I can still use the function, for example I have a file with 20000 frames, set seek to 5000 and frames 10000. And I want to divide this area into 6 chunks ...

A complicated twist of --seek/--frames options, huh? 😆 Fully plausible scenario though. 😄

[Patman86]

I can still use the function, for example I have a file with 20000 frames, set seek to 5000 and frames 10000. And I want to divide this area into 6 chunks ...

A complicated twist of --seek/--frames options, huh? 😆 Fully plausible scenario though. 😄

But possible 😅😂

As the saying goes, having is better than need.

[ghost]

Wait everyone... I have been busy the last hours making some food reserves before we go for full lockdown for 11 days in my country. First let me remind you that I made tests combining trim (ranges) and seek+frames and they all worked perfectly (manual tests using cli).

I'm not sure I understand correctly what @Patman86 meant.

You have a file with 20000 frames You want to consider range 5000-10000 => this is handled in AVS by trim(5000,10000) You want to cut this to 6 equal parts (6 chunks) => Staxrip is supposed to do something like this number of frames = 10000-5000+1 = 5001 number of frames by chunk (chunks 1 to 5) = round (5001/6) = 833 number of frames last chunk (chunk 6) = number of frames - 5x(833). I should be between 833 and 838 (depending on method of rounding)

The seek implementation in avs and vpy will seek inside the range defined by trim. And this is the expected behaviour. I have checked that and it worked. I use a video showing a timecode for every frame. So every frame is a visual index. This is how I make sure that Of course i will test with 6 chunks also to make sur no frames are skipped at the end due to rounding ! But all tomorrow, please, because today I'm too tired running across the city to buy stuff among all terrified people...

EDIT: I was really expecting a new beta or hotfix would be released... ???????? I'm surprised there are not. So waiting hopefully for tomorrow's hotfix release 🤣

[Patman86]

Only StaxRip.exe @44vince44

StaxRip.zip

Here some screenshots... look at command line

[ghost]

OK ! @Patman86 : You have used seek and frames from the options AND chunks, Of course it wont work!! Don't even need to test it! I didn't think about that. Well that didn't work before, it isn't a new problem. But the problem will show differently whether you use pipe or not. It's a conceptual problem. Seek and Frames should be removed from the options, because they interfere with AVS/VPY range selection. They only lead to inconsistency. Either Seek and Frames are removed from the options, or they are left there with a warning message next to the selection box saying : "this option must not be used with AVS/VPY range selection, nor with Chunks"

Note: I will do all the testing in the morning, I'm really too tired! Thanks for the exe !

[Dendraspis]

Don't worry ... it works .... now 😁

[ghost]

What did you do ? ignore them or removed them? or ...?

[Patman86]

[Dendraspis]

Moment please, have a new bug found ... or created:

What I've done so far is make chunks, seek and frames work without and with pipe ... including some rude tests.

[JJKylee]

here is my version

x265M-test_seek_avs_and_vpy.zip

BTW, Windows Defender diagnoses this file as having Trojan:Script/Wacatac.B!ml.

I'm pretty sure it's mistaken, but Windows Defender simply deletes the x265.exe file without asking me anything. It refuses to do any operation on the file. It's my master now. ☹️

ADD: BTW, what's wrong with the new code? This kind of action was never taken by Windows Defender on Patman's previous builds. ❓

[DJATOM]

Does it UPX-packed? If yes, that's probably a reason of false detection.

[JJKylee]

@DJATOM Thanks. That was exactly the reason.

upx -d x265.exe gave me a new non-misdiagnosed exe file.

But again, this thing has never happened over the last year on so many UPX'ed exe files provided by @Patman86. Stupid Defender. ☹️

[ghost]

@JJKylee My Windows defender didn't block anything.... @Dendraspis having seek and frame allowed in the options considerably complicates things. If you manage to make them work with Chunks, then there'll be certainly issues of some kinds. And especially if you use the range selection of AVS and VPY.... I'm somehow convinced the best thing to do is to remove them. The Advanced users will always be able to type them in the custom field. Of course they wouldn't imagine combining them with chunks and with AVS/VPY range definition. Staxrip has a UI that helps you define multiple range. It is WAY more enhanced than seek and frames. Let seek and frames be only used internally for the Chunks. It has really no other meaning otherwise!

[Dendraspis]

It is heavily UPX-packed 😁

@Patman86 and @DJATOM The problem from the picture above is x265 related. It happens only with pipe usage when the --trim in AVS or --end in VPY is set and x265 does not get the --frames parameter set.

This works: vspipe.exe A:\XXX_temp\XXX_new.vpy - --y4m --start 361 --end 610 | x265.exe --crf 18 --preset placebo --frames 500 --y4m --output A:\XXX_temp\XXX_new_out_chunk2.hevc -

This doesn't work (regarding the output): vspipe.exe A:\XXX_temp\XXX_new.vpy - --y4m --start 111 --end 2610 | x265.exe --crf 18 --preset placebo --y4m --output A:\XXX_temp\XXX_new_out_chunk2.hevc -

Is this the normal/expected behaviour ?

[ghost]

The presence of --frames seems a requirement with the pipes, yes. I'm assuming so because it's there in all the syntaxes involving pipes (vs and avs)

[Dendraspis]

@44vince44 Thanks for the information. I was wondering, that the encode worked, only the output seemed to be strange. But I fixed it, eve it looks strange. 😄

I tested AVS, VPY and DirectInput with Seek, Frames and both. Cutting via script/preview is also possible and takes place before --trim or --end, so that --seekand --frames are used on the pre-selection.

If anybody wants to test: [removed]

Ok ... will be fixed this night.

So I was not lying. 😁

[Dendraspis]

@44vince44

OK ! @Patman86 : You have used seek and frames from the options AND chunks, Of course it wont work!! Don't even need to test it! I didn't think about that. Well that didn't work before, it isn't a new problem.

But it works now. 😉 A lot of brainstorming, testing and thousands of mouse click were needed to find the right solution, but I'm pretty happy with the result. 😄

But the problem will show differently whether you use pipe or not. It's a conceptual problem.

Not anymore I guess. 🤔

Seek and Frames should be removed from the options, because they interfere with AVS/VPY range selection. They only lead to inconsistency.

😱

---

@Patman86 Thanks for the screenshots and of course the GIF, which helped a lot. 😁

---

@JJKylee

I can still use the function, for example I have a file with 20000 frames, set seek to 5000 and frames 10000. And I want to divide this area into 6 chunks ...

A complicated twist of --seek/--frames options, huh? 😆 Fully plausible scenario though. 😄

🤣 🤣 🤣 🤣 Till now I don't know if it was a joke when @Patman86 said "for example I have a file with 20000 frames, [..]" ... I would believe it. 🤣

[JJKylee]

@44vince44, aren't you supposed to be sleeping now? Anyway, my Windows 10 Home version is 19042.685 with all updates patched. I have no idea why it's freaking about UPX. ☹️

@Dendraspis Believing other chunk encoding test will be done more than enough by @44vince44 and @Patman86, I tried some quirky one. To make things more complicated/interesting, I chose ffmpeg(DXVA2) as the Decoder for x265 and None(default value) for Pipe, expecting an error will pop up.

Yeah, an error pops up as was expected.

This is because the default no pipe option selects avs/vpy as input and ffmpeg(DXVA2) as the decoder tries to pipe the frames as y4m to x265 regardless of the piping method.

Here's the code that caused the error:

D:\Utilities\StaxRip\Apps\Encoders\ffmpeg\ffmpeg.exe -threads 1 -hwaccel dxva2 -i D:\Work\tmp\tst\test_1audio.mkv -f yuv4mpegpipe -pix_fmt yuv420p -strict -1 -loglevel fatal -hide_banner - | D:\Utilities\StaxRip\Apps\Encoders\x265\x265.exe --output-depth 10 --output D:\Work\tmp\tst\test_1audio_temp\test_1audio_out_chunk3.hevc D:\Work\tmp\tst\test_1audio_temp\test_1audio.avs

You see the two conflicting inputs, huh?

We need to disable the default none pipe option (and avs2pipemod as well) when the user selects ffmpeg(DXVA2) as the decoder.

Well, this may be a separate issue, but since we're focusing on Pipe and Chunk Encoding here, I guess this is a relevant place to talk about this issue as well. 😄

[JJKylee]

IMHO, implementing chunks when the input frames are fed by ffmpeg via pipe is not as easy as the other two methods: avs2pipemod/vspipe or none. We can put -ss and -t before -i in ffmpeg but unlike the other two methods, their unit is time, not frame. So we may end up with incorrect frames divided into chunks in that case.

So, I guess we'd better disable Chunks option when ffmpeg(DXVA2) is chosen as the decoder (with other Pipe methods disabled as well). Or maybe we can put a warning about it when the option is activated.

This way, we can avoid imprecise trim issue related with -ss and -t.

If you guys have any better idea, please enlighten me. 👶

[ghost]

I agree with @JJKyle, it's virtually imposible to have virtually all combinations working ! especially if there are parts working with time index, and parts working with frames !!! Some warnings must be given!

@JJKylee (antivirus issue) my version is still 19041.685 Apparently I'm still not scheduled for the october major update. That would maybe explain the difference in Windows Defender behaviour

@JJKylee (decoding with ffmeg dxva2) if this decoder forces y4m then there is something missing in the x265 options of this command

D:\Utilities\StaxRip\Apps\Encoders\ffmpeg\ffmpeg.exe -threads 1 -hwaccel dxva2 -i D:\Work\tmp\tst\test_1audio.mkv -f yuv4mpegpipe -pix_fmt yuv420p -strict -1 -loglevel fatal -hide_banner - | D:\Utilities\StaxRip\Apps\Encoders\x265\x265.exe --output-depth 10 --output D:\Work\tmp\tst\test_1audio_temp\test_1audio_out_chunk3.hevc D:\Work\tmp\tst\test_1audio_temp\test_1audio.avs

Isn't there is a missing --y4m parameter (usually set just before the --output switch) ? shouldn't it be there because of the ffmpeg pipe ? (this requires some testing 😞 )

EDIT: this appears WORKING with @Dendraspis 's latest StaxRip.v2.1.7.1M.zip. The --y4m appears: "C:\Program Files\StaxRip\Apps\Encoders\ffmpeg\ffmpeg.exe" -threads 1 -hwaccel dxva2 -i "D:\Downloads\_IDM Downloads\Timecode sample - 25fps.mp4" -f yuv4mpegpipe -pix_fmt yuv420p -strict -1 -loglevel fatal -hide_banner - | "C:\Program Files\StaxRip\Apps\Encoders\x265\x265.exe" --crf 24 --preset slow --output-depth 10 --hist-threshold 0.01 --frames 15 --y4m --output "C:\Temp\_StaxRip\Timecode sample - 25fps_temp\Timecode sample - 25fps range_out.hevc" -

@Dendraspis I am really astonished you made all these seeking and cutting work together, I'll test your exe in a few minutes. Expect me to do lots of tests indeed !

[ghost]

@Dendraspis Your exe "StaxRip v2.1.7.1M.zip" is the one I'm going to test now. Question: does it include as well official commits till 04d1898 ? "x265 muxing won't mux chunks from recent encodes" ?