search

jackun / TestAMFVFW

15 stars 4 forks source link

crashing in VirtualDub and Magix MEP #12

Closed petermg closed 8 years ago

petermg commented 8 years ago

`VirtualDub crash report -- build 35491 (release)

Disassembly: 62c246e0: 243a and al, 3ah 62c246e2: 8b7dfc mov edi, [ebp-04h] 62c246e5: 660f72d310 psrld xmm3, 10h 62c246ea: 660f73d806 psrldq xmm0, 06h 62c246ef: 660ffd45b0 paddw xmm0, [ebp-50h] 62c246f4: 660f7e0c3e movd [esi+edi], xmm1 62c246f9: 0f28cf movaps xmm1, xmm7 62c246fc: 660f73f30e psllq xmm3, 0eh 62c24701: 660f73d906 psrldq xmm1, 06h 62c24706: 660ffada psubd xmm3, xmm2 62c2470a: 660ffdcf paddw xmm1, xmm7 62c2470e: 660f72d309 psrld xmm3, 09h 62c24713: 660f6cc8 punpcklqdq xmm1, xmm0 62c24717: 660ff51d602cc3 pmaddwd xmm3, [62c32c60] 62
62c2471f: 660f72d110 psrld xmm1, 10h 62c24724: 660f73f10e psllq xmm1, 0eh 62c24729: 660ffe1db02cc3 paddd xmm3, [62c32cb0] 62
62c24731: 660ffacd psubd xmm1, xmm5 62c24735: 8345f40c add dword ptr [ebp-0ch], 0ch 62c24739: 8bcf mov ecx, edi 62c2473b: 0f2825802cc362 movaps xmm4, [62c32c80] 62c24742: 83c104 add ecx, 04h 62c24745: 0f282da02cc362 movaps xmm5, [62c32ca0] 62c2474c: 0f57c0 xorps xmm0, xmm0 62c2474f: 8b7ddc mov edi, [ebp-24h] 62c24752: 660f72d109 psrld xmm1, 09h 62c24757: 660f67db packuswb xmm3, xmm3 62c2475b: 660ff50d602cc3 pmaddwd xmm1, [62c32c60] 62
62c24763: 894dfc mov [ebp-04h], ecx 62c24766: 660ffe0db02cc3 paddd xmm1, [62c32cb0] 62
62c2476e: 660f67c9 packuswb xmm1, xmm1 62c24772: 660fe0d9 pavgb xmm3, xmm1 62c24776: 660f71d308 psrlw xmm3, 08h 62c2477b: 660f67db packuswb xmm3, xmm3 62c2477f: 660f db 0fh <-- FAULT 62c24781: 3a16 cmp dl, [esi] 62c24783: 1a00 sbb al, [eax] 62c24785: 8b55e0 mov edx, [ebp-20h] 62c24788: 83c20c add edx, 0ch 62c2478b: 8955e0 mov [ebp-20h], edx 62c2478e: 3b4b08 cmp ecx, [ebx+08h] 62c24791: 0f822bfeffff jc 62c245c2 62c24797: 0f2825802cc362 movaps xmm4, [62c32c80] 62c2479e: 0f282da02cc362 movaps xmm5, [62c32ca0] 62c247a5: 8b75d8 mov esi, [ebp-28h] 62c247a8: 8b7df0 mov edi, [ebp-10h] 62c247ab: 8b4dec mov ecx, [ebp-14h] 62c247ae: 8b530c mov edx, [ebx+0ch] 62c247b1: 8b45d0 mov eax, [ebp-30h] 62c247b4: 83c602 add esi, 02h 62c247b7: 037dd4 add edi, [ebp-2ch] 62c247ba: 034dd4 add ecx, [ebp-2ch] 62c247bd: 0145e8 add [ebp-18h], eax 62c247c0: 0145e4 add [ebp-1ch], eax 62c247c3: 8975d8 mov [ebp-28h], esi 62c247c6: 897df0 mov [ebp-10h], edi 62c247c9: 894dec mov [ebp-14h], ecx 62c247cc: 3bf2 cmp esi, edx 62c247ce: 0f82b1fdffff jc 62c24585 62c247d4: 5f pop edi 62c247d5: 5e pop esi 62c247d6: 8be5 mov esp, ebp 62c247d8: 5d pop ebp 62c247d9: 8be3 mov esp, ebx 62c247db: 5b pop ebx 62c247dc: c3 ret
62c247dd: cc int 3 62c247de: cc int 3 62c247df: cc int 3

Built on Althena on Sun Oct 27 15:59:07 2013 using compiler version 1400

Windows 6.3 (Windows 7 x64 build 9600) [] Memory status: virtual free 1448M/2048M, commit limit 32761M, physical total 16377M

EAX = 06ac1b00 EBX = 0ffdfa8c ECX = 00000004 EDX = 06ac1b00 EBP = 0ffdfa80 ESI = 06a2b600 EDI = fff6a000 ESP = 0ffdfa20 EIP = 62c2477f EFLAGS = 00010202 FPUCW = 027f FPUTW = ffff

Crash reason: Illegal instruction

Crash context: An instruction not supported by the CPU was executed in module 'TestAMFVFW'...

...while compressing frame 0 from 1877a020 to 17cbc020 using codec "TestAMFVFW Encoder" (w32videocodecpack.cpp:787)...

...while running thread "Processing" (thread.cpp:197).

Pointer dumps:

EAX 06ac1b00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 EBX 0ffdfa88: 095150cc 08c887d4 62c23e7e 00000500 000003c0 00000500 62c32220 0b3cfd80 EDX 06ac1b00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ESI 06a2b600: 10101010 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ESP 0ffdfa20: 0b3cfd80 0b3cfd8c 68f9f8d4 5067b3ab 00000000 00000000 00000000 00000000 0ffdfa40: 00000000 00000000 00000000 00000000 00001e00 fffff600 00000000 fff6a000 0ffdfa60: 1877af20 1877a020 1877af20 06a2b600 06a2bb00 1877a02c 06900000 00000004 0ffdfa80: 0ffdfb00 62c23e7e 095150cc 08c887d4 62c23e7e 00000500 000003c0 00000500 EBP 0ffdfa80: 0ffdfb00 62c23e7e 095150cc 08c887d4 62c23e7e 00000500 000003c0 00000500 0ffdfaa0: 62c32220 0b3cfd80 0ffdfbb8 ffff01fe 000003c0 00000500 1877a020 09e1fa50 0ffdfac0: 095150cc 06900000 00000500 001c2000 00000500 000003c0 00000001 00000001 0ffdfae0: 00000067 00000001 00000000 00000003 00000000 00010000 00000000 5bb74da8

Thread call stack: 62c2477f: TestAMFVFW!0000477f 68f9f8d4: d3d11!D3D11CoreCreateLayeredDevice [68f50000+3ab00+14dd4] 62c23e7e: TestAMFVFW!00003e7e 62c23e7e: TestAMFVFW!00003e7e 62c23548: TestAMFVFW!00003548 62c2c525: TestAMFVFW!DriverProc [62c20000+c430+f5] 62d12da3: MSVFW32!ICSendMessage [62d10000+2d60+43] 62d13193: MSVFW32!ICCompress [62d10000+3130+63] 005da2ee: VDVideoCompressorVCM::PackFrameInternal() 005dad85: VDVideoCompressorVCM::CompressFrame() 00570df3: VDPixmapGenResampleCol_d2_pnqrtr_lin_u8_ISSE::Compute() 0048c99b: VDThreadedVideoCompressor::ProcessFrame() 0048ceed: VDThreadedVideoCompressor::ExchangeBuffer() 00424353: VDDubVideoProcessor::WriteFinishedVideoFrame() 00424dec: VDDubVideoProcessor::ProcessVideoFrame() 004250ca: VDDubVideoProcessor::RunPathWriteOutputFrames() 00425cdf: VDDubVideoProcessor::WriteVideo() 00422d3e: VDDubProcessThread::ThreadRun() 004f9917: ?$VDProtectedAutoScope1::?$VDProtectedAutoScope1() 0051a2dc: VDThread::StaticThreadStart() 005fcc0e: _callthreadstartex() 005fccb3: _threadstartex@4() 749f38f4: KERNEL32!BaseThreadInitThunk [749e0000+138d0+24] 77105de3: ntdll!RtlUnicodeStringToInteger [770a0000+65b90+253] 77105dae: ntdll!RtlUnicodeStringToInteger [770a0000+65b90+21e]

-- End of report `

Tried different encoding settings, can't seem to get past this.

jackun commented 8 years ago

Ah, there is SSE4.1/AVX instruction pextrd for some reason though it should limit to SSE2, hm. What CPU you have?

E: Oh crap, _mm_extract_epi32 is SSE4. Set the converter type to OpenCL, should bypass CPU converter code and is faster too or set color depth to NV12 directly from virtualdub.

petermg commented 8 years ago

I have an AMD Phenom 1055T CPU.

On Jul 31, 2016 11:15 PM, "jackun" notifications@github.com wrote:

Ah, there is SSE4.1/AVX instruction pextrd. What CPU you have?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jackun/TestAMFVFW/issues/12#issuecomment-236497697, or mute the thread https://github.com/notifications/unsubscribe-auth/ATzXTWuCGY8iz2IjDOWyDOzCCySrzNcIks5qbY73gaJpZM4JZO6_ .

petermg commented 8 years ago

Ok, for some reason setting the encoder to OpenCL is the only one that will work for me. I have a Radeon R9 270x graphics card. I'm running Windows 10 64bit. Here are the results when I use the first 2 options.

Here are the results when I use "AMF Converter (RGB32 only):

`VirtualDub crash report -- build 35491 (release)

Disassembly: 668e46e0: 243a and al, 3ah 668e46e2: 8b7dfc mov edi, [ebp-04h] 668e46e5: 660f72d310 psrld xmm3, 10h 668e46ea: 660f73d806 psrldq xmm0, 06h 668e46ef: 660ffd45b0 paddw xmm0, [ebp-50h] 668e46f4: 660f7e0c3e movd [esi+edi], xmm1 668e46f9: 0f28cf movaps xmm1, xmm7 668e46fc: 660f73f30e psllq xmm3, 0eh 668e4701: 660f73d906 psrldq xmm1, 06h 668e4706: 660ffada psubd xmm3, xmm2 668e470a: 660ffdcf paddw xmm1, xmm7 668e470e: 660f72d309 psrld xmm3, 09h 668e4713: 660f6cc8 punpcklqdq xmm1, xmm0 668e4717: 660ff51d602c8f pmaddwd xmm3, [668f2c60] 66
668e471f: 660f72d110 psrld xmm1, 10h 668e4724: 660f73f10e psllq xmm1, 0eh 668e4729: 660ffe1db02c8f paddd xmm3, [668f2cb0] 66
668e4731: 660ffacd psubd xmm1, xmm5 668e4735: 8345f40c add dword ptr [ebp-0ch], 0ch 668e4739: 8bcf mov ecx, edi 668e473b: 0f2825802c8f66 movaps xmm4, [668f2c80] 668e4742: 83c104 add ecx, 04h 668e4745: 0f282da02c8f66 movaps xmm5, [668f2ca0] 668e474c: 0f57c0 xorps xmm0, xmm0 668e474f: 8b7ddc mov edi, [ebp-24h] 668e4752: 660f72d109 psrld xmm1, 09h 668e4757: 660f67db packuswb xmm3, xmm3 668e475b: 660ff50d602c8f pmaddwd xmm1, [668f2c60] 66
668e4763: 894dfc mov [ebp-04h], ecx 668e4766: 660ffe0db02c8f paddd xmm1, [668f2cb0] 66
668e476e: 660f67c9 packuswb xmm1, xmm1 668e4772: 660fe0d9 pavgb xmm3, xmm1 668e4776: 660f71d308 psrlw xmm3, 08h 668e477b: 660f67db packuswb xmm3, xmm3 668e477f: 660f db 0fh <-- FAULT 668e4781: 3a16 cmp dl, [esi] 668e4783: 1a00 sbb al, [eax] 668e4785: 8b55e0 mov edx, [ebp-20h] 668e4788: 83c20c add edx, 0ch 668e478b: 8955e0 mov [ebp-20h], edx 668e478e: 3b4b08 cmp ecx, [ebx+08h] 668e4791: 0f822bfeffff jc 668e45c2 668e4797: 0f2825802c8f66 movaps xmm4, [668f2c80] 668e479e: 0f282da02c8f66 movaps xmm5, [668f2ca0] 668e47a5: 8b75d8 mov esi, [ebp-28h] 668e47a8: 8b7df0 mov edi, [ebp-10h] 668e47ab: 8b4dec mov ecx, [ebp-14h] 668e47ae: 8b530c mov edx, [ebx+0ch] 668e47b1: 8b45d0 mov eax, [ebp-30h] 668e47b4: 83c602 add esi, 02h 668e47b7: 037dd4 add edi, [ebp-2ch] 668e47ba: 034dd4 add ecx, [ebp-2ch] 668e47bd: 0145e8 add [ebp-18h], eax 668e47c0: 0145e4 add [ebp-1ch], eax 668e47c3: 8975d8 mov [ebp-28h], esi 668e47c6: 897df0 mov [ebp-10h], edi 668e47c9: 894dec mov [ebp-14h], ecx 668e47cc: 3bf2 cmp esi, edx 668e47ce: 0f82b1fdffff jc 668e4585 668e47d4: 5f pop edi 668e47d5: 5e pop esi 668e47d6: 8be5 mov esp, ebp 668e47d8: 5d pop ebp 668e47d9: 8be3 mov esp, ebx 668e47db: 5b pop ebx 668e47dc: c3 ret
668e47dd: cc int 3 668e47de: cc int 3 668e47df: cc int 3

Built on Althena on Sun Oct 27 15:59:07 2013 using compiler version 1400

Windows 6.3 (Windows 7 x64 build 9600) [] Memory status: virtual free 1411M/2048M, commit limit 32761M, physical total 16377M

EAX = 07a22b00 EBX = 0c09fa8c ECX = 00000004 EDX = 07a22b00 EBP = 0c09fa80 ESI = 0798c600 EDI = fff6a000 ESP = 0c09fa20 EIP = 668e477f EFLAGS = 00010202 FPUCW = 027f FPUTW = ffff

Crash reason: Illegal instruction

Crash context: An instruction not supported by the CPU was executed in module 'TestAMFVFW'...

...while compressing frame 0 from 19ae2020 to 18711020 using codec "TestAMFVFW Encoder" (w32videocodecpack.cpp:787)...

...while running thread "Processing" (thread.cpp:197).

Pointer dumps:

EAX 07a22b00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 EBX 0c09fa88: 0b1b87c4 08cd7a8c 668e3e7e 00000500 000003c0 00000500 668f2220 0b817af8 EDX 07a22b00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ESI 0798c600: 10101010 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ESP 0c09fa20: 0b817af8 0b817b04 68f9f8d4 5bb4a11c 00000000 00000000 00000000 00000000 0c09fa40: 00000000 00000000 00000000 00000000 00001e00 fffff600 00000000 fff6a000 0c09fa60: 19ae2f20 19ae2020 19ae2f20 0798c600 0798cb00 19ae202c 07861000 00000004 0c09fa80: 0c09fb00 668e3e7e 0b1b87c4 08cd7a8c 668e3e7e 00000500 000003c0 00000500 EBP 0c09fa80: 0c09fb00 668e3e7e 0b1b87c4 08cd7a8c 668e3e7e 00000500 000003c0 00000500 0c09faa0: 668f2220 0b817af8 0c09fbb8 00000109 000003c0 00000500 19ae2020 0a1bf790 0c09fac0: 0b1b87c4 07861000 00000500 001c2000 00000500 000003c0 00000001 00000001 0c09fae0: 00000067 00000001 00000000 00000003 00000000 00010000 00000000 54286560

Thread call stack: 668e477f: TestAMFVFW!0000477f 68f9f8d4: d3d11!D3D11CoreCreateLayeredDevice [68f50000+3ab00+14dd4] 668e3e7e: TestAMFVFW!00003e7e 668e3e7e: TestAMFVFW!00003e7e 668e3548: TestAMFVFW!00003548 668ec525: TestAMFVFW!DriverProc [668e0000+c430+f5] 669d2da3: MSVFW32!ICSendMessage [669d0000+2d60+43] 669d3193: MSVFW32!ICCompress [669d0000+3130+63] 005da2ee: VDVideoCompressorVCM::PackFrameInternal() 005dad85: VDVideoCompressorVCM::CompressFrame() 00570df3: VDPixmapGenResampleCol_d2_pnqrtr_lin_u8_ISSE::Compute() 0048c99b: VDThreadedVideoCompressor::ProcessFrame() 0048ceed: VDThreadedVideoCompressor::ExchangeBuffer() 00424353: VDDubVideoProcessor::WriteFinishedVideoFrame() 00424dec: VDDubVideoProcessor::ProcessVideoFrame() 004250ca: VDDubVideoProcessor::RunPathWriteOutputFrames() 00425cdf: VDDubVideoProcessor::WriteVideo() 00422d3e: VDDubProcessThread::ThreadRun() 004f9917: ?$VDProtectedAutoScope1::?$VDProtectedAutoScope1() 0051a2dc: VDThread::StaticThreadStart() 005fcc0e: _callthreadstartex() 005fccb3: _threadstartex@4() 749f38f4: KERNEL32!BaseThreadInitThunk [749e0000+138d0+24] 77105de3: ntdll!RtlUnicodeStringToInteger [770a0000+65b90+253] 77105dae: ntdll!RtlUnicodeStringToInteger [770a0000+65b90+21e]

-- End of report `

and HERE are the results when I use D3D11 DirectCompute (Win8+, RGB32 only)

`VirtualDub crash report -- build 35491 (release)

Disassembly: 669746e0: 243a and al, 3ah 669746e2: 8b7dfc mov edi, [ebp-04h] 669746e5: 660f72d310 psrld xmm3, 10h 669746ea: 660f73d806 psrldq xmm0, 06h 669746ef: 660ffd45b0 paddw xmm0, [ebp-50h] 669746f4: 660f7e0c3e movd [esi+edi], xmm1 669746f9: 0f28cf movaps xmm1, xmm7 669746fc: 660f73f30e psllq xmm3, 0eh 66974701: 660f73d906 psrldq xmm1, 06h 66974706: 660ffada psubd xmm3, xmm2 6697470a: 660ffdcf paddw xmm1, xmm7 6697470e: 660f72d309 psrld xmm3, 09h 66974713: 660f6cc8 punpcklqdq xmm1, xmm0 66974717: 660ff51d602c98 pmaddwd xmm3, [66982c60] 66
6697471f: 660f72d110 psrld xmm1, 10h 66974724: 660f73f10e psllq xmm1, 0eh 66974729: 660ffe1db02c98 paddd xmm3, [66982cb0] 66
66974731: 660ffacd psubd xmm1, xmm5 66974735: 8345f40c add dword ptr [ebp-0ch], 0ch 66974739: 8bcf mov ecx, edi 6697473b: 0f2825802c9866 movaps xmm4, [66982c80] 66974742: 83c104 add ecx, 04h 66974745: 0f282da02c9866 movaps xmm5, [66982ca0] 6697474c: 0f57c0 xorps xmm0, xmm0 6697474f: 8b7ddc mov edi, [ebp-24h] 66974752: 660f72d109 psrld xmm1, 09h 66974757: 660f67db packuswb xmm3, xmm3 6697475b: 660ff50d602c98 pmaddwd xmm1, [66982c60] 66
66974763: 894dfc mov [ebp-04h], ecx 66974766: 660ffe0db02c98 paddd xmm1, [66982cb0] 66
6697476e: 660f67c9 packuswb xmm1, xmm1 66974772: 660fe0d9 pavgb xmm3, xmm1 66974776: 660f71d308 psrlw xmm3, 08h 6697477b: 660f67db packuswb xmm3, xmm3 6697477f: 660f db 0fh <-- FAULT 66974781: 3a16 cmp dl, [esi] 66974783: 1a00 sbb al, [eax] 66974785: 8b55e0 mov edx, [ebp-20h] 66974788: 83c20c add edx, 0ch 6697478b: 8955e0 mov [ebp-20h], edx 6697478e: 3b4b08 cmp ecx, [ebx+08h] 66974791: 0f822bfeffff jc 669745c2 66974797: 0f2825802c9866 movaps xmm4, [66982c80] 6697479e: 0f282da02c9866 movaps xmm5, [66982ca0] 669747a5: 8b75d8 mov esi, [ebp-28h] 669747a8: 8b7df0 mov edi, [ebp-10h] 669747ab: 8b4dec mov ecx, [ebp-14h] 669747ae: 8b530c mov edx, [ebx+0ch] 669747b1: 8b45d0 mov eax, [ebp-30h] 669747b4: 83c602 add esi, 02h 669747b7: 037dd4 add edi, [ebp-2ch] 669747ba: 034dd4 add ecx, [ebp-2ch] 669747bd: 0145e8 add [ebp-18h], eax 669747c0: 0145e4 add [ebp-1ch], eax 669747c3: 8975d8 mov [ebp-28h], esi 669747c6: 897df0 mov [ebp-10h], edi 669747c9: 894dec mov [ebp-14h], ecx 669747cc: 3bf2 cmp esi, edx 669747ce: 0f82b1fdffff jc 66974585 669747d4: 5f pop edi 669747d5: 5e pop esi 669747d6: 8be5 mov esp, ebp 669747d8: 5d pop ebp 669747d9: 8be3 mov esp, ebx 669747db: 5b pop ebx 669747dc: c3 ret
669747dd: cc int 3 669747de: cc int 3 669747df: cc int 3

Built on Althena on Sun Oct 27 15:59:07 2013 using compiler version 1400

Windows 6.3 (Windows 7 x64 build 9600) [] Memory status: virtual free 1454M/2048M, commit limit 32761M, physical total 16377M

EAX = 06b21b00 EBX = 0ff3fa8c ECX = 00000004 EDX = 06b21b00 EBP = 0ff3fa80 ESI = 06a8b600 EDI = fff6a000 ESP = 0ff3fa20 EIP = 6697477f EFLAGS = 00010202 FPUCW = 027f FPUTW = ffff

Crash reason: Illegal instruction

Crash context: An instruction not supported by the CPU was executed in module 'TestAMFVFW'...

...while compressing frame 0 from 18c7c020 to 181bb020 using codec "TestAMFVFW Encoder" (w32videocodecpack.cpp:787)...

...while running thread "Processing" (thread.cpp:197).

Pointer dumps:

EAX 06b21b00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 EBX 0ff3fa88: 099d35d4 099633ec 66973e7e 00000500 000003c0 00000500 66982220 0b620320 EDX 06b21b00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ESI 06a8b600: 10101010 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ESP 0ff3fa20: 0b620320 0b62032c 68f9f8d4 30837b83 00000000 00000000 00000000 00000000 0ff3fa40: 00000000 00000000 00000000 00000000 00001e00 fffff600 00000000 fff6a000 0ff3fa60: 18c7cf20 18c7c020 18c7cf20 06a8b600 06a8bb00 18c7c02c 06960000 00000004 0ff3fa80: 0ff3fb00 66973e7e 099d35d4 099633ec 66973e7e 00000500 000003c0 00000500 EBP 0ff3fa80: 0ff3fb00 66973e7e 099d35d4 099633ec 66973e7e 00000500 000003c0 00000500 0ff3faa0: 66982220 0b620320 0ff3fbb8 ffff01fe 000003c0 00000500 18c7c020 09a26ed0 0ff3fac0: 099d35d4 06960000 00000500 001c2000 00000500 000003c0 00000001 00000001 0ff3fae0: 00000067 00000001 00000000 00000003 00000000 00010000 00000000 3f08e587

Thread call stack: 6697477f: TestAMFVFW!0000477f 68f9f8d4: d3d11!D3D11CoreCreateLayeredDevice [68f50000+3ab00+14dd4] 66973e7e: TestAMFVFW!00003e7e 66973e7e: TestAMFVFW!00003e7e 66973548: TestAMFVFW!00003548 6697c525: TestAMFVFW!DriverProc [66970000+c430+f5] 669d2da3: MSVFW32!ICSendMessage [669d0000+2d60+43] 669d3193: MSVFW32!ICCompress [669d0000+3130+63] 005da2ee: VDVideoCompressorVCM::PackFrameInternal() 005dad85: VDVideoCompressorVCM::CompressFrame() 00570df3: VDPixmapGenResampleCol_d2_pnqrtr_lin_u8_ISSE::Compute() 0048c99b: VDThreadedVideoCompressor::ProcessFrame() 0048ceed: VDThreadedVideoCompressor::ExchangeBuffer() 00424353: VDDubVideoProcessor::WriteFinishedVideoFrame() 00424dec: VDDubVideoProcessor::ProcessVideoFrame() 004250ca: VDDubVideoProcessor::RunPathWriteOutputFrames() 00425cdf: VDDubVideoProcessor::WriteVideo() 00422d3e: VDDubProcessThread::ThreadRun() 004f9917: ?$VDProtectedAutoScope1::?$VDProtectedAutoScope1() 0051a2dc: VDThread::StaticThreadStart() 005fcc0e: _callthreadstartex() 005fccb3: _threadstartex@4() 749f38f4: KERNEL32!BaseThreadInitThunk [749e0000+138d0+24] 77105de3: ntdll!RtlUnicodeStringToInteger [770a0000+65b90+253] 77105dae: ntdll!RtlUnicodeStringToInteger [770a0000+65b90+21e]

-- End of report `

jackun commented 8 years ago

yes, redownload, should be fixed mostly.

petermg commented 8 years ago

THANK YOU!!!! Now it works in VirtualDub but only in the 32bit version. Same with Magix Movie Edit Pro. Works in the 32 bit version of each, but in the 64bit versions, it crashes. Here's the crash from VirtualDub 64bit version:

`VirtualDub crash report -- build 35491 (release-AMD64)

Disassembly: 7ffd68654cc0: 60 db 60h 7ffd68654cc1: ca660f retf 0f66 7ffd68654cc4: 70d4 jo 168654c9a 7ffd68654cc6: b166 mov cl, 66h 7ffd68654cc8: 410f6fd9 movq mm3, mm1 7ffd68654ccc: 66410ff5db pmaddwd xmm3, xmm3 7ffd68654cd1: 660ffed4 paddd xmm2, xmm4 7ffd68654cd5: 66410ffed5 paddd xmm2, xmm5 7ffd68654cda: 660f6be4 packssdw xmm4, xmm4 7ffd68654cde: 66410ff5d6 pmaddwd xmm2, xmm6 7ffd68654ce3: 660f67e4 packuswb xmm4, xmm4 7ffd68654ce7: 66430f7e641d00 movd [r13+r11+00h], xmm4 7ffd68654cee: 660f70c3b1 pshufd xmm0, xmm3, 0b1h 7ffd68654cf3: 660ffec3 paddd xmm0, xmm3 7ffd68654cf7: 660f6fde movdqa xmm3, xmm6 7ffd68654cfb: 660f73db06 psrldq xmm3, 06h 7ffd68654d00: 660f73d804 psrldq xmm0, 04h 7ffd68654d05: 660ffdde paddw xmm3, xmm6 7ffd68654d09: 660f6cc8 punpcklqdq xmm1, xmm0 7ffd68654d0d: 660f6fc7 movdqa xmm0, xmm7 7ffd68654d11: 66410ffecc paddd xmm1, xmm4 7ffd68654d16: 660f73d806 psrldq xmm0, 06h 7ffd68654d1b: 660f72d10f psrld xmm1, 0fh 7ffd68654d20: 660ffdc7 paddw xmm0, xmm7 7ffd68654d24: 660f6cd8 punpcklqdq xmm3, xmm0 7ffd68654d28: 66410f6fc1 movdqa xmm0, xmm1 7ffd68654d2d: 660f70e9b1 pshufd xmm5, xmm1, 0b1h 7ffd68654d32: 660ffee9 paddd xmm5, xmm1 7ffd68654d36: 660f72d310 psrld xmm3, 10h 7ffd68654d3b: 660f6bc9 packssdw xmm1, xmm1 7ffd68654d3f: 66410ffeed paddd xmm5, xmm5 7ffd68654d44: 660f67c9 packuswb xmm1, xmm1 7ffd68654d48: 66420f7e0c19 movd [rcx+r11], xmm1 <-- FAULT 7ffd68654d4e: 66410f6fc8 movdqa xmm1, xmm0 7ffd68654d53: 660f73f30e psllq xmm3, 0eh 7ffd68654d58: 660ffada psubd xmm3, xmm2 7ffd68654d5c: 660f73d906 psrldq xmm1, 06h 7ffd68654d61: 660f72d309 psrld xmm3, 09h 7ffd68654d66: 66410ffdc8 paddw xmm1, xmm0 7ffd68654d6b: 660ff51d1dfd00 pmaddwd xmm3, [168664a90] 00
7ffd68654d73: 660f73d806 psrldq xmm0, 06h 7ffd68654d78: 66410ffdc1 paddw xmm0, xmm1 7ffd68654d7d: 66410ff5ee pmaddwd xmm5, xmm6 7ffd68654d82: 660f6cc8 punpcklqdq xmm1, xmm0 7ffd68654d86: 660f72d110 psrld xmm1, 10h 7ffd68654d8b: 66410ffedf paddd xmm3, xmm7 7ffd68654d90: 660f73f10e psllq xmm1, 0eh 7ffd68654d95: 4183c70c add edi, 0ch 7ffd68654d99: 660ffacd psubd xmm1, xmm5 7ffd68654d9d: 660f67db packuswb xmm3, xmm3 7ffd68654da1: 660f72d109 psrld xmm1, 09h 7ffd68654da6: 660ff50de2fc00 pmaddwd xmm1, [168664a90] 00
7ffd68654dae: 66410ffecf paddd xmm1, xmm7 7ffd68654db3: 660f67c9 packuswb xmm1, xmm1 7ffd68654db7: 660fe0d9 pavgb xmm3, xmm1 7ffd68654dbb: 660f71d308 psrlw xmm3, 08h

Built on Althena on Sun Oct 27 16:00:02 2013 using compiler version 1400

Windows 6.3 (Windows 7 x64 build 9600) [] Memory status: virtual free 134217021M/134217728M, commit limit 32761M, physical total 16377M

RAX = 1d4dd820 RBX = 1684 RCX = fffff800 RDX = 220000 RSI = ffffe980 RDI = 800 RBP = 1bad4040 R8 = 780 R9 = 0 R10 = 780 R11 = 1d3cb820 R12 = ffffe97c R13 = 0 R14 = 1d1b0020 R15 = 1684 RSP = 1c2cf510 RIP = 7ffd68654d48 EFLAGS = 00010287

Crash reason: Access Violation

Crash context: An out-of-bounds memory access (access violation) occurred in module 'atiumd6a'...

...writing address 000000011D3CB020...

...while compressing frame 0 from 1bad4040 to 1a8c8040 using codec "TestAMFVFW64 Encoder" (w32videocodecpack.cpp:787)...

...while running thread "Processing" (thread.cpp:197).

Pointer dumps:

RAX 1d4dd820: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 RSP 1c2cf510: 00002d00 00000000 00220000 00000000 00000000 00000000 00000000 00000000 1c2cf530: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 1c2cf550: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 1c2cf570: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 RBP 1bad4040: 2c3a332a 352d3b34 3c352d3c 2e3d362e 362e3d36 3d362e3d 2d38312a 312a3b34 1bad4060: 38312a38 273a332a 3129362f 362f2738 26362f28 2c25342d 352e2733 22312a23 1bad4080: 2d263029 332c2534 22302922 2a233029 332c2531 28332c25 2c25362f 362f2833 1bad40a0: 2a342d26 312a3831 37302938 29362f27 2f273831 38312936 27352e26 3028362f R11 1d3cb820: 3f3f3e3d 00000000 00000000 00000000 00000000 00000000 00000000 00000000 R14 1d1b0020: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Thread call stack: 7ffd68654d48: TestAMFVFW!00004d48 7ffd686543f2: TestAMFVFW!000043f2 7ffd68653650: TestAMFVFW!00003650 7ffd6865d407: TestAMFVFW!DriverProc [7ffd68650000+d2c0+147] 7ffd68722936: MSVFW32!ICSendMessage [7ffd68720000+28c0+76] 7ffd68722e0d: MSVFW32!ICCompress [7ffd68720000+2da0+6d] 14026af09: VDVideoCompressorVCM::PackFrameInternal() 7ffd78985bd4: ntdll!RtlAllocateHeap [7ffd78960000+255d0+604] 14026bd50: VDVideoCompressorVCM::CompressFrame() 7ffd758daadf: KERNELBASE!WaitForSingleObjectEx [7ffd758c0000+1aa50+8f] 1400c6eea: VDThreadedVideoCompressor::ProcessFrame() 1401d3e95: VDPixmapCreateBlitter() 1400c78cc: VDThreadedVideoCompressor::ExchangeBuffer() 140181b5d: VDMemcpyRect() 140030b06: VDDubVideoProcessor::WriteFinishedVideoFrame() 140032618: VDDubVideoProcessor::ProcessVideoFrame() 1400334c6: VDDubVideoProcessor::RunPathReadFrame() 140033eba: VDDubVideoProcessor::WriteVideo() 1400375c4: VDStreamInterleaver::GetNextAction() 14002ff38: VDDubProcessThread::ThreadRun() 140002981: ?$VDProtectedAutoScope1::?$VDProtectedAutoScope1() 14018a14b: VDInitThreadData() 14017c00a: VDThread::StaticThreadStart() 140298b7f: threadstartex() 7ffd787a8102: KERNEL32!BaseThreadInitThunk [7ffd78790000+180e0+22] 7ffd789bc5b4: ntdll!RtlUserThreadStart [7ffd78960000+5c580+34]

-- End of report `

petermg commented 8 years ago

Ok so Github flagged me as a robot for a bit there so this issue went invisible during that time. Hopefully the crash report I posted from the VirtualDub 64bit version is helpful. Thanks again for making it at least work now on the 32 bit versions!

jackun commented 8 years ago

Oh joy, instead of subtracting from the pointer 0x800, it added 0xFFFFF800 on 64bit. Gotta use correct integer types, heh.