Closed qyot27 closed 3 years ago
pinterf
commented
3 years ago Perhaps AVSFunction::name is uninitialized to 0 (NULL)? Could you please add the 'else' part of the below code to PluginManager.cpp line 243 (in AVSFunction::AVSFunction)
it should look like this:
if (NULL != _name)
{
size_t len = strlen(_name);
auto tmp = new char[len + 1];
memcpy(tmp, _name, len);
tmp[len] = 0;
name = tmp;
}
else {
name = NULL;
}
qyot27
commented
3 years ago Tried it (even with all the plugins in plugins64_gcc moved away so they wouldn't autoload), but same result:
Thread 1 received signal SIGSEGV, Segmentation fault.
0x00007ffaf2c8d2d0 in strlen () from /c/Windows/System32/msvcrt.dll
(gdb) bt
#0 0x00007ffaf2c8d2d0 in strlen () from /c/Windows/System32/msvcrt.dll
#1 0x00007ffaa7da56a7 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::append(char const*) () from /e/Programs/AviSynth+/AviSynth64.dll
#2 0x00007ffaa777b589 in ScriptEnvironment::ExportBuiltinFilters (this=0x275fd1038d0)
at ../avs_core/core/avisynth.cpp:2867
#3 0x00007ffaa7778fe9 in ScriptEnvironment::ScriptEnvironment (this=0x275fd1038d0)
at ../avs_core/core/avisynth.cpp:2324
#4 0x00007ffaa7784392 in CreateScriptEnvironment2 (version=6) at ../avs_core/core/avisynth.cpp:5195
#5 0x00007ffaa778431a in CreateScriptEnvironment (version=6) at ../avs_core/core/avisynth.cpp:5167
#6 0x00007ffaa778894b in avs_create_script_environment (version=6) at ../avs_core/core/avisynth_c.cpp:1419
#7 0x00007ff653a4387a in avisynth_context_create (s=0x275fb9253c0) at src/libavformat/avisynth.c:182
#8 avisynth_open_file (s=s@entry=0x275fb9253c0) at src/libavformat/avisynth.c:793
#9 0x00007ff653a448d0 in avisynth_read_header (s=0x275fb9253c0) at src/libavformat/avisynth.c:1007
#10 0x00007ff652bab911 in avformat_open_input (ps=ps@entry=0x785bbff650,
filename=filename@entry=0x275fb9251d6 "test.avs", fmt=fmt@entry=0x0, options=0x275fb925328)
at src/libavformat/demux.c:297
#11 0x00007ff6528da99c in open_input_file (o=o@entry=0x785bbff850, filename=<optimized out>)
at src/fftools/ffmpeg_opt.c:1209
#12 0x00007ff6528df28f in open_files (open_file=0x7ff6528d9c50 <open_input_file>, inout=0x7ff653acd427 "input",
l=0x275fb925258) at src/fftools/ffmpeg_opt.c:3393
#13 ffmpeg_parse_options (argc=argc@entry=3, argv=argv@entry=0x275fb924f20) at src/fftools/ffmpeg_opt.c:3433
#14 0x00007ff653a25774 in main (argc=3, argv=0x275fb924f20) at src/fftools/ffmpeg.c:5014I also tried with avs2yuv, which points at a smaller set of the same lines as FFmpeg:
Thread 1 received signal SIGSEGV, Segmentation fault.
0x00007ffaf2c8d2d0 in strlen () from /c/Windows/System32/msvcrt.dll
(gdb) bt
#0 0x00007ffaf2c8d2d0 in strlen () from /c/Windows/System32/msvcrt.dll
#1 0x00007ffaa7a856a7 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::append(char const*) () from /e/Programs/AviSynth+/AviSynth64.dll
#2 0x00007ffaa745b589 in ScriptEnvironment::ExportBuiltinFilters (this=0x202dde78e60)
at ../avs_core/core/avisynth.cpp:2867
#3 0x00007ffaa7458fe9 in ScriptEnvironment::ScriptEnvironment (this=0x202dde78e60)
at ../avs_core/core/avisynth.cpp:2324
#4 0x00007ffaa7464392 in CreateScriptEnvironment2 (version=8) at ../avs_core/core/avisynth.cpp:5195
#5 0x00007ffaa746431a in CreateScriptEnvironment (version=8) at ../avs_core/core/avisynth.cpp:5167
#6 0x00007ffaa746894b in avs_create_script_environment (version=8) at ../avs_core/core/avisynth_c.cpp:1419
#7 0x00007ff750755b90 in main (argc=<optimized out>, argv=0x202dc831620) at avs2yuv.c:238Hold on. Why is avs2yuv reporting version=8 and FFmpeg is reporting version=6 at lines 4-6?
pinterf
commented
3 years ago Try fprintf f->name to stdout to see where the crash happens?
qyot27
commented
3 years ago Inserting fprintf into line 2876 in avisynth.cpp:
std::string param_var_name;
param_var_name.reserve(128);
param_var_name.append("$Plugin!");
param_var_name.append(f->name);
fprintf(stdout,"%s\n",f->name);
param_var_name.append("!Param$");Resulted in FFmpeg outputting:
ffmpeg version r103928+10 master-68815d6791 HEAD-6e6a0e3367
contains: mergedin
Copyright (c) 2000-2021 the FFmpeg developers
built on Oct 10 2021 11:35:30 with gcc 11.1.0 (GCC)
configuration: --prefix=/home/qyot27/ffmpeg64_build_debug --cross-prefix=x86_64-w64-mingw32- --enable-gpl --enable-version3 --disable-w32threads --disable-doc --enable-debug --disable-stripping --enable-avisynth --target-os=mingw32 --arch=x86_64
libavutil 57. 7.100 / 57. 7.100
libavcodec 59. 9.101 / 59. 9.101
libavformat 59. 5.100 / 59. 5.100
libavdevice 59. 0.101 / 59. 0.101
libavfilter 8. 10.101 / 8. 10.101
libswscale 6. 1.100 / 6. 1.100
libswresample 4. 0.100 / 4. 0.100
libpostproc 56. 0.100 / 56. 0.100
DelayAudio
AmplifydB
Amplify
AssumeSampleRate
Normalize
MixAudio
ResampleAudio
ConvertToMono
EnsureVBRMP3Sync
MergeChannels
MonoToStereo
GetLeftChannel
GetRightChannel
GetChannel
GetChannels
KillVideo
KillAudio
ConvertAudioTo16bit
ConvertAudioTo8bit
ConvertAudioTo24bit
ConvertAudioTo32bit
ConvertAudioToFloat
ConvertAudio
StackVertical
StackHorizontal
ShowFiveVersions
Animate
Animate
ApplyRange
ConvertToRGB
ConvertToRGB24
ConvertToRGB32
ConvertToRGB48
ConvertToRGB64
ConvertToPlanarRGB
ConvertToPlanarRGBA
ConvertToY8
ConvertToYV12
ConvertToYV24
ConvertToYV16
ConvertToYV411
ConvertToYUY2
ConvertBackToYUY2
ConvertToY
ConvertToYUV411
ConvertToYUV420
ConvertToYUV422
ConvertToYUV444
ConvertTo8bit
ConvertTo16bit
ConvertToFloat
ConvertBits
AddAlphaPlane
RemoveAlphaPlane
GeneralConvolution
AudioTrim
AudioTrim
AudioTrim
AudioTrim
Trim
Trim
Trim
Trim
FreezeFrame
DeleteFrame
DuplicateFrame
UnalignedSplice
AlignedSplice
UnalignedSplice
AlignedSplice
Dissolve
AudioDub
AudioDubEx
Reverse
FadeOut0
FadeOut
FadeOut2
FadeIn0
FadeIn
FadeIn2
FadeIO0
FadeIO
FadeIO2
Loop
ComplementParityIt bails out after ComplementParity, before it ever gets to Version.
pinterf
commented
3 years ago I get a suspicious warning in Avisynth.cpp line 1133 related to the threadvar syntax.
Try adding: !defined(__GNUC__)
#if defined(AVS_WINDOWS) && !defined(__GNUC__)
(Meanwhile I struggle to get it work, nothing is clear to me :) . Do I need a special ffmpeg build in order to use mingw built windows avisynth?)
qyot27
commented
3 years ago Do I need a special ffmpeg build in order to use mingw built windows avisynth?
Not for 64bit. 32-bit builds do require a special build of FFmpeg, but that's working fine. This problem is only occurring in 64-bit builds of AviSynth+, and only post-3.5.1 (I can build 3.5.1 and the same 64-bit FFmpeg build used for normal MSVC builds of AviSynth+ is fine with it).
Playing around a little bit, if I disable NEW_AVSVALUE, the function list gets farther along before segfaulting - AssumeFPS is reached before it bails out. I don't know if this means there's something about the interaction of things from Neo with the NEW_AVSVALUE arrays that 64-bit GCC or MinGW-w64 doesn't like, or if it's just masking something else.
pinterf
commented
3 years ago Sorry, I'm stuck again. No meaningful error message is given, even if I put a fprintf(stdout, "I'm here\n") into avs_create_script_environment so it looks like it doesn't even able to call the entry point.
I downloaded an actual ffmpeg build but I only get a not too helpful Unknown error occurred. No hint of my stdout debug text.
ffmpeg version 2021-10-07-git-b6aeee2d8b-full_build-www.gyan.dev Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 10.3.0 (Rev5, Built by MSYS2 project)
configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libdav1d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libglslang --enable-vulkan --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
libavutil 57. 7.100 / 57. 7.100
libavcodec 59. 10.100 / 59. 10.100
libavformat 59. 5.101 / 59. 5.101
libavdevice 59. 0.101 / 59. 0.101
libavfilter 8. 11.100 / 8. 11.100
libswscale 6. 1.100 / 6. 1.100
libswresample 4. 0.100 / 4. 0.100
libpostproc 56. 0.100 / 56. 0.100
test.avs: Unknown error occurred
I built my avisynth.dll with
@rem cd avisynth-build
del ..\CMakeCache.txt
cmake .. -G "MinGW Makefiles" -DBUILD_DIRECTSHOWSOURCE:bool=off -DENABLE_PLUGINS:bool=on -DENABLE_INTEL_SIMD:bool=ON
cmake --build . --config Debug
then I have put the resulting dll into c:\windows\system32
qyot27
commented
3 years ago There's definitely something wrong with however MSys2's GCC is configured when it comes to building AviSynth+, as I encountered the exact same thing, even with 3.4 and 3.5. I've been able to produce builds using my normal cross toolchain from Linux (or transplanting it from Linux into WSL).
pinterf
commented
3 years ago O.K. new challenge then. At least I get familiar again with these command line build mechanisms. Could you upload your debug build somewhere?
qyot27
commented
3 years ago FFmpeg & AviSynth+: https://www.mediafire.com/file/xl5zekrjzkoapws/ffmpeg_r104328%252B10_debug.7z/file
Contains standard 32-bit, special 32-bit (avsgcc), and standard 64-bit FFmpeg builds. AviSynth+ git in 32-bit and 64-bit (to show that 32-bit works and 64-bit doesn't), and 3.5.1 in 32-bit and 64-bit (to show that both work). The only changes to AviSynth+ are in BufferPool.h (the addition of cstddef to the includes to allow it to build with GCC 11.1.0, like I mentioned in #227), and the 3.5.1 builds likewise needed some extra headers included to allow building with that version of GCC. All debug builds.
Given the oddities with MSys2, here's the MinGW-w64/GCC basic cross environment for Ubuntu (21.04/WSL). https://www.mediafire.com/file/doflifdaw42la35/mingw-w64-gcc_11.1.0.tar.xz/file
pinterf
commented
3 years ago Thanks, got them
pinterf
commented
3 years ago A made a quick comparison between our Avisynth versions.
Using dependency walker: https://github.com/lucasg/Dependencies/releases
My MinGW-built (non-working) avisynth.dll version listed these additional libraries:
C:\msys64\mingw64\bin\libwinpthread-1.dll
C:\msys64\mingw64\bin\libgcc_s_seh-1.dll
C:\msys64\mingw64\bin\libstdc++-6.dllYour (sooner or later crashing) version does not have these libs, only the usual Microsoft libraries.
pinterf
commented
3 years ago Well, probably a gcc bug. edit: we'd better do not say such things until proven. Though the overwrite is detected before the very first avs scriptenvironment call, other static initializations would mess up something.
When initializing structs some memory overwrite happens
On DLL initialization compiler automatically populates builtin_functions array from addresses of other extern arrays
(Audio_filters, Field_filters, etc)
Field_filters as an array of AVSFunction which inherits from Function
Function is a struct with many pointers and a single bool inside.
https://github.com/AviSynth/AviSynthPlus/blob/master/avs_core/core/function.h#L15
struct Function {
typedef AVSValue(__cdecl *apply_func_t)(AVSValue args, void* user_data, IScriptEnvironment* env);
apply_func_t apply;
const char* name;
const char* canon_name;
const char* param_types;
void* user_data;
const char* dll_path;
bool isAvs25; // can be filled during plugin load
};It seems that when the runtime is filling up the array of AVSFunction,
this array declaration is calling AVSFunction ctor for each entry: https://github.com/AviSynth/AviSynthPlus/blob/master/avs_core/filters/field.cpp#L61
the next user_data parameter (when using such constructor such as AssumeBFF ) is partially overwriting the name field of the previous record.
I'm still investigating under what circumstances it can happen, a struct alignment attribute to 16 bytes seemed to mute the problem.
edit: not passing user_data is still overwrites a previous field. The addresses below are for function name char* pointers. The second one is garbage -> crash on calling strlen on such a wrong address.
(writing known nice-looking patterns into the record fields in constructor. name is a real pointer)
struct size 56:
apply_func_t name canon_name param_types user_data dll_path is_avs25
[0][0] ffff000000d00201 0000010d9a1affff 99aabbccddeeff00 1122334455667788 0000000000000000 0123456789abcdef 0000000000000000
[0][1] fedcba9876543210 ffff000000e00201 99aabbccddeeffff 1122334455667788 0000000000000000 0123456789abcdef 0000000000000000
[0][2] fedcba9876543210 0000010d9a1a0b60 99aabbccddeeff00 1122334455667788 0000000000000000 0123456789abcdef 0000000000000000
struct size 64 (test to see the shifts)
apply_func_t name canon_name param_types user_data dll_path is_avs25
[0][0] fedcba9876543210 000001a417ed0ad0 99aabbccddeeff00 1122334455667788 ffff000000e00201 0123456789abffff 0000000000000000 ????????????????
[0][1] fedcba9876543210 000001a417ed0b20 99aabbccddeeff00 1122334455667788 ffff000000f00201 0123456789abffff 0000000000000000 ????????????????
[0][2] fedcba9876543210 000001a417ed0b60 99aabbccddeeff00 1122334455667788 0000000000000000 0123456789abcdef 0000000000000000 ????????????????
Overwrite Pattern:
Address: 0*64 + 0: (8 bytes) ffff000000d00201
Address: 0*64 + 14: (2 bytes) ffff
Address: 1*64 + 0: (8 bytes) ffff000000e00201
Address: 1*64 + 14: (2 bytes) ffff
I had a feeling that the main issue between MSys2-built and my environment were the additional system libraries. I think you can tell the MSys2 GCC to use static linking for all those libraries via CFLAGS or LDFLAGS, but in my case I've always made sure GCC and MinGW builds its library supports as static only.
pinterf
commented
3 years ago Yes, statis linking and pthread usage was the key for mingw. After that I was able to reproduce the issue and narrow it down. The issue probably affects 32 bit as well I think, all depends on how variables are mapped in memory one after another. So when I made struct padding to 16 bytes instead of 8, the issue was well hidden away but after watching the memory bytes it turned out that the overwriting is there but affected lucky bytes. Debugging this kind of bug can take some days to many months, seems to be a good fun.
pinterf
commented
3 years ago Commit e53cba0205dc9201292e6f555eb6be7ff78ce0ec fixed it for me.
qyot27
commented
3 years ago Seems to have fixed it for me, too.
However, I had to remove the !defined(__GNUC__) from line 1132 in avisynth.cpp to get it to compile.
https://github.com/AviSynth/AviSynthPlus/blob/master/avs_core/core/avisynth.cpp#L1132
pinterf
commented
3 years ago Do you have XP_TLS defined?
qyot27
commented
3 years ago 1eebe2e resolved it. It was silly to assume XP compatibility with the GCC builds anyway, since my own MinGW-w64 environment has the Secure API enabled (not sure if MSys2's does), which requires Vista or higher.
I've narrowed it down to something that changed in the integration of the Neo patches, as I can build 3.5.1 with GCC and have it work, but 3.6.0 and up cause a segfault. I tried going through it last night to figure out the exact commit, but as there are occasionally build problems more complicated than just including some headers, I wasn't able to work it out.
Oddly, 32-bit GCC builds are unaffected, and even ones from the current git can be used in the special
AVSC_WIN32_GCC3232-bit builds of FFmpeg that switch on support for 32-bit GCC builds of AviSynth+ (when I provide builds of FFmpeg, that's what the-avsgccbuilds are).Script:
gdb output: