Heap corruption

[mmanyen]

I took your extras/mk42sqfs.c file and replaced the file part with a file list. sqfsTest.zip

An exception on heap check is triggered on line 183 ( sqfs_meta_writer_flush(inode_m);)

I am using Windows 10 / Visual Studio 2019 / the windows binary package (squashfs-tools-ng-1.1.0-mingw64)

I fully expect this to be an incompatibility between VS and MinGW, but I thought I would ask.

Note: The provided gensquashfs.exe runs fine

[AgentD]

Hi @mmanyen ,

thanks for reporting this! I already tried to compile your sample on my Fedora system and it seems to work here. If it was a MSVC C++ to Mingw C calling convention issue, I would expect it to fail earlier.

One other thing that I suspect at the moment: version 1.1 adds a custom pool allocator that uses mmap on Unix-like systems and VirtualAlloc on Windows. Maybe this interferes with the C++ runtime heap somehow?

Anyway, I do have a Windows 10 installation with Visual Studio sitting around and will try to reproduce/debug this on my end.

[mmanyen]

That would be super! Thanks

[AgentD]

I got my Windows back in shape and took a look. The error is actually not at line 183 but in the free() statement above. As far as I understood it, I think the problem is this:

• libsquashfs.dll and the application are linked against different versions of the MSVCRT runtime.
• The function sqfs_dir_writer_create_inode internally calls malloc() on its runtime library and returns the result pointer.
• The application calls free() on the pointer, but using the different runtime library, which has its own heap and immediately sees that the pointer does not belong to it.
• The applications runtime library screams heap corruption.

On typical Linux distributions, everything works out fine, because there is only a single, global C runtime that everything is linked against.

This will need some kind of workaround, like an sqfs_free function that calls free() from within the library itself.

In theory it should work on Windows if you link the library statically, but I ran into further problems trying that. The predef header is currently missing some way to change the SQFS_API definition to not try a dll import and even then, the library is missing some ___heapchk_ms symbol, apparently from libgcc. I tried to patch in a dummy definition for that, but the resulting program throws an error that it supposedly cannot find ZSTD_compressCCtx in libzstd.dll, which is pretty much where I got stuck, for now.

[mmanyen]

Wow, I started something...

Thanks for looking into this, if I can help please let me know.

I have started a cmake build for this, if I can get it working would you like to incorporate the new build?

[AgentD]

I did at one point start a VisualStudio and also considered simply adding a CMake project file instead. Both approaches have the problem that it would require maintaining two build systems in parallel.

In theory a CMake build system could replace the existing Autotools one, but from my previous experience with CMake I suspect that this would become rather hacker/cumbersome given how many aspects of CMake seemed to have been designed around msvc with many workarounds tacked on later.

For this and a few other reasons I would strongly prefer staying with Autotools, and rather undust the VisualStudio port if necessary.

[AgentD]

A brief update: The recently released version 1.1.1 (Windows binaries: 64 bit mingw and 32 bit mingw) contain the sqfs_free function in libsquashfs.dll, which should solve the heap corruption issue on Windows.