Closed bkocev closed 5 months ago
Hi @bkocev
Thanks for your question.
This wouldn't be a web
ticket, I am moving this to the conan
repo, though it might need to be moved to the conan-center-index
repo if this results to be a recipe issue.
We would need something to reproduce. Could you please share the following?:
conanfile
with the minimal to reproduceconan install
command and output (including the Conan version)CMakeLists.txt
+ main.cpp
cmake
exact commands used.Thanks very much!
Hi,
I am not sure I can provide with all the files quickly. This is how I configure the libvips:
conan_cmake_configure( libtiff/4.6.0 libvips/8.14.2 libwebp/1.3.2 GENERATORS cmake_find_package IMPORTS "bin, *.dll -> ${PROJECT_BINARY_DIR}/bin/" )
conan --version Conan version 1.62.0
If I manage to prepare all the files you asked for, I will get back to you.
Best, Bojan.
Some quick feedback about it:
cmake_find_package
generator is legacy, even in 1.X, and it is recommended to move to the CMakeDeps
generatorimports
approach is legacy, it is recommended to move to explicit copy()
in the generate()
method, though for dlls
, it might be recommended to use environment generators like VirtualRunEnv
libvips
might stop being compatible with Conan 1.X some time in the following monthsLooking forward to getting your feedback and files to reproduce.
Hi,
I finally managed to prepare a conan project: https://drive.google.com/file/d/1O44xXwru8to9LgXVWLRMToOlMzVvjJLB/view?usp=drive_link. I was not able to upload the zip because it is too big because of the image therein.
I followed the following example https://docs.conan.io/2/tutorial/consuming_packages/build_simple_cmake_project.html. I tested the program on Ubuntu. The command conan profile detect --force created a default profile in Release mode. After doing all the commands and building the project in Release and then running it I got a segmentation fault.
However, after editing the conan default profile to be in Debug mode and then repeating all the commands and building the project in Debug mode, I was able to create a correct pyramidal file. Then repeating it again in Release mode, I was able to create a correct pyramidal file. So, I am guessing there is some issue with the release version of the libvips library in conan, but I am not sure.
Maybe, you can have a look at the problem using the files I provided above and try to make better sense of the potential problem.
Thank you and kind regards, Bojan.
P.S. The original multi-resolution data is available in the demo data: https://openslide.org/demo/ Hamamatsu NDPI -> OS-1.
Thanks for sharing your feedback.
I have tried to check
I finally managed to prepare a conan project: https://drive.google.com/file/d/1O44xXwru8to9LgXVWLRMToOlMzVvjJLB/view?usp=drive_link. I was not able to upload the zip because it is too big because of the image therein.,
But it seems that file in google drive is private, and need access. We usually recommend to put things in a Github repo instead, it is easy and very convenient, because we can do Pull Requests, or do code reviews directly here, so much better than in a google drive zip.
Sorry about this, what about this link https://drive.google.com/file/d/1O44xXwru8to9LgXVWLRMToOlMzVvjJLB/view?usp=sharing
I will see if I can put it later on github.
Nop, sorry, too big (almost 1Gb) to be scanned, so download now is public but blocked by security. If you can put it later in github, let me know.
Sorry, here is the github repo with the code https://github.com/bkocev/conan-libvips-problem. And the image file is available here : https://drive.google.com/file/d/1lkezyFyQn14Khwq2kbDIjPB5RF1-3Hbq/view?usp=sharing
I was not able to upload the image file on github as it is too big.
Hi, were you able to get access to the code and the image file?
Thank you, Bojan.
Thanks for the ping.
I have reproduced the steps in https://github.com/bkocev/conan-libvips-problem/blob/main/run_example.sh in the conanio/gcc11
docker image, using Conan 2.0, and it worked fine:
conan@61aba5f145c1:~/conanws/kk/conan-libvips-problem/build$ ./PyramidalImagesConverter ../hamamatsu-os-1-as-singleres.tiff
Successfully loaded the image.
VIPS works!!!
I'll try with 1.62 too
It seems the project you put together is for 2.0 only, it is not working with 1.62 (it would need a different [layout], for example, and different commands). So it seems that you also did it with Conan 2.0.
It would be great if you can test it in the docker image I suggest above, to see if you can reproduce it with it.
HI,
I am using Conan 1.62.0 on Windows, but I was not able to prepare an example project for you under Windows and hence I provided one with Conan 2.0 under Linux. I tried it on my native Ubuntu installation, i.e., not in a docker container. And, I was able to reproduce the problem as I described.
Best, Bojan.
I have tried in my Ubuntu 22.04 (WSL) not docker, and it works too:
./PyramidalImagesConverter ../hamamatsu-os-1-as-singleres.tiff
Successfully loaded the image.
VIPS works!!!
It could be something related to having an older glibc in your specific ubuntu (but I wouldn't know in Windows, this shouldn't be the case). What ubuntu are you using? If using an older one, it is worth trying:
--build=*
Hi,
I am using Ubuntu 22.04.3 LTS. I could reproduce the problem only once on Ubuntu, as described above. However, after editing the conan default profile to be in Debug mode and then repeating all the commands and building the project in Debug mode, I was able to create a correct pyramidal file. Then repeating it again in Release mode, I was able to create a correct pyramidal file. So, my guess was that there is some issue with the release version of the libvips library in conan, but I am not sure. At the moment, the problem is that you cannot reproduce the issue I have and I do not know what to suggest. It would be nice if you can try it on Windows too. I could not get the exampleto work on Windows though.
Best, Bojan.
I have done the steps in Windows, and it seems to run:
(conanpip2_310) λ PyramidalImagesConverter ../hamamatsu-os-1-as-singleres.tiff
Successfully loaded the image.
And I can see the file hamamatsu-os-1-as-singleres.tiff_multires
which is double the size.
It is true that the message printf("VIPS works!!!\n");
is not printed. It would seem that the program is not finalizing correctly.
This kind of runtime error can be quite challenging to understand where is coming from 😞
Maybe that could be a library issue? From the discussion in https://github.com/libvips/libvips/discussions/3828 it is not fully explicit that they are trying that in Windows?
Hi, this is Jonas, a colleague of Bojan.
I managed yesterday to build his example in release mode with MSVC 2022 on Windows. I do get an access violation after the program runs for some time. I tried three times, it always stops when the result file reaches exactly 2 GB file size. @jcupitt does this ring a bell to you? Maybe some configuration setting in the dependencies? I'll try to get a meaningful callstack and get back to you.
Oh, interesting @bdal-jsinge!
TIFF pyramid generation runs in two main phases: it renders every level in parallel to a set of separate TIFF files, then in a second pass gathers the $n TIFFs together into one pyramid image. Perhaps the gather step is not working correctly for >2GB files? That's the size limit for non-bigtiff TIFFs. I'll have a look.
Windows is a bit more complicated -- they have a separate API for large files which you need to enable at compile time. That could be missing on the windows build you are using.
I tried on ubuntu:
$ vips tiffsave C2023000589S1-0-1_743eee76-f53b-aca2-4da5-fa97a44d9933_125232.svs x.tif --tile --pyramid --compression jpeg --bigtiff --vips-progress
vips temp-5: 211213 x 87581 pixels, 32 threads, 128 x 128 tiles, 640 lines in buffer
vips temp-5: done in 217s
$ ls -l x.tif
-rw-r--r-- 1 john john 2610298042 Feb 17 15:42 x.tif
I examined the TIFF and it seems fine -- I could view every pixel in every level. I tried without the --bigtiff
flag and that worked too, I suppose libtiff have improved their handling of files in the 2gb to 4gb range since I'm pretty sure this used to fail.
My guess would be that conan is not building libtiff on windows with the large file backend enabled (but it's just a guess).
A simple test for this might be:
$ vips black x.tif 100000 100000 --vips-progress
vips temp-1: 100000 x 100000 pixels, 32 threads, 100000 x 1 tiles, 640 lines in buffer
vips temp-1: done in 11.7s
$ file x.tif
x.tif: Big TIFF image data, little-endian
$ vips avg x.tif --vips-progress
vips temp-1: 100000 x 100000 pixels, 32 threads, 100000 x 1 tiles, 640 lines in buffer
vips temp-1: done in 43.6s
0.000000
If that fails, then I think it's likely to be a misconfiguration of libtiff.
Yes, the test you suggested is failing for vips.exe from the conan windows package, already when trying to write the image:
C:\Users\jonas>C:\Users\jonas\.conan2\p\b\libvidf931674c7877\b\build-debug\tools\vips.exe black x.tif 100000 100000 --vips-progress vips.exe temp-0: 100000 x 100000 pixels, 20 threads, 100000 x 16 tiles, 384 lines in buffer vips.exe temp-0: done in 0.876s TIFFAppendToStrip: Maximum TIFF file size exceeded wbuffer_write: write failed memory: high-water mark 85.45 MB error buffer: TIFFAppendToStrip: Maximum TIFF file size exceeded wbuffer_write: write failed
I looked a little bit around for configuration options on the libtiff library, but it's not clear yet to me how to enable files larger than 2 GB for msvc builds. Found some documentation here but nothing specific how to enable large file support: https://libtiff.gitlab.io/libtiff/build.html#id2 Here's a configuration file that seems to enable it only for mingw or unix platforms: https://gitlab.com/libtiff/libtiff/-/blob/master/cmake/LargeFileSupport.cmake?ref_type=heads
I also don't see anything specific in the vcpkg recipe. But I also don't know if it would show the same problem or not: https://github.com/microsoft/vcpkg/blob/master/ports/tiff/portfile.cmake
Regarding the crash I observed: I guess it's a subsequent problem after the tiff library write failure. Can't tell whether it's worth fixing it, but here's some more info:
The crash happens due to nullpointer access in src\libvips\iofuncs\target.c:459, as VipsTargetClass *class is a nullpointer. It happens when releasing resources after the vips_sink_disc call fails src\libvips\foreign\vips2tiff.c:2285
Callstack looks like this:
PyramidalImagesConverter.exe!vips_target_write_unbuffered(_VipsTarget target, const void data, unsigned int64 length) Line 459 at packagePath\src\libvips\iofuncs\target.c(459) PyramidalImagesConverter.exe!vips_target_flush(_VipsTarget target) Line 488 at packagePath\src\libvips\iofuncs\target.c(488) PyramidalImagesConverter.exe!vips_target_seek(_VipsTarget target, long position, int whence) Line 584 at packagePath\src\libvips\iofuncs\target.c(584) PyramidalImagesConverter.exe!openout_target_seek(void * st, unsigned int64 offset, int whence) Line 228 at packagePath\src\libvips\foreign\tiff.c(228) PyramidalImagesConverter.exe!TIFFLinkDirectory(tiff tif) Line 2956 at tiffPackagePath\src\libtiff\tif_dirwrite.c(2956) PyramidalImagesConverter.exe!TIFFWriteDirectorySec(tiff tif, int isimage, int imagedone, unsigned __int64 pdiroff) Line 1098 at tiffPackagePath\src\libtiff\tif_dirwrite.c(1098) PyramidalImagesConverter.exe!TIFFWriteDirectory(tiff tif) Line 238 at tiffPackagePath\src\libtiff\tif_dirwrite.c(238) PyramidalImagesConverter.exe!TIFFRewriteDirectory(tiff tif) Line 317 at tiffPackagePath\src\libtiff\tif_dirwrite.c(317) PyramidalImagesConverter.exe!TIFFFlush(tiff tif) Line 50 at tiffPackagePath\src\libtiff\tif_flush.c(50) PyramidalImagesConverter.exe!TIFFCleanup(tiff tif) Line 51 at tiffPackagePath\src\libtiff\tif_close.c(51) PyramidalImagesConverter.exe!TIFFClose(tiff tif) Line 156 at tiffPackagePath\src\libtiff\tif_close.c(156) PyramidalImagesConverter.exe!layer_free(_Layer layer) Line 946 at packagePath\src\libvips\foreign\vips2tiff.c(946) PyramidalImagesConverter.exe!layer_free_all(_Layer layer) Line 961 at packagePath\src\libvips\foreign\vips2tiff.c(961) PyramidalImagesConverter.exe!layer_free_all(_Layer layer) Line 960 at packagePath\src\libvips\foreign\vips2tiff.c(960) PyramidalImagesConverter.exe!layer_free_all(_Layer layer) Line 960 at packagePath\src\libvips\foreign\vips2tiff.c(960) PyramidalImagesConverter.exe!layer_free_all(_Layer layer) Line 960 at packagePath\src\libvips\foreign\vips2tiff.c(960) PyramidalImagesConverter.exe!layer_free_all(_Layer layer) Line 960 at packagePath\src\libvips\foreign\vips2tiff.c(960) PyramidalImagesConverter.exe!layer_free_all(_Layer layer) Line 960 at packagePath\src\libvips\foreign\vips2tiff.c(960) PyramidalImagesConverter.exe!layer_free_all(_Layer layer) Line 960 at packagePath\src\libvips\foreign\vips2tiff.c(960) PyramidalImagesConverter.exe!layer_free_all(_Layer layer) Line 960 at packagePath\src\libvips\foreign\vips2tiff.c(960) PyramidalImagesConverter.exe!layer_free_all(_Layer layer) Line 960 at packagePath\src\libvips\foreign\vips2tiff.c(960) PyramidalImagesConverter.exe!layer_free_all(_Layer layer) Line 960 at packagePath\src\libvips\foreign\vips2tiff.c(960) PyramidalImagesConverter.exe!layer_free_all(_Layer layer) Line 960 at packagePath\src\libvips\foreign\vips2tiff.c(960) PyramidalImagesConverter.exe!wtiff_free(_Wtiff wtiff) Line 975 at packagePath\src\libvips\foreign\vips2tiff.c(975) PyramidalImagesConverter.exe!vips__tiff_write_target(_VipsImage input, _VipsTarget target, VipsForeignTiffCompression compression, int Q, VipsForeignTiffPredictor predictor, const char profile, int tile, int tile_width, int tile_height, int pyramid, int bitdepth, int miniswhite, VipsForeignTiffResunit resunit, double xres, double yres, int bigtiff, int rgbjpeg, int properties, int strip, VipsRegionShrink region_shrink, int level, int lossless, VipsForeignDzDepth depth, int subifd, int premultiply, int page_height) Line 2286 at packagePath\src\libvips\foreign\vips2tiff.c(2286) PyramidalImagesConverter.exe!vips_foreign_save_tiff_build(_VipsObject object) Line 213 at packagePath\src\libvips\foreign\tiffsave.c(213) PyramidalImagesConverter.exe!vips_foreign_save_tiff_file_build(_VipsObject object) Line 519 at packagePath\src\libvips\foreign\tiffsave.c(519) PyramidalImagesConverter.exe!vips_object_build(_VipsObject object) Line 367 at packagePath\src\libvips\iofuncs\object.c(367) PyramidalImagesConverter.exe!vips_cache_operation_buildp(_VipsOperation operation) Line 912 at packagePath\src\libvips\iofuncs\cache.c(912) PyramidalImagesConverter.exe!vips_call_required_optional(_VipsOperation operation, char required, char optional) Line 913 at packagePath\src\libvips\iofuncs\operation.c(913) PyramidalImagesConverter.exe!vips_call_by_name(const char operation_name, const char option_string, char required, char optional) Line 953 at packagePath\src\libvips\iofuncs\operation.c(953) PyramidalImagesConverter.exe!vips_call_split(const char operation_name, char optional, ...) Line 1057 at packagePath\src\libvips\iofuncs\operation.c(1057) PyramidalImagesConverter.exe!vips_tiffsave(_VipsImage in, const char filename, ...) Line 748 at packagePath\src\libvips\foreign\tiffsave.c(748) PyramidalImagesConverter.exe!main(int argc, char * argv) Line 33 at C:_DEV\conan-libvips-problem\src\main.c(33) [Inline Frame] PyramidalImagesConverter.exe!invoke_main() Line 78 at D:\a_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl(78) PyramidalImagesConverter.exe!__scrt_common_main_seh() Line 288 at D:\a_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl(288) kernel32.dll!BaseThreadInitThunk() ntdll.dll!RtlUserThreadStart()
libvips is opening the output TIFF in w8
mode, ie. bigtiff write. For some reason the conan libtiff build does not seem to support this (I think?) hence the failure.
It looks like libtiff last revised all this back in 4.0.5:
https://github.com/libsdl-org/libtiff/blob/master/doc/releases/v4.0.5.rst
Does any of that look relevant? Could conan be building for a very old MSVC runtime?
You could try to check that conan's tiffcp
has bigtiff working correctly, I see:
$ tiffcp -8 k2.tif x.tif
$ file x.tif
x.tif: Big TIFF image data, little-endian
$ file k2.tif
k2.tif: TIFF image data, little-endian, direntries=18, height=2048, bps=23318, compression=JPEG, PhotometricInterpretation=YCbCr, orientation=upper-left, width=1450
I think the big tiff format extension itself is enabled in the conan libtiff artifact.
By default, tiff-tools are not built, but I modified the recipe and have built it locally to get the tiffcp tool.
With a small tif file I get the bigtiff file from tiffcp:
tiffcp -8 -m 0 C:\_DEV\conan-libvips-problem\example.tiff tiffcp-900mb.tiff
file ~/tiffcp-900mb.tiff
/c/Users/jonas/tiffcp-900mb.tiff: Big TIFF image data, little-endian
and for a file of 20 GB size as well:
tiffcp -8 -m 0 C:\_DEV\conan-libvips-problem\in-20gb.tif tiffcp-20gb.tif
$ file ~/tiffcp-20gb.tif
/c/Users/jonas/tiffcp-20gb.tif: Big TIFF image data, little-endian
$ ls -lh ~/tiffcp-20gb.tif
-rw-r--r-- 1 Jonas 1049089 21G Feb 19 14:33 /c/Users/jonas/tiffcp-20gb.tif
Note that "file" is not available on windows natively, but I used the one from git bash.
How strange! Could you try:
vips.exe black x.tif[bigtiff] 100000 100000 --vips-progress
I fired up a windows VM and tried with the vips.exe
we distribute:
F:\vips-dev-8.15\bin>vips.exe black x.tif[bigtiff] 100000 100000 --vips-progress
vips.exe temp-1: 100000 x 100000 pixels, 8 threads, 100000 x 1 tiles, 256 lines in buffer
vips.exe temp-1: done in 3.36s
x.tif: write error
windows error: The printer is out of paper.
TIFFAppendToStrip: Write error at scanline 23552
wbuffer_write: write failed
So I suppose the libvips build system has this issue with libtiff configuration too. @kleisauke, could I ping you on this?
I was unable to reproduce this on Windows 11 using the prebuilt binaries available at https://github.com/libvips/build-win64-mxe. Tested both 32-bit and 64-bit versions.
PS> vips black x.tif[bigtiff] 100000 100000 --vips-progress
vips.exe temp-1: 100000 x 100000 pixels, 24 threads, 100000 x 1 tiles, 512 lines in buffer
vips.exe temp-1: done in 2.31s
PS> vipsheader x.tif
x.tif: 100000x100000 uchar, 1 band, b-w, tiffload
PS> "Size in bytes: {0}" -f (Get-Item "x.tif").Length
Size in bytes: 10000009716
Ooof, so sorry, my disk was filling up and that was stopping the write! I suppose the error codes are not being mapped into strings correctly -- it should be disk full
, not printer out of paper
.
Yes, it's all working for me too with the official libvips binaries, thank you @kleisauke
You can see how libvips is building libtiff here:
https://github.com/libvips/build-win64-mxe/blob/master/build/overrides.mk#L663-L683
That's using configure
and I think conan is using cmake
, so you'd need to investigate the cmake build script.
I suppose the error codes are not being mapped into strings correctly -- it should be
disk full
, notprinter out of paper
.
Indeed, that error message looks weird. It looks like the conversion takes place here: https://github.com/libvips/libvips/blob/v8.15.1/libvips/iofuncs/error.c#L301-L311
Perhaps we should use the portable and thread-safe g_strerror()
function there? That one uses strerror_s()
on Windows:
https://github.com/GNOME/glib/blob/2.79.2/glib/gstrfuncs.c#L1305
Yes, I wonder why we didn't use g_strerror()
? That must be very old code.
I changed it: https://github.com/libvips/libvips/commit/ea86a0d03b05c3146b93ed210b835188334bde8a
How strange! Could you try:
vips.exe black x.tif[bigtiff] 100000 100000 --vips-progress
With vips.exe from the conan package:
C:\Users\jonas.singe>C:\Users\jonas\.conan2\p\b\libvidf931674c7877\b\build-debug\tools\vips.exe black x.tif[bigtiff] 100000 100000 --vips-progress
vips.exe temp-0: 100000 x 100000 pixels, 20 threads, 100000 x 16 tiles, 384 lines in buffer
vips.exe temp-0: done in 0.933s
TIFFAppendToStrip: Maximum TIFF file size exceeded
wbuffer_write: write failed
memory: high-water mark 85.45 MB
error buffer: TIFFAppendToStrip: Maximum TIFF file size exceeded
wbuffer_write: write failed
Regarding your question which MSVC runtime is used in the conan distribution: I checked the dependencies of vips.exe distributed via conan, it lists C:\Windows\System32\vcruntime140.dll
Where vips.exe and libtiff-6.dll from the vips distribution list C:\Windows\System32\ucrtbase.dll
Thank you for testing @bdal-jsinge. So it certainly seems like an error in the conan libtiff package. I would find out who maintains that, and ask if they have a view on this.
Thanks for all the input! Will look for somebody who can provide more input on the libtiff artifact.
What really puzzles me is why the tiffcp command doesn't seem to have a problem with large files. Do you have any idea what might cause the difference between vips.exe and tiffcp.exe?
Sorry, I'm puzzled by that too :(
While debugging this, I noticed this line: https://github.com/libvips/libvips/blob/588e7ffdb17e1431618c85f18c3f4ab14d9f333a/libvips/iofuncs/target.c#L215
I think that should be calling vips__seek()
or vips__seek_no_error()
to ensure it uses _lseeki64()
on Windows, similar to how it's done here:
https://github.com/libvips/libvips/blob/588e7ffdb17e1431618c85f18c3f4ab14d9f333a/libvips/iofuncs/source.c#L340-L345
This is not a issue on MinGW-w64 since that just aliases lseek
to lseek64
:
https://github.com/mingw-w64/mingw-w64/blob/57f796c80bfac3c75725e4e7a086afe43968b3ae/mingw-w64-headers/crt/io.h#L352
@jcupitt Should I open a PR for this targeting 8.15?
@kleisauke great find! That could be it -- tiff write needs seek, annoyingly.
Yes, please make a PR.
I just opened PR https://github.com/libvips/libvips/pull/3864 for this.
Summary: it probably wasn't a libtiff build issue, instead it was a stray off_t
in libvips (discovered by Kleis). This fix will be in libvips 8.15.2.
Thank you very much for the fruitful discussion. :) When can we expect the release of libvips 8.15.2?
Best, Bojan.
Summary: it probably wasn't a libtiff build issue, instead it was a stray
off_t
in libvips (discovered by Kleis). This fix will be in libvips 8.15.2.
I can confirm that this fixes the issue in the conan/windows/msvc environment: I have built a local conan package with kleisaukes changes and indeed this allows to write large files. The commands from above vips black ... vips tiffsave ... now all work as expected.
Thanks a lot!
Ticket can be closed, issue is fixed in vips 8.15.2 which is available now on conan
Excellent, thanks very much all for the feedback and support!
Dear all,
when using the C api of libvips/8.15.1, the conversion to pyramidal files does not work at all using the function:
vips_tiffsave( inputImage, filename, "pyramid", true, "tile", true, "compression", VIPS_FOREIGN_TIFF_COMPRESSION_LZW, "bigtiff", true, NULL )
while when using libvips/8.14.2 it seems to work for "smaller" images (e.g., 13811 X 6122). But for "larger" images (e.g., 122880 X 110592) images, the conversion to a pyramidal file fails with this error:
VIPS:ERROR:../src/libvips/iofuncs/target.c:482:vips_target_flush: assertion failed: (target->write_point >= 0) Bail out! VIPS:ERROR:../src/libvips/iofuncs/target.c:482:vips_target_flush: assertion failed: (target->write_point >= 0)
We suspect there is an issue with the conan build. Please see the discussion at the libvips forum: https://github.com/libvips/libvips/discussions/3828
Is it possible to fix this error with the conan build of libvips?
Thank you, Bojan.