search

DanBloomberg / leptonica

Leptonica is an open source library containing software that is broadly useful for image processing and image analysis applications. The official github repository for Leptonica is: danbloomberg/leptonica. See leptonica.org for more documentation.
Other
1.72k stars 384 forks source link

fix temporary directory handling on macOS #709

Closed benoit-pierre closed 10 months ago

benoit-pierre commented 11 months ago

Similarly to Windows, redirect /tmp/… paths accesses to the correct folder on macOS, fixing running the testsuite on the later.

stweil commented 11 months ago

See previous pull request #384 and related discussion.

Which parts of the test suite fail in your tests?

stweil commented 11 months ago

According to my tests, there is a regression which was introduced in @rsesek's pull request #583 by commit 05ff50e097cb3f65e4a8b603db2cbaf61336c98d. That commit changed makeTempDirname for macOS. Reverting the commit partially with this patch fixes two failing tests:

diff --git a/src/utils2.c b/src/utils2.c
index aabf2886..2c9a05f4 100644
--- a/src/utils2.c
+++ b/src/utils2.c
@@ -3195,7 +3195,7 @@ size_t   pathlen;

     memset(result, 0, nbytes);

-#ifdef __APPLE__
+#ifdef OS_IOS
     {
         size_t n = confstr(_CS_DARWIN_USER_TEMP_DIR, result, nbytes);
         if (n == 0) {

Leptonica must either stick to using /tmp for macOS everywhere. Or it could use confstr(_CS_DARWIN_USER_TEMP_DIR, tmpdir, PATH_MAX) not only for iOS, but also for macOS everywhere. But as I already explained in my comment to PR #384, that is risky because it generates a lengthy directory name which might cause buffer overruns or other failures in code which does not expect a long path for temporary files and directories.

The current mix of both alternatives causes the failing tests.

stweil commented 11 months ago

There are more risks when using _CS_DARWIN_USER_TEMP_DIR, see its documentation: "Provides the path to a user's temporary items directory. The directory will be created it if does not already exist. This directory is created with access permissions of 0700 and restricted by the umask(2) of the calling process and is a good location for temporary files. By default, files in this location may be cleaned (removed) by the system if they are not accessed in 3 days."

So that alternative uses a temporary directory which can only be accessed by its owner, and any file there can be cleaned by the system. /tmp can be accessed by all users, and the system won't clean any files there while it is running (Linux systems typically clean all files in /tmp at system start).

So I see three reasons why Leptonica should use /tmp on macOS. @benoit-pierre, @rsesek, which arguments remain to use _CS_DARWIN_USER_TEMP_DIR?

benoit-pierre commented 11 months ago

Which parts of the test suite fail in your tests?

Running #708 without the fix or parallel testing, here are the failures with the cmake workflow:

+ ctest --test-dir build --build-config Release --output-on-failure
Internal ctest changing into directory: /Users/runner/work/leptonica/leptonica/build
Test project /Users/runner/work/leptonica/leptonica/build
        Start   1: adaptmap_reg
  1/147 Test   #1: adaptmap_reg .....................   Passed    1.35 sec
[…]
        Start  39: compare_reg
 39/147 Test  #39: compare_reg ......................***Failed    0.99 sec

////////////////////////////////////////////////
////////////////   compare_reg   ///////////////
////////////////////////////////////////////////
leptonica-1.84.0 : libgif 5.2.1 : libjpeg 8d (libjpeg-turbo 3.0.0) : libpng 1.6.40 : libtiff 4.5.1 : zlib 1.2.11 : libwebp 1.3.1 : libopenjp2 2.5.0
delta cx = 32, delta cy = 12
delx = 32, dely = 12, score =  0.9999
Leptonica Error in fopenWriteStream: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/correl_5.png
Leptonica Error in l_binaryWrite: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/correl_5.png
Leptonica Error in fopenReadStream: file not found: correl_5.png
Leptonica Error in l_binaryRead: file stream not opened: /tmp/lept/regout/correl_5.png
Error in fileCopy: data not returned
Level 3: delx = 6, dely = -3, score =  0.7780
Level 2: delx = 11, dely = -6, score =  0.8651
Level 1: delx = 22, dely = -12, score =  0.8326
Level 0: delx = 45, dely = -25, score =  0.9795

concatenating ... done
delx = 45, dely = -25
Leptonica Error in fopenWriteStream: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/correl.pdf
Leptonica Error in l_binaryWrite: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/correl.pdf
Leptonica Error in fopenWriteStream: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/compare.pdf
Leptonica Error in l_binaryWrite: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/compare.pdf
Leptonica Error in fopenReadStream: file not found: compare.pdf
Leptonica Error in l_binaryRead: file stream not opened: /tmp/lept/regout/compare.pdf
Error in fileCopy: data not returned
Leptonica Error in fopenReadStream: file not found: correl.pdf
Leptonica Error in l_binaryRead: file stream not opened: /tmp/lept/regout/correl.pdf
Error in fileCopy: data not returned
Fraction of color pixels = 0.061252
Fraction of grayscale pixels = 0.046928
Time:   0.393 sec

////////////////////////////////////////////////
////////////////   compare_reg   ///////////////
////////////////////////////////////////////////
leptonica-1.84.0 : libgif 5.2.1 : libjpeg 8d (libjpeg-turbo 3.0.0) : libpng 1.6.40 : libtiff 4.5.1 : zlib 1.2.11 : libwebp 1.3.1 : libopenjp2 2.5.0
delta cx = 32, delta cy = 12
delx = 32, dely = 12, score =  0.9999
Leptonica Error in fopenWriteStream: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/correl_5.png
Leptonica Error in l_binaryWrite: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/correl_5.png
Leptonica Error in fopenReadStream: file not found: correl_5.png
Leptonica Error in findFileFormat: image file not found: /tmp/lept/regout/correl_5.png
Leptonica Error in fopenReadStream: file not found: correl_5.png
Leptonica Error in nbytesInFile: stream not opened: /tmp/lept/regout/correl_5.png
Leptonica Error in fopenReadStream: file not found: compare_golden.02.png
Leptonica Error in nbytesInFile: stream not opened: /tmp/lept/golden/compare_golden.02.png
Leptonica Error in fopenReadStream: file not found: correl_5.png
Leptonica Error in l_binaryRead: file stream not opened: /tmp/lept/regout/correl_5.png
Error in filesAreIdentical: array1 not read
Failure in compare_reg, index 2: comparing /tmp/lept/regout/correl_5.png with /tmp/lept/golden/compare_golden.02.png
Level 3: delx = 6, dely = -3, score =  0.7780
Level 2: delx = 11, dely = -6, score =  0.8651
Level 1: delx = 22, dely = -12, score =  0.8326
Level 0: delx = 45, dely = -25, score =  0.9795

concatenating ... done
delx = 45, dely = -25
Leptonica Error in fopenWriteStream: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/correl.pdf
Leptonica Error in l_binaryWrite: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/correl.pdf
Leptonica Error in fopenWriteStream: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/compare.pdf
Leptonica Error in l_binaryWrite: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/compare.pdf
Leptonica Error in fopenReadStream: file not found: compare.pdf
Leptonica Error in findFileFormat: image file not found: /tmp/lept/regout/compare.pdf
Leptonica Error in fopenReadStream: file not found: compare.pdf
Leptonica Error in nbytesInFile: stream not opened: /tmp/lept/regout/compare.pdf
Leptonica Error in fopenReadStream: file not found: compare_golden.05.pdf
Leptonica Error in nbytesInFile: stream not opened: /tmp/lept/golden/compare_golden.05.pdf
Leptonica Error in fopenReadStream: file not found: compare.pdf
Leptonica Error in l_binaryRead: file stream not opened: /tmp/lept/regout/compare.pdf
Error in filesAreIdentical: array1 not read
Failure in compare_reg, index 5: comparing /tmp/lept/regout/compare.pdf with /tmp/lept/golden/compare_golden.05.pdf
Leptonica Error in fopenReadStream: file not found: correl.pdf
Leptonica Error in findFileFormat: image file not found: /tmp/lept/regout/correl.pdf
Leptonica Error in fopenReadStream: file not found: correl.pdf
Leptonica Error in nbytesInFile: stream not opened: /tmp/lept/regout/correl.pdf
Leptonica Error in fopenReadStream: file not found: compare_golden.06.pdf
Leptonica Error in nbytesInFile: stream not opened: /tmp/lept/golden/compare_golden.06.pdf
Leptonica Error in fopenReadStream: file not found: correl.pdf
Leptonica Error in l_binaryRead: file stream not opened: /tmp/lept/regout/correl.pdf
Error in filesAreIdentical: array1 not read
Failure in compare_reg, index 6: comparing /tmp/lept/regout/correl.pdf with /tmp/lept/golden/compare_golden.06.pdf
Fraction of color pixels = 0.061252
Fraction of grayscale pixels = 0.046928
Time:   0.393 sec

        Start  40: compfilter_reg
 40/147 Test  #40: compfilter_reg ...................   Passed    0.74 sec
[…]
        Start 121: rectangle_reg
121/147 Test #121: rectangle_reg ....................***Failed    2.07 sec

////////////////////////////////////////////////
////////////////   rectangle_reg   ///////////////
////////////////////////////////////////////////
leptonica-1.84.0 : libgif 5.2.1 : libjpeg 8d (libjpeg-turbo 3.0.0) : libpng 1.6.40 : libtiff 4.5.1 : zlib 1.2.11 : libwebp 1.3.1 : libopenjp2 2.5.0
Info in pixFindRectangleInCC: Writing debug files to /tmp/lept/rect/
Leptonica Error in fopenWriteStream: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.02.png
Leptonica Error in l_binaryWrite: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.02.png
Leptonica Error in fopenReadStream: file not found: rectangle.02.png
Leptonica Error in l_binaryRead: file stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.02.png
Error in fileCopy: data not returned
Info in pixFindRectangleInCC: Writing debug files to /tmp/lept/rect/
Leptonica Error in fopenWriteStream: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.03.png
Leptonica Error in l_binaryWrite: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.03.png
Leptonica Error in fopenReadStream: file not found: rectangle.03.png
Leptonica Error in l_binaryRead: file stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.03.png
Error in fileCopy: data not returned
Info in pixFindRectangleInCC: Writing debug files to /tmp/lept/rect/
Leptonica Error in fopenWriteStream: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.04.png
Leptonica Error in l_binaryWrite: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.04.png
Leptonica Error in fopenReadStream: file not found: rectangle.04.png
Leptonica Error in l_binaryRead: file stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.04.png
Error in fileCopy: data not returned
Info in pixFindRectangleInCC: Writing debug files to /tmp/lept/rect/
Leptonica Error in fopenWriteStream: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.05.png
Leptonica Error in l_binaryWrite: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.05.png
Leptonica Error in fopenReadStream: file not found: rectangle.05.png
Leptonica Error in l_binaryRead: file stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.05.png
Error in fileCopy: data not returned
Info in pixFindRectangleInCC: Writing debug files to /tmp/lept/rect/
Leptonica Error in fopenWriteStream: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.06.png
Leptonica Error in l_binaryWrite: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.06.png
Leptonica Error in fopenReadStream: file not found: rectangle.06.png
Leptonica Error in l_binaryRead: file stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.06.png
Error in fileCopy: data not returned
Info in pixFindRectangleInCC: Writing debug files to /tmp/lept/rect/
Leptonica Error in fopenWriteStream: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.07.png
Leptonica Error in l_binaryWrite: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.07.png
Leptonica Error in fopenReadStream: file not found: rectangle.07.png
Leptonica Error in l_binaryRead: file stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.07.png
Error in fileCopy: data not returned
Info in pixFindRectangleInCC: Writing debug files to /tmp/lept/rect/
Leptonica Error in fopenWriteStream: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.08.png
Leptonica Error in l_binaryWrite: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.08.png
Leptonica Error in fopenReadStream: file not found: rectangle.08.png
Leptonica Error in l_binaryRead: file stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.08.png
Error in fileCopy: data not returned
Info in pixFindRectangleInCC: Writing debug files to /tmp/lept/rect/
Leptonica Error in fopenWriteStream: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.09.png
Leptonica Error in l_binaryWrite: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.09.png
Leptonica Error in fopenReadStream: file not found: rectangle.09.png
Leptonica Error in l_binaryRead: file stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.09.png
Error in fileCopy: data not returned
Time:   0.931 sec

////////////////////////////////////////////////
////////////////   rectangle_reg   ///////////////
////////////////////////////////////////////////
leptonica-1.84.0 : libgif 5.2.1 : libjpeg 8d (libjpeg-turbo 3.0.0) : libpng 1.6.40 : libtiff 4.5.1 : zlib 1.2.11 : libwebp 1.3.1 : libopenjp2 2.5.0
Info in pixFindRectangleInCC: Writing debug files to /tmp/lept/rect/
Leptonica Error in fopenWriteStream: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.02.png
Leptonica Error in l_binaryWrite: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.02.png
Leptonica Error in fopenReadStream: file not found: rectangle.02.png
Leptonica Error in findFileFormat: image file not found: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.02.png
Leptonica Error in fopenReadStream: file not found: rectangle.02.png
Leptonica Error in nbytesInFile: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.02.png
Leptonica Error in fopenReadStream: file not found: rectangle_golden.01.png
Leptonica Error in nbytesInFile: stream not opened: /tmp/lept/golden/rectangle_golden.01.png
Leptonica Error in fopenReadStream: file not found: rectangle.02.png
Leptonica Error in l_binaryRead: file stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.02.png
Error in filesAreIdentical: array1 not read
Failure in rectangle_reg, index 1: comparing /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.02.png with /tmp/lept/golden/rectangle_golden.01.png
Info in pixFindRectangleInCC: Writing debug files to /tmp/lept/rect/
Leptonica Error in fopenWriteStream: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.03.png
Leptonica Error in l_binaryWrite: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.03.png
Leptonica Error in fopenReadStream: file not found: rectangle.03.png
Leptonica Error in findFileFormat: image file not found: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.03.png
Leptonica Error in fopenReadStream: file not found: rectangle.03.png
Leptonica Error in nbytesInFile: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.03.png
Leptonica Error in fopenReadStream: file not found: rectangle_golden.02.png
Leptonica Error in nbytesInFile: stream not opened: /tmp/lept/golden/rectangle_golden.02.png
Leptonica Error in fopenReadStream: file not found: rectangle.03.png
Leptonica Error in l_binaryRead: file stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.03.png
Error in filesAreIdentical: array1 not read
Failure in rectangle_reg, index 2: comparing /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.03.png with /tmp/lept/golden/rectangle_golden.02.png
Info in pixFindRectangleInCC: Writing debug files to /tmp/lept/rect/
Leptonica Error in fopenWriteStream: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.04.png
Leptonica Error in l_binaryWrite: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.04.png
Leptonica Error in fopenReadStream: file not found: rectangle.04.png
Leptonica Error in findFileFormat: image file not found: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.04.png
Leptonica Error in fopenReadStream: file not found: rectangle.04.png
Leptonica Error in nbytesInFile: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.04.png
Leptonica Error in fopenReadStream: file not found: rectangle_golden.03.png
Leptonica Error in nbytesInFile: stream not opened: /tmp/lept/golden/rectangle_golden.03.png
Leptonica Error in fopenReadStream: file not found: rectangle.04.png
Leptonica Error in l_binaryRead: file stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.04.png
Error in filesAreIdentical: array1 not read
Failure in rectangle_reg, index 3: comparing /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.04.png with /tmp/lept/golden/rectangle_golden.03.png
Info in pixFindRectangleInCC: Writing debug files to /tmp/lept/rect/
Leptonica Error in fopenWriteStream: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.05.png
Leptonica Error in l_binaryWrite: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.05.png
Leptonica Error in fopenReadStream: file not found: rectangle.05.png
Leptonica Error in findFileFormat: image file not found: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.05.png
Leptonica Error in fopenReadStream: file not found: rectangle.05.png
Leptonica Error in nbytesInFile: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.05.png
Leptonica Error in fopenReadStream: file not found: rectangle_golden.04.png
Leptonica Error in nbytesInFile: stream not opened: /tmp/lept/golden/rectangle_golden.04.png
Leptonica Error in fopenReadStream: file not found: rectangle.05.png
Leptonica Error in l_binaryRead: file stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.05.png
Error in filesAreIdentical: array1 not read
Failure in rectangle_reg, index 4: comparing /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.05.png with /tmp/lept/golden/rectangle_golden.04.png
Info in pixFindRectangleInCC: Writing debug files to /tmp/lept/rect/
Leptonica Error in fopenWriteStream: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.06.png
Leptonica Error in l_binaryWrite: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.06.png
Leptonica Error in fopenReadStream: file not found: rectangle.06.png
Leptonica Error in findFileFormat: image file not found: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.06.png
Leptonica Error in fopenReadStream: file not found: rectangle.06.png
Leptonica Error in nbytesInFile: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.06.png
Leptonica Error in fopenReadStream: file not found: rectangle_golden.05.png
Leptonica Error in nbytesInFile: stream not opened: /tmp/lept/golden/rectangle_golden.05.png
Leptonica Error in fopenReadStream: file not found: rectangle.06.png
Leptonica Error in l_binaryRead: file stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.06.png
Error in filesAreIdentical: array1 not read
Failure in rectangle_reg, index 5: comparing /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.06.png with /tmp/lept/golden/rectangle_golden.05.png
Info in pixFindRectangleInCC: Writing debug files to /tmp/lept/rect/
Leptonica Error in fopenWriteStream: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.07.png
Leptonica Error in l_binaryWrite: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.07.png
Leptonica Error in fopenReadStream: file not found: rectangle.07.png
Leptonica Error in findFileFormat: image file not found: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.07.png
Leptonica Error in fopenReadStream: file not found: rectangle.07.png
Leptonica Error in nbytesInFile: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.07.png
Leptonica Error in fopenReadStream: file not found: rectangle_golden.06.png
Leptonica Error in nbytesInFile: stream not opened: /tmp/lept/golden/rectangle_golden.06.png
Leptonica Error in fopenReadStream: file not found: rectangle.07.png
Leptonica Error in l_binaryRead: file stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.07.png
Error in filesAreIdentical: array1 not read
Failure in rectangle_reg, index 6: comparing /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.07.png with /tmp/lept/golden/rectangle_golden.06.png
Info in pixFindRectangleInCC: Writing debug files to /tmp/lept/rect/
Leptonica Error in fopenWriteStream: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.08.png
Leptonica Error in l_binaryWrite: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.08.png
Leptonica Error in fopenReadStream: file not found: rectangle.08.png
Leptonica Error in findFileFormat: image file not found: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.08.png
Leptonica Error in fopenReadStream: file not found: rectangle.08.png
Leptonica Error in nbytesInFile: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.08.png
Leptonica Error in fopenReadStream: file not found: rectangle_golden.07.png
Leptonica Error in nbytesInFile: stream not opened: /tmp/lept/golden/rectangle_golden.07.png
Leptonica Error in fopenReadStream: file not found: rectangle.08.png
Leptonica Error in l_binaryRead: file stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.08.png
Error in filesAreIdentical: array1 not read
Failure in rectangle_reg, index 7: comparing /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.08.png with /tmp/lept/golden/rectangle_golden.07.png
Info in pixFindRectangleInCC: Writing debug files to /tmp/lept/rect/
Leptonica Error in fopenWriteStream: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.09.png
Leptonica Error in l_binaryWrite: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.09.png
Leptonica Error in fopenReadStream: file not found: rectangle.09.png
Leptonica Error in findFileFormat: image file not found: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.09.png
Leptonica Error in fopenReadStream: file not found: rectangle.09.png
Leptonica Error in nbytesInFile: stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.09.png
Leptonica Error in fopenReadStream: file not found: rectangle_golden.08.png
Leptonica Error in nbytesInFile: stream not opened: /tmp/lept/golden/rectangle_golden.08.png
Leptonica Error in fopenReadStream: file not found: rectangle.09.png
Leptonica Error in l_binaryRead: file stream not opened: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.09.png
Error in filesAreIdentical: array1 not read
Failure in rectangle_reg, index 8: comparing /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/lept/regout/rectangle.09.png with /tmp/lept/golden/rectangle_golden.08.png
Time:   0.929 sec

        Start 122: rotate1_reg
122/147 Test #122: rotate1_reg ......................   Passed    7.95 sec
[…]
        Start 147: webpio_reg
147/147 Test #147: webpio_reg .......................   Passed    3.10 sec

99% tests passed, 2 tests failed out of 147

Note: I don't have access to a macOS machine.

benoit-pierre commented 11 months ago

Leptonica must either stick to using /tmp for macOS everywhere. Or it could use confstr(_CS_DARWIN_USER_TEMP_DIR, tmpdir, PATH_MAX) not only for iOS, but also for macOS everywhere. […] The current mix of both alternatives causes the failing tests.

Agreed.

benoit-pierre commented 11 months ago

There are more risks when using _CS_DARWIN_USER_TEMP_DIR, see its documentation: "Provides the path to a user's temporary items directory. The directory will be created it if does not already exist. This directory is created with access permissions of 0700 and restricted by the umask(2) of the calling process and is a good location for temporary files. By default, files in this location may be cleaned (removed) by the system if they are not accessed in 3 days."

Except for that last part, all good things I would say. For that last part: are the files removed when the system is running? Also note the wording: "may be", so that might be conditional on other conditions (e.g. low disk space).

rsesek commented 11 months ago

which arguments remain to use _CS_DARWIN_USER_TEMP_DIR?

Sandboxed macOS applications, which all modern Mac apps should be and which is also a requirement for submission into the Mac App Store, cannot access /tmp. Even for non-sandboxed applications, the preferred approach on macOS is to use the secure, per-user temporary directory. Both of those reasons are documented for the change in #583.

This is corroborated by Table 1-1 in the Mac App Programming Guide. That suggests using NSTemporaryDirectory() to acquire the path, but that requires linking against the Foundation framework. The same directory returned by that function is returned by the confstr() call. The preference for the secure temp dir is also demonstrated by the fact that $TMPDIR points to the same location.

I do not think Leptonica should revert to using /tmp on macOS. The tests should be fixed to use the same temp dir location as the library.

There are more risks when using _CS_DARWIN_USER_TEMP_DIR, see its documentation:

The same "risks" apply to /tmp, which is also periodically cleaned, per the configuration in /etc/defaults/periodic.conf.

it generates a lengthy directory name which might cause buffer overruns or other failures in code which does not expect a long path for temporary files and directories.

I personally don't think this is a good reason for a library to not do the functionally correct thing.

stweil commented 11 months ago

Thank you for taking the time and explaining your arguments.

The same "risks" apply to /tmp, which is also periodically cleaned, per the configuration in /etc/defaults/periodic.conf.

So auto-clean is identical for both alternatives, and you don't think that the risk of breaking existing code because of the longer path (48 characters instead of 4) is a good reason why Leptonica should stick to /tmp. What about the third risk, the different accessibility?

stweil commented 11 months ago

By the way, I noticed that even Apple and Google still use /tmp, at least for xquartz and keystone.

rsesek commented 11 months ago

What about the third risk, the different accessibility?

The difference in accessibility is the purpose of the secure, per-user temp directory and is the desired property. Why should these temp files be made available to all users on the system? System-wide temp directories are generally not desirable from security or privacy perspectives, which is why macOS introduced this feature.

By the way, I noticed that even Apple and Google still use /tmp, at least for xquartz and keystone.

Keystone is a legacy codebase, and the product that is replacing it, the Chromium Updater, uses the modern temp dir location. And most Apple software does too, because it obtains the temp dir path from the Foundation framework, on which most Mac software is based.

DanBloomberg commented 11 months ago

Looking at the failures in the reg tests (only 2 out of 147), which were in compare_reg and rectangle_reg: they all involved the function lept_cp() or lept_mv(), that was doing a copy or a move from a named file in /tmp/lept/...

And those are the only such calls in the regression test suite.

I can't tell for sure, but my guess is that the files are not being written into the specified /tmp/lept/... path. Instead, the path is rewritten and the lept_cp() and lept_mv() fail because they are using the hardcoded paths.

stweil commented 11 months ago

I can't tell for sure, but my guess is that the files are not being written into the specified /tmp/lept/... path. Instead, the path is rewritten and the lept_cp() and lept_mv() fail because they are using the hardcoded paths.

Dan, both lept_cp and lept_mv use makeTempDirname for the target directory, and that's the function which no longer uses /tmp for macOS since pull request #583 was merged. That merge caused the test regression because now the copied or moved file is in /var/folders/..., but it is is expected in /tmp/...

DanBloomberg commented 10 months ago

It would be nice to resolve this. Here are my thoughts:

(1) I like putting all the rewriting in genPathname() (2) I like having makeTempFilename() be simple and rely on genPathname() to handle all rewrites. (3) We need to fix regression tests compare_reg and rectangle_reg for macOS (4) With (1) and (2) the best way to fix the regression tests is to fix lept_mv() and lept_cp() (and also lept_rm()). People writing these programs and other applications shouldn't need to consider the temp file rewrites. (5) For example, we would simpiify lept_cp() by using genPathname() to get both srcpath and newpath (as with current windows code), and then using platform-specific functions for copying. I believe the lept_rm() would be OK as is.

Consequently my inclination is to merge 709 if it's not too risky in terms of running up against small buffers. If we do that, I'll make a stab at simplifying the lept_* functions, and Stefan can fix my errors because he can test on macOS.

stweil commented 10 months ago

Consequently my inclination is to merge 709

I think that this pull request contains new code which is not needed. The initial version of #384 already covered iOS and macOS and was modified to handle iOS only because I thought that the risk of getting longer path names for temporary files should be avoided. If we now accept that risk, it is sufficient to enable the iOS code for macOS, too.

DanBloomberg commented 10 months ago

OK, I admit this is far above my pay grade.

I see your comment that changing line.3195 to only use iOS will fix the two tests for macOS, and is it true that we shouldn't care about testing on iOS?

As you can see from my comments above, although I do not understand the level of risk for apps that use small buffers (for example, would any supported version of tesseract fail?), I do like the simplicity and consistency of the code that results from this PR.

DanBloomberg commented 10 months ago

@stweil

I understand your concern. Here is a compromise. We keep the functionality as is for macOS (using "/tmp" without rewrite) and have the Darwin rewrite only on iOS, as you suggest. We also adopt the reorganization in this PR that puts all rewriting directly into genPathname() and simplifies makeTempDirname(), lept_cp() and lept_mv().

I've attached my proposed utils2.c: utils2.c.gz

stweil commented 10 months ago

I am closing this pull request because it has been superseded by PR #713.