oss-fuzz 35265: graphicsmagick:coder_PGX_fuzzer: Out-of-memory in coder_PGX_fuzzer

[bobfriesenhahn]

It seems that the PGX decoder has an issue which allows a 197 byte PGX file to consume all of the memory on the system and then fail to allocate more memory. The oss-fuzz test case is attached.

oss-fuzz-35265.tar.gz

[mdadams]

@bobfriesenhahn I have made more changes to the API for configuration/initialization and memory allocation. I have also updated the documentation to reflect these changes. In order to reduce the possibility of ABI breakage, I have changed the manner in which configuration settings are specified, based on your suggestions. Now, the configuration parameters are set via the jasconf* family of functions. The jas_conf_t type is no longer part of the API. I have updated the notes.txt in the top-level directory of the repository with a detailed explanation of how the revised API works. The relevant commit is edcede0ed58038fecc5ed98067524a5e84c369bd. My work is still ongoing, but any comments are welcome on this most recent commit. I still need to give more thought to multithreading support. So far, I have mainly been addressing configurability of the library at run time (but trying to do this work in a way that would be friendly to more complete multithreading support later).

[mdadams]

@bobfriesenhahn I added two more customizable settings (i.e., the image format table and the function used to output error/warning/informational messages). I also fixed some compile errors on Windows reported by the Appveyor CI. The relevant commit is aa142c2a83fc47a0415abb17e7a1107b68e47405. I think that this covers the main customization points that I plan to add. (The issue of allowing multiple contexts for multithreading support still is under investigation.)

[mdadams]

@bobfriesenhahn I have just made a new commit (ca07c7041f553d688093cc777b1f1e76c81b3a36) on the mdadams-callbacks branch. This commit has most of the key API functionality that I was hoping to add. It provides the ability to controls things like the memory allocator, the log message output function, memory limit, max. number of samples for the decoder, etc.). It also provides support for multiple library contexts. The library should also now be thread safe. If multiple threads would like to use the library, each thread can create its own context. I have updated the notes.txt file in the top-level directory of the repository. This file has fairly detailed information on the use of the new API. (It could be further refined, but it's a start. If anything is unclear, let me know and I can revise it.) I'd appreciate if you could try out this revised code when you have time. I think that this is slowly converging to something that might be mergeable back to the master branch. For the time being, I have only provided a C11 implementation of the JasPer thread/mutex/TSD primitives. I think that an implementation in terms of POSIX Threads will be easy to add, since it is very similar to C11. Windows might be a little more work to accommodate. At least you can try the new code in its current form on POSIX systems with GCC/Clang and C11 support.

[bobfriesenhahn]

On Fri, 30 Jul 2021, Michael Adams wrote:

more work to accommodate. At least you can try the new code in its current form on POSIX systems with GCC/Clang and C11 support.

I do plan to update GraphicsMagick code to be able to use the new API.

In the mean time, it seems that JasPer does not compile reliably on all systems:

swdev:~/build/jasper-callbacks% make Consolidate compiler generated dependencies of target libjasper [ 1%] Building C object src/libjasper/CMakeFiles/libjasper.dir/base/jas_debug.c.o In file included from /home/bfriesen/src/graphics/jasper-callbacks.git/src/libjasper/base/jas_debug.c:68:0: /home/bfriesen/src/graphics/jasper-callbacks.git/src/libjasper/include/jasper/jas_init.h:144:38: error: unknown type name 'va_list'; did you mean '__va_list'? int (veprintf)(const char format, va_list ap); ^~~

According to documentation on this sytem, the stdarg.h header needs to be included. Is JasPer making sure that this header is included?

STDARG(3EXT) Extended Library Functions STDARG(3EXT)

NAME stdarg - handle variable argument list

SYNOPSIS

include

    va_list pvar;

    void va_start(va_list pvar, void name);

    (type *) va_arg(va_list pvar, type);

    void va_copy(va_list dest, va_list src);

    void va_end(va_list pvar);

-- Bob Friesenhahn @.***, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

[mdadams]

Hi Bob,

On Fri, 30 Jul 2021, Bob Friesenhahn wrote:

In the mean time, it seems that JasPer does not compile reliably on all systems: ... error: unknown type name 'va_list'; did you mean '__va_list'? int (veprintf)(const char format, va_list ap); ^~~

According to documentation on this sytem, the stdarg.h header needs to be included. Is JasPer making sure that this header is included?

I'm sorry about that. The stdarg.h header is now included in all of the source files that use va_list. So, I think that this problem should now be fixed for you. The commit with the fix is 008a8dcf489b1bf4ecb279dec9847ba65ee7d3f1.

--Michael

[mdadams]

@bobfriesenhahn I have now added support (in commit 378f1ac05ee1205ca1b46324bcffed00957c037b) for the POSIX Threads library. So, the CMake-based build will check for C11 threading support and the POSIX Threads library. It will prefer C11 over the POSIX Threads library if both are found, but you can override this in the top-level CMakeLists.txt file. Incidentally, I looked at OpenMP in more detail and I don't think that I can implement all of JasPer's concurrency primitives with it, as it doesn't use an explicit threading model. For example, I cannot explicitly create threads with OpenMP, as far as I can tell.

[bobfriesenhahn]

On Fri, 30 Jul 2021, Michael Adams wrote:

@bobfriesenhahn I have now added support (in commit 378f1ac05ee1205ca1b46324bcffed00957c037b) for the POSIX Threads library. So, the CMake-based build will check for C11 threading support and the POSIX Threads library. It will prefer C11 over the POSIX Threads library if both are found, but you can override this in the top-level CMakeLists.txt file. Incidentally, I looked at OpenMP in more detail and I don't think that I can implement all of JasPer's concurrency primitives with it, as it doesn't use an explicit threading model. For example, I cannot explicitly create threads with OpenMP, as far as I can tell.

I pulled the latest changes the JasPer mdadams-callbacks branch still builds and passes tests under Linux and OmniOS.

It is true that OpenMP does not provide any means to create a thread. An OpenMP program might only have one "worker" thread which is one and the same as the master thread. Each thread does have a distinct id (which is usable as an index since it is not sparse) and there is a function to return the total number of OpenMP threads available. OpenMP is most useful to speed up CPU hungry loops and is best reserved for cases where a large amount of work can be assigned to each worker thread. I recall that OpenJPEG (another JPEG 2000 implementation) now provides support for it.

Under OmniOS (likely using GCC 7.5.0 since I did not tell cmake to use anything else) I still see many warnings similar to:

/home/bfriesen/src/graphics/jasper-callbacks.git/src/libjasper/base/jas_image.c: In function 'jas_image_readcmpt': /home/bfriesen/src/graphics/jasper-callbacks.git/src/libjasper/base/jas_image.c:552:47: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] if (jas_stream_read(stream, buffer, width) != width) /home/bfriesen/src/graphics/jasper-callbacks.git/src/libjasper/bmp/bmp_enc.c: In function 'bmp_encode': /home/bfriesen/src/graphics/jasper-callbacks.git/src/libjasper/bmp/bmp_enc.c:162:54: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] if (jas_image_cmptwidth(image, enc->cmpts[cmptno]) != width || ^~ /home/bfriesen/src/graphics/jasper-callbacks.git/src/libjasper/bmp/bmp_enc.c:163:53: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] jas_image_cmptheight(image, enc->cmpts[cmptno]) != height || ^~ /home/bfriesen/src/graphics/jasper-callbacks.git/src/libjasper/jp2/jp2_cod.c: In function 'jp2_box_put': /home/bfriesen/src/graphics/jasper-callbacks.git/src/libjasper/jp2/jp2_cod.c:553:21: warning: comparison is always false due to limited range of data type [-Wtype-limits] extlen = (box->len >= (((uint_fast64_t)1) << 32)) != 0; /home/bfriesen/src/graphics/jasper-callbacks.git/src/libjasper/pnm/pnm_enc.c: In function 'pnm_encode': /home/bfriesen/src/graphics/jasper-callbacks.git/src/libjasper/pnm/pnm_enc.c:187:54: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] if (jas_image_cmptwidth(image, enc->cmpts[cmptno]) != width || ^~ /home/bfriesen/src/graphics/jasper-callbacks.git/src/libjasper/pnm/pnm_enc.c:188:53: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] jas_image_cmptheight(image, enc->cmpts[cmptno]) != height || /home/bfriesen/src/graphics/jasper-callbacks.git/src/libjasper/ras/ras_enc.c: In function 'ras_encode': /home/bfriesen/src/graphics/jasper-callbacks.git/src/libjasper/ras/ras_enc.c:154:51: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] jas_image_cmptprec(image, enc->cmpts[cmptno]) != depth || ^~ /home/bfriesen/src/graphics/jasper-callbacks.git/src/appl/imgcmp.c: In function 'main': /home/bfriesen/src/graphics/jasper-callbacks.git/src/appl/imgcmp.c:324:47: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] if (jas_image_cmptwidth(reconimage, compno) != width || ^~ /home/bfriesen/src/graphics/jasper-callbacks.git/src/appl/imgcmp.c:325:45: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] jas_image_cmptheight(reconimage, compno) != height) {

Bob -- Bob Friesenhahn @.***, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

[mdadams]

Hi Bob,

On Sat, 31 Jul 2021, Bob Friesenhahn wrote:

I pulled the latest changes the JasPer mdadams-callbacks branch still builds and passes tests under Linux and OmniOS.

Thank you for testing this new version as well as the previous one. It's good to know that the code behaves sanely on platforms other than the ones that I use for development (which are Fedora based).

Under OmniOS (likely using GCC 7.5.0 since I did not tell cmake to use anything else) I still see many warnings similar to:

/home/bfriesen/src/graphics/jasper-callbacks.git/src/libjasper/base/jas_ima ge.c: In function 'jas_image_readcmpt': /home/bfriesen/src/graphics/jasper-callbacks.git/src/libjasper/base/jas_ima ge.c:552:47: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] if (jas_stream_read(stream, buffer, width) != width) /home/bfriesen/src/graphics/jasper-callbacks.git/src/libjasper/bmp/bmp_enc. c:

On my machines, the code compiles with GCC without warnings because I used the default compile options, which does not enable the -Wsign-compare option for GCC. I am not sure why the -Wsign-compare option is enabled for GCC for your compile. By default, it is not enabled on my system. I will add an explicit -Wno-sign-compare option to the top-level CMakeLists.txt file disable it. Having the -Wsign-compare option enabled will create far too many false positives (i.e., warnings for code where there is no bug) to be worthwhile.

--Michael

[mdadams]

@bobfriesenhahn I have made a few more changes to the code on the mdadams-callbacks branch in addition to doing further work on the documentation. Also, I have changed the way that the JasPer Reference Manual is deployed to the JasPer Reference Manual GitHub Pages site so that multiple versions of the manual can now be made available at the same time. Incidentally, the URL for the GitHub Pages site is: https://jasper-software.github.io/jasper-manual/ This will allow me to build draft versions of the manual for the mdadams-callbacks branch and make them available through this page. For example, the link "mdadams-20210802-1024" on this page refers to the most recent version of the manual that I have built for the mdadams-callbacks branch. I have merged the "notes.txt" file from the top-level directory of the repository into a section of the manual on "Initialization and Shutdown". See: https://jasper-software.github.io/jasper-manual/releases/manual-version-mdadams-20210802-1024/html/init__page.html I think that this HTML version of the former notes.txt file is easier to read and navigate.

[mdadams]

@bobfriesenhahn Since the multithreading support for JasPer is becoming more stable now (although some further tweaks and additional documentation are needed), I have merged the changes from the mdadams-callbacks branch into the master branch. I have also added an implementation of threading support for Windows to JasPer. I cannot actually test this Windows support currently, as I don't currently have access to a Windows systems with MSVC. The code for mutexes and TSS support under Windows seems fairly straightforward. (Thread create/join is a little more tricky to someone who has never done code development under Windows, but the JasPer library itself does not use this functionality, only the multithread demo application program does. The code builds on Appveyor CI (with MSVC). So, there is some hope it may work for Windows as is. The manual for the most recent commit (as of the time of this message) can be found at: https://jasper-software.github.io/jasper-manual/releases/manual-version-mdadams-20210818-1024/html/index.html

[bobfriesenhahn]

On Wed, 18 Aug 2021, Michael Adams wrote:

@bobfriesenhahn Since the multithreading support for JasPer is becoming more stable now (although some further tweaks and additional documentation are needed), I have merged the changes from the mdadams-callbacks branch into the master branch. I have also

Thanks for the heads up. I have not yet had time to create an adaptation in GraphicsMagick for the new API.

added an implementation of threading support for Windows to JasPer. I cannot actually test this Windows support currently, as I don't currently have access to a Windows systems with MSVC. The code for

If you have access to a Windows system at all, I find MSYS2 (a Unix-like development/shell environment) to be a very useful development environment targeting native Windows APIs, and it even provides a Jasper package.

Visual Studio is now a free download so just about anyone with a modern Windows system can use it.

Bob -- Bob Friesenhahn @.***, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

[bobfriesenhahn]

Unfortunately, since merging the Jasper branch, the GraphicsMagick oss-fuzz build no longer succeeds due to this failure while building Jasper:

Step #4: [ 63%] Building C object src/libjasper/CMakeFiles/libjasper.dir/jpc/jpc_t1dec.c.o Step #4: In file included from /src/jasper/src/libjasper/base/jas_debug.c:68: Step #4: In file included from /src/jasper/src/libjasper/include/jasper/jas_init.h:73: Step #4: In file included from /src/jasper/src/libjasper/include/jasper/jas_malloc.h:81: Step #4: /src/jasper/src/libjasper/include/jasper/jas_thread.h:88:10: fatal error: 'threads.h' file not found Step #4: #include Step #4: ^~~

The build can be viewed at "https://oss-fuzz-build-logs.storage.googleapis.com/index.html#graphicsmagick" and then select a recent build-date with the red exclamation mark.

Oss-fuzz uses very recent Clang compilers and recent Linux versions.

I see that cmake tested for pthread.h but I don't see any reported test for threads.h. I do see this:

Step #4: -- Looking for pthread.h Step #4: -- Looking for pthread.h - found Step #4: -- Performing Test CMAKE_HAVE_LIBC_PTHREAD Step #4: -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success Step #4: -- Found Threads: TRUE Step #4: Threads library found: true Step #4: Prefer POSIX Threads library: OFF Step #4: POSIX Threads library found: true Step #4: Win32 Threads library found: false

Are you able to fix this build issue? It is useful if Jasper can build (by default) under Linux with the same cmake configuration options that worked before. If Jasper cmake options in order to achieve a successful build need to change in order to build on popular platforms, then there are sure to be issues and complaints.

These are the cmake configuration options used and that worked before:

cmake -G "Unix Makefiles" -H. -Bstaging -DJAS_ENABLE_SHARED=false -DCMAKE_INSTALL_PREFIX=$WORK -DJAS_INCLUDE_BMP_CODEC=false -DJAS_INCLUDE_JPG_CODEC=false -DJAS_INCLUDE_MIF_CODEC=false -DJAS_INCLUDE_PNM_CODEC=false -DJAS_INCLUDE_RAS_CODEC=false -DJAS_ENABLE_OPENGL=false -DJAS_ENABLE_LIBJPEG=false -DJAS_ENABLE_OPENGL=false

Bob -- Bob Friesenhahn @.***, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

[bobfriesenhahn]

Unfortunately, I am not able to build Jasper under OmniOS either:

These warnings come out of CMake:

CMake Warning (dev) at src/appl/CMakeLists.txt:38 (add_executable): Policy CMP0028 is not set: Double colon in target name means ALIAS or IMPORTED target. Run "cmake --help-policy CMP0028" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

Target "multithread" links to target "Threads::Threads" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing? This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) at src/appl/CMakeLists.txt:18 (add_executable): Policy CMP0028 is not set: Double colon in target name means ALIAS or IMPORTED target. Run "cmake --help-policy CMP0028" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

Target "imgcmp" links to target "Threads::Threads" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing? This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) at src/appl/CMakeLists.txt:12 (add_executable): Policy CMP0028 is not set: Double colon in target name means ALIAS or IMPORTED target. Run "cmake --help-policy CMP0028" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

Target "imginfo" links to target "Threads::Threads" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing? This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) at src/appl/CMakeLists.txt:6 (add_executable): Policy CMP0028 is not set: Double colon in target name means ALIAS or IMPORTED target. Run "cmake --help-policy CMP0028" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

Target "jasper" links to target "Threads::Threads" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing? This warning is for project developers. Use -Wno-dev to suppress it.

and then I get a great many other problems due to apparently Jasper not properly including these requisite headers to support open and close:

NAME open, openat - open a file

SYNOPSIS

include <sys/types.h>

    #include <sys/stat.h>
    #include <fcntl.h>

    int open(const char *path, int oflag, /* mode_t mode */);

    int openat(int fildes, const char *path, int oflag,
         /* mode_t mode */);

NAME close - close a file descriptor

SYNOPSIS

include

    int close(int fildes);

[ 16%] Building C object src/libjasper/CMakeFiles/libjasper.dir/base/jas_stream.c.o /home/bfriesen/src/graphics/jasper.git/src/libjasper/base/jas_stream.c: In function 'jas_stream_fopen': /home/bfriesen/src/graphics/jasper.git/src/libjasper/base/jas_stream.c:327:15: error: 'O_RDWR' undeclared (first use in this function) openflags = O_RDWR; ^~ /home/bfriesen/src/graphics/jasper.git/src/libjasper/base/jas_stream.c:327:15: note: each undeclared identifier is reported only once for each function it appears in /home/bfriesen/src/graphics/jasper.git/src/libjasper/base/jas_stream.c:329:15: error: 'O_RDONLY' undeclared (first use in this function); did you mean 'O_RDWR'? openflags = O_RDONLY; ^~~~ O_RDWR /home/bfriesen/src/graphics/jasper.git/src/libjasper/base/jas_stream.c:331:15: error: 'O_WRONLY' undeclared (first use in this function); did you mean 'O_RDONLY'? openflags = O_WRONLY; ^~~~ O_RDONLY /home/bfriesen/src/graphics/jasper.git/src/libjasper/base/jas_stream.c:336:16: error: 'O_APPEND' undeclared (first use in this function) openflags |= O_APPEND; ^~~~ /home/bfriesen/src/graphics/jasper.git/src/libjasper/base/jas_stream.c:342:16: error: 'O_CREAT' undeclared (first use in this function) openflags |= O_CREAT | O_TRUNC; ^~~ /home/bfriesen/src/graphics/jasper.git/src/libjasper/base/jas_stream.c:342:26: error: 'O_TRUNC' undeclared (first use in this function); did you mean 'B_TRUE'? openflags |= O_CREAT | O_TRUNC; ^~~ B_TRUE /home/bfriesen/src/graphics/jasper.git/src/libjasper/base/jas_stream.c:361:17: warning: implicit declaration of function 'open'; did you mean 'popen'? [-Wimplicit-function-declaration] if ((obj->fd = open(filename, openflags, JAS_STREAM_PERMS)) < 0) { ^~~~ popen /home/bfriesen/src/graphics/jasper.git/src/libjasper/base/jas_stream.c: In function 'jas_stream_tmpfile': /home/bfriesen/src/graphics/jasper.git/src/libjasper/base/jas_stream.c:533:29: warning: implicit declaration of function 'unlink'; did you mean 'unix'? [-Wimplicit-function-declaration] if (*obj->pathname != 0 && unlink(obj->pathname) < 0) { ^~ unix /home/bfriesen/src/graphics/jasper.git/src/libjasper/base/jas_stream.c: In function 'file_read': /home/bfriesen/src/graphics/jasper.git/src/libjasper/base/jas_stream.c:1401:9: warning: implicit declaration of function 'read'; did you mean 'fread'? [-Wimplicit-function-declaration] return read(fileobj->fd, buf, cnt); ^~~~ fread /home/bfriesen/src/graphics/jasper.git/src/libjasper/base/jas_stream.c: In function 'file_write': /home/bfriesen/src/graphics/jasper.git/src/libjasper/base/jas_stream.c:1410:9: warning: implicit declaration of function 'write'; did you mean 'fwrite'? [-Wimplicit-function-declaration] return write(fileobj->fd, buf, cnt); ^~~~~ fwrite /home/bfriesen/src/graphics/jasper.git/src/libjasper/base/jas_stream.c: In function 'file_seek': /home/bfriesen/src/graphics/jasper.git/src/libjasper/base/jas_stream.c:1419:9: warning: implicit declaration of function 'lseek'; did you mean 'fseek'? [-Wimplicit-function-declaration] return lseek(fileobj->fd, offset, origin); ^~~~~ fseek /home/bfriesen/src/graphics/jasper.git/src/libjasper/base/jas_stream.c: In function 'file_close': /home/bfriesen/src/graphics/jasper.git/src/libjasper/base/jas_stream.c:1429:9: warning: implicit declaration of function 'close'; did you mean 'pclose'? [-Wimplicit-function-declaration] ret = close(fileobj->fd); ^~~~~ pclose make[2]: *** [src/libjasper/CMakeFiles/libjasper.dir/build.make:202: src/libjasper/CMakeFiles/libjasper.dir/base/jas_stream.c.o] Error 1

-- Bob Friesenhahn @.***, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

[mdadams]

The threads.h header should not be included unless C11 support is present, and C11 support implies that header must exist. I didn't check for __STDC_VERSION__ previously, however, which is likely the problem. This is now fixed in commit cd4266f3b0580ed9b0c44e0cd703e8e7bbf57071 (on master). Does this fix the "threads.h" missing problem for you? Also, it would probably be helpful to state the exact version of the compiler being used and enabling verbose makefiles (CMAKE_VERBOSE_MAKEFILE=1) when reporting bugs like this (so I can see the specific compile commands being used).

[mdadams]

@bobfriesenhahn As for the second set of problems: O_RDWR and things like that are defined in fcntl.h, at least according to this: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/fcntl.h.html#tag_13_11 The JasPer streams code does include fcntl.h if it exists, and the check for if it exists has not been changed in a very long time. Maybe something weird has changed in the build environment on your side?

[bobfriesenhahn]

On Sat, 21 Aug 2021, Michael Adams wrote:

@bobfriesenhahn As for the second set of problems: O_RDWR and things like that are defined in fcntl.h, at least according to this: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/fcntl.h.html#tag_13_11 The JasPer streams code does include fcntl.h if it exists, and the check for if it exists has not been changed in a very long time. Maybe something weird has changed in the build environment on your side?

I am not sure how anything could have changed. What changed is that I am building the version from origin/master rather than the branch.

Something related to cmake seems to be horribly broken. I am using cmake 3.20.2 yet the JasPer CMake script is complaining as if it is older than 2.8.12.

swdev:~/build/jasper-master% cmake --version cmake version 3.20.2

swdev:~/build/jasper-master% cmake -G "Unix Makefiles" -H/home/bfriesen/src/graphics/jasper.git -B. -DCMAKE_VERBOSE_MAKEFILE=1 -DCMAKE_INSTALL_PREFIX=/opt/local CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required): Compatibility with CMake < 2.8.12 will be removed from a future version of CMake.

Update the VERSION argument value or use a ... suffix to tell CMake that the project does not need compatibility with older versions.

-- The C compiler identification is GNU 7.5.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /opt/local/bin/cc - skipped -- Detecting C compile features -- Detecting C compile features - done Software version: 3.0.0 Shared library ABI version: 4 Shared library build version: 4.0.0 JAS_MULTICONFIGURATION_GENERATOR 0 CMake Warning at CMakeLists.txt:154 (message): If this software is being built as a package for a Linux distribution, you should probably set JAS_ENABLE_AUTOMATIC_DEPENDENCIES to false.

-- Performing Test JAS_HAVE_VISIBILITY -- Performing Test JAS_HAVE_VISIBILITY - Success -- Performing Test HAVE_FUNCTION_SECTIONS -- Performing Test HAVE_FUNCTION_SECTIONS - Success -- Performing Test HAVE_DATA_SECTIONS -- Performing Test HAVE_DATA_SECTIONS - Success -- Performing Test HAVE_GC_SECTIONS -- Performing Test HAVE_GC_SECTIONS - Success -- Found OpenGL: /opt/local/lib/libGL.so OpenGL library found: TRUE OpenGL include directory: /opt/local/include OpenGL libraries: /opt/local/lib/libGL.so;/opt/local/lib/libGLU.so -- Could NOT find GLUT (missing: GLUT_glut_LIBRARY GLUT_INCLUDE_DIR) GLUT library found: FALSE CMake Warning at build/cmake/modules/JasOpenGL.cmake:43 (message): Clearing bogus value for GLUT_Xmi_LIBRARY. Call Stack (most recent call first): CMakeLists.txt:232 (include)

CMake Warning at build/cmake/modules/JasOpenGL.cmake:44 (message): Your version of CMake may be buggy. Call Stack (most recent call first): CMakeLists.txt:232 (include)

-- Looking for include file fcntl.h -- Looking for include file fcntl.h - not found -- Looking for include file io.h -- Looking for include file io.h - not found -- Looking for include file unistd.h -- Looking for include file unistd.h - not found -- Looking for include file sys/time.h -- Looking for include file sys/time.h - not found -- Looking for include file sys/types.h -- Looking for include file sys/types.h - not found -- Looking for gettimeofday -- Looking for gettimeofday - not found -- Looking for getrusage -- Looking for getrusage - not found -- Looking for mkostemp -- Looking for mkostemp - not found -- Looking for pthread.h -- Looking for pthread.h - not found -- Could NOT find Threads (missing: Threads_FOUND) Threads library found: false Prefer POSIX Threads library: OFF POSIX Threads library found: false Win32 Threads library found: false -- Found JPEG: /opt/local/lib/libjpeg.so (found version "80") JPEG library found: TRUE JPEG include directory: /opt/local/include JPEG libraries: /opt/local/lib/libjpeg.so -- Performing Test JAS_HAVE_JPEGLIB_H -- Performing Test JAS_HAVE_JPEGLIB_H - Failed CMake Warning at CMakeLists.txt:311 (message): The header file jpeglib.h appears to be missing.

CMake Warning at CMakeLists.txt:312 (message): Disabling LIBJPEG.

GLUT library not found; skipping build of jiv program -- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) -- Could NOT find LATEX (missing: LATEX_COMPILER PDFLATEX) PDFLATEX_COMPILER: PDFLATEX_COMPILER-NOTFOUND LATEX_FOUND FALSE -- Configuring done CMake Warning (dev) at src/appl/CMakeLists.txt:38 (add_executable): Policy CMP0028 is not set: Double colon in target name means ALIAS or IMPORTED target. Run "cmake --help-policy CMP0028" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

Target "multithread" links to target "Threads::Threads" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing? This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) at src/appl/CMakeLists.txt:18 (add_executable): Policy CMP0028 is not set: Double colon in target name means ALIAS or IMPORTED target. Run "cmake --help-policy CMP0028" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

Target "imgcmp" links to target "Threads::Threads" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing? This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) at src/appl/CMakeLists.txt:12 (add_executable): Policy CMP0028 is not set: Double colon in target name means ALIAS or IMPORTED target. Run "cmake --help-policy CMP0028" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

Target "imginfo" links to target "Threads::Threads" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing? This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) at src/appl/CMakeLists.txt:6 (add_executable): Policy CMP0028 is not set: Double colon in target name means ALIAS or IMPORTED target. Run "cmake --help-policy CMP0028" for policy details. Use the cmake_policy command to set the policy and suppress this warning.

Target "jasper" links to target "Threads::Threads" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing? This warning is for project developers. Use -Wno-dev to suppress it.

-- Generating done -- Build files have been written to: /home/bfriesen/build/jasper-master

-- Bob Friesenhahn @.***, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

[bobfriesenhahn]

Is this cmake syntax correct?

cmake -G "Unix Makefiles" -H/home/bfriesen/src/graphics/jasper-callbacks.git -B. -DCMAKE_INSTALL_PREFIX=/usr/local

The reason why I ask is that it is not working under Ubuntu 20.04 either. It behaves the same as it did under OmniOS.

Bob -- Bob Friesenhahn @.***, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

[mdadams]

That looks wrong to me. I would set the current directory to the top-level directory in the source tree and then do something like:

cmake -H. -Btmp_cmake -DCMAKE_INSTALL_PREFIX=/usr/local && cmake --build tmp_cmake --target install

The -G option is not needed if you use cmake to build as well, as I have done above. Does that help?

[bobfriesenhahn]

On Sat, 21 Aug 2021, Michael Adams wrote:

That looks wrong to me. I would set the current directory to the top-level directory in the source tree and then do something like:

cmake -H. -Btmp_cmake -DCMAKE_INSTALL_PREFIX=/usr/local && cmake --build tmp_cmake --target install

The -G option is not needed if you use cmake to build as well, as I have done above. Does that help?

No, it does not work at all. The problems are the same. CMake is still malfunctioning from the very start.

As far as I can tell, I am using the correct files:

swdev:src/graphics/jasper.git% git status On branch master Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

This used to work fine. While I was pulling changes from JasPer git I was usually just doing 'make clean' 'make' and thus not re-running cmake all the time. Now that the callbacks branch has been merged with master they are similar and the callbacks branch does not work any more either. There is no telling when cmake stopped working.

I did not invent the cmake invocation that I have been using since I am using exactly what is documented in the JasPer INSTALL file. It seems odd to recommend different instructions than are in JasPer's own INSTALL file.

The documented cmake invocation is quite convoluted compared to what I am used to (e.g. Autoconf-based configure).

The out-of-tree cmake procedure described for libtiff (http://www.simplesystems.org/libtiff/build.html) which looks like:

% gzip -dc tiff-4.0.5.tar.gz | tar -xf - % mkdir tiff-4.0.5-build % cd ./tiff-4.0.5-build % cmake ../tiff-4.0.5 % make % make test % make install

seems vastly less convoluted. However, the much simpler invocation still encounters exactly the same errors.

The total errors might happen (for example) if the CMake script was to add an option to request a specific strict C standard level, which (by definition) blocks functions outside of the C standard.

Bob -- Bob Friesenhahn @.***, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

[bobfriesenhahn]

As a reminder, oss-fuzz is doing almost daily builds of JasPer and the build may be viewed at

https://oss-fuzz-build-logs.storage.googleapis.com/index.html#graphicsmagick

For some reason the errors in this build are different but it is still ending up with errors like:

Step #4: /src/jasper/src/libjasper/include/jasper/jas_thread.h:88:10: fatal error: 'threads.h' file not found Step #4: #include

Bob -- Bob Friesenhahn @.***, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

[bobfriesenhahn]

Ok, I have now discovered the source of the problem

If I add '-DJAS_ENABLE_HIDDEN=0' to the cmake options, then cmake tests work under both OmniOS and Ubuntu 20.04 and I am able to build the software.

The -fvisibility option is not very portable. Great care is required when using it.

I find it interesting that this was also important for Ubuntu 20.04 since it does use the GNU linker and does support this feature. OmniOS does not use the GNU linker and I am not sure what visibility features it supports.

Bob -- Bob Friesenhahn @.***, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

[mdadams]

On Sun, 22 Aug 2021, Bob Friesenhahn wrote:

If I add '-DJAS_ENABLE_HIDDEN=0' to the cmake options, then cmake tests work under both OmniOS and Ubuntu 20.04 and I am able to build the software.

The -fvisibility option is not very portable. Great care is required when using it.

It is better to avoid using -DJAS_ENABLE_HIDDEN=0, as this is really only a hack to workaround bugs related to shared libraries. And this should not be needed on Ubuntu and many Unix-based systems.

After more experimentation, I was able to reproduce your problem on both Fedora and Ubuntu (via the GitHub Actions CI). The root cause was that some changes broke things for building JasPer as a shared library. I didn't notice this because another change accidentally disabled the CI workflow that builds JasPer as a shared library so that only the static library case was being tested, and normally I don't use the shared library case for my own testing (relying on the CI to test the shared library case). Generally, there were two problems with shared library builds:

1) The support for hidden symbols was not done in a portable way.
While I didn't change this support, I did enable it by default, which
exposed an old bug that you found.
I have now fixed this bug by using CMake to choose the right compiler
options and attributes for hiding symbols.
The should work for both Windows and Unix variants.
At least, I am able to build a JasPer shared library on both Windows and
Ubuntu using Appveyor CI (for Windows) and GitHub Action CI (for
Ubuntu).

2) I forgot to export the function jas_eprintf, which caused an
undefined symbol when linking.

Can you please try again with the commit ac15ab8294e2229dffa9ef5b730d52d2b849dadb (or newer) on the master branch. This definitely builds with both GCC and Clang on Ubuntu because the GitHub Actions CI workflow automatically builds JasPer for these compilers under Ubuntu and this build was successful.

I am not sure about the other OS that you mentioned, but it looks like that problem is also likely due to broken shared library build support, which should now be fixed. I would appreciate if you could let me know if your earlier problems are now fixed by this new commit. Thanks.

[mdadams]

On Sun, 22 Aug 2021, Bob Friesenhahn wrote:

For some reason the errors in this build are different but it is still ending up with errors like:

Step #4: /src/jasper/src/libjasper/include/jasper/jas_thread.h:88:10: fatal error: 'threads.h' file not found Step #4: #include

The only thing that I can think of that would cause this error is if a C compiler claims that it is compliant with C11 (via __STDC_VERSION) and does not correctly define STDC_NO_THREADS as required by the C11 standard. In any case, I have added an explicit check for the existence of the threads.h header instead of relying on STDC_NO_THREADS__. I think that this should fix the problem. Let me know if it does. The fix is in commit 7da6f69dea6356e23a8c1f546b316c79dfb52bb5 (on master).

[mdadams]

When you asked about cmake usage, I was only suggesting not to use the source root as the build directory. The instructions that I gave and the ones in the INSTALL file should both work. I have just updated the instructions in the INSTALL file for Unix systems avoid the need to specify the native build tool (via -G). The number of steps in the installation is now reduced by one. In any case, it appears that all of your problems might be related to the shared library build problem that I have now fixed.

[bobfriesenhahn]

On Sun, 22 Aug 2021, Michael Adams wrote:

I am not sure about the other OS that you mentioned, but it looks like that problem is also likely due to broken shared library build support, which should now be fixed. I would appreciate if you could let me know if your earlier problems are now fixed by this new commit. Thanks.

It does build again and pass tests under OmniOS and Ubuntu 20.04 LTS.

We shall have to wait and see regarding the threads.h header and oss-fuzz since oss-fuzz has control over the builds.

Bob -- Bob Friesenhahn @.***, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

[mdadams]

On Sun, 22 Aug 2021, Bob Friesenhahn wrote:

It does build again and pass tests under OmniOS and Ubuntu 20.04 LTS.

Okay, thank you for checking this. I think that the build method is more robust now. I have also fixed build problems for MacOS by iterating on the errors produced by the MacOS runner in the GitHub Actions CI (since I don't have MacOS running on any of my machines). As of commit 58d8d40d9a0c7fbb96d50bf4ed2479a05243366f, the code also builds correctly for MacOS (and Ubuntu, Fedora, and Windows).

We shall have to wait and see regarding the threads.h header and oss-fuzz since oss-fuzz has control over the builds.

In a way, I sort of wish that the problem would not be fixed because, if it is, this would seem to imply that the compiler is not compliant with the C11 standard. In any case, I guess we shall know soon enough...