build - compiling C files targeting static libraries.

[callendorph]

In an attempted work around of #174 - I tried to build a C wrapper that addresses gsl_complex by pointers instead of passing by value. Here I ran into another problem but with the build. I've setup my build like this:

package gsl/GComplex requires :
  ccfiles:
    "src/GComplex.c"
    "{GSL_LIBDIR}/libgsl.a"
    "{GSL_LIBDIR}/libgslcblas.a"
  ccflags :
    "-I{GSL_INCDIR}"
    "-DGSL_COMPLEX_LEGACY"

Here is the C file:

#include "gsl_complex.h"
#include "gsl_complex_math.h"

void w_gsl_complex_rect(gsl_complex *obj, double x, double y) {
  GSL_SET_COMPLEX(obj,x,y);
}

void w_gsl_complex_polar(gsl_complex *obj, double r, double th) {
  (*obj) = gsl_complex_polar(r, th);
}

double w_gsl_complex_arg(gsl_complex *obj) {
  return gsl_complex_arg(*obj);
}

Here is my stanza file:

defpackage gsl/GComplex :
  import core
  import gsl/Errors

public lostanza deftype GComplex :
  x: double
  y: double

extern w_gsl_complex_rect : (ptr<?>, double, double) -> int
extern w_gsl_complex_polar : (ptr<?>, double, double) -> int
extern w_gsl_complex_arg : (ptr<?>) -> double

public lostanza defn GComplex (x:ref<Double>, y:ref<Double>) -> ref<GComplex> :
  val ret = new GComplex{0.0, 0.0}
  call-c w_gsl_complex_rect(addr!(ret.x), x.value, y.value)
  return ret

public lostanza defn GComplex (r:ref<Double>, th:ref<Double>) -> ref<GComplex> :
  val ret = new GComplex{0.0, 0.0}
  call-c w_gsl_complex_polar(addr!(ret.x), r.value, th.value)
  return ret

public lostanza defn arg (obj:ref<GComplex>) -> ref<Double> :
  val ret = call-c w_gsl_complex_arg(addr!(obj.x))
  return new Double{ret}

When I compile this - I get:

stanza build gsl-tests -verbose
Create temporary file "temp1274769366.s".
Reading pre-compiled package from "pkgs/gsl$ElementaryFuncs$tests.pkg".
Reading pre-compiled package from "pkgs/gsl$GVector$tests.pkg".
Reading pre-compiled package from "pkgs/gsl$GComplex$tests.pkg".
Reading pre-compiled package from "pkgs/gsl$Errors$tests.pkg".
Reading pre-compiled package from "/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/pkgs/stz$test-driver.pkg".
Input packages: gsl/ElementaryFuncs/tests, gsl/GVector/tests, gsl/GComplex/tests, gsl/Errors/tests, stz/test-driver
Reading pre-compiled package from "/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/pkgs/core$stack-trace.pkg".
Reading pre-compiled package from "/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/pkgs/clib.pkg".
Reading pre-compiled package from "/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/pkgs/core.pkg".
Reading pre-compiled package from "/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/pkgs/core$parsed-path.pkg".
Reading pre-compiled package from "/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/pkgs/collections.pkg".
Reading pre-compiled package from "/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/pkgs/stz$test-framework.pkg".
Reading pre-compiled package from "/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/pkgs/arg-parser.pkg".
Reading pre-compiled package from "/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/pkgs/line-wrap.pkg".
Reading pre-compiled package from "pkgs/gsl$ElementaryFuncs.pkg".
Reading pre-compiled package from "pkgs/gsl$GVector.pkg".
Reading pre-compiled package from "pkgs/gsl$Errors.pkg".
Reading pre-compiled package from "pkgs/gsl$GComplex.pkg".
Computed package initialization order: core, core/parsed-path, collections, core/stack-trace, clib, line-wrap, arg-parser, stz/test-framework, gsl/ElementaryFuncs, gsl/ElementaryFuncs/tests, gsl/Errors, gsl/GVector, gsl/GVector/tests, gsl/GComplex, gsl/GComplex/tests, gsl/Errors/tests, stz/test-driver
Call C compiler with arguments:
  "cc"
  "temp1274769366.s"
  "/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/runtime/driver.c"
  "/usr/local/lib/libgsl.a"
  "/usr/local/lib/libgslcblas.a"
  "src/GComplex.c"
  "-std=gnu99"
  "-lm"
  "-lpthread"
  "-ldl"
  "-fPIC"
  "-D"
  "PLATFORM_LINUX"
  "-I/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/include"
  "-I/usr/local/include/gsl"
  "-DGSL_COMPLEX_LEGACY"
  "-o"
  "gsl-tests.exe"
/tmp/ccg1FAUB.o: In function `w_gsl_complex_polar':
GComplex.c:(.text+0x5d): undefined reference to `gsl_complex_polar'
/tmp/ccg1FAUB.o: In function `w_gsl_complex_arg':
GComplex.c:(.text+0x9b): undefined reference to `gsl_complex_arg'
collect2: error: ld returned 1 exit status
Delete temporary file "temp1274769366.s".

Note that it is only the functions in the static library that are failing. The GSL_SET_COMPLEX macro doesn't complain.

Notice that the cc compiler arguments show the following:

 "temp1274769366.s"
  "/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/runtime/driver.c"
  "/usr/local/lib/libgsl.a"
  "/usr/local/lib/libgslcblas.a"
  "src/GComplex.c"

I'm pretty sure this is what is breaking the build. The compiler (gcc) will throw away any symbols in the static library libgsl.a that haven't been used in previous files. I've checked with nm and the symbols for gsl_complex_* are definitely there.

Is there a reason why the files under the ccfiles directive are re-ordered with the *.c files at the end ?

[CuppoJava]

Thanks for the bug report! Let me fix this immediately.

[CuppoJava]

Give 0.17.39 a try. I think that did it.

[callendorph]

Here is the code that I'm trying to build:

https://github.com/callendorph/lbstanza-gsl/pull/2

I downloaded the latest release from the website and tried to build:

$> stanza version

             L.B.Stanza Programming Language

                    Version 0.17.39

Copyright (c) 2016-2022, Patrick Shaobai Li, The Regents of the
University of California. All Rights Reserved.

$> stanza build gsl-tests -verbose
Create temporary file "temp1690217952.s".
Reading pre-compiled package from "pkgs/gsl$ElementaryFuncs$tests.pkg".
Reading pre-compiled package from "pkgs/gsl$GVector$tests.pkg".
Reading from input file "tests/GComplex_tests.stanza".
Expanding macros in input file "tests/GComplex_tests.stanza".
Input file "tests/GComplex_tests.stanza" contains packages gsl/GComplex/tests.
Reading pre-compiled package from "pkgs/gsl$Errors$tests.pkg".
Reading pre-compiled package from "/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/pkgs/stz$test-driver.pkg".
Input packages: gsl/ElementaryFuncs/tests, gsl/GVector/tests, gsl/GComplex/tests, gsl/Errors/tests, stz/test-driver
Reading pre-compiled package from "/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/pkgs/core$stack-trace.pkg".
Reading pre-compiled package from "/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/pkgs/clib.pkg".
Reading pre-compiled package from "/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/pkgs/core.pkg".
Reading pre-compiled package from "/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/pkgs/core$parsed-path.pkg".
Reading pre-compiled package from "/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/pkgs/collections.pkg".
Reading pre-compiled package from "/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/pkgs/stz$test-framework.pkg".
Reading pre-compiled package from "/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/pkgs/arg-parser.pkg".
Reading pre-compiled package from "/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/pkgs/line-wrap.pkg".
Reading pre-compiled package from "pkgs/gsl$ElementaryFuncs.pkg".
Reading pre-compiled package from "pkgs/gsl$GVector.pkg".
Reading pre-compiled package from "pkgs/gsl$Errors.pkg".
Reading pre-compiled package from "pkgs/gsl$GComplex.pkg".
Computed package initialization order: core, core/parsed-path, collections, core/stack-trace, clib, line-wrap, arg-parser, stz/test-framework, gsl/ElementaryFuncs, gsl/ElementaryFuncs/tests, gsl/Errors, gsl/GVector, gsl/GVector/tests, gsl/GComplex, gsl/GComplex/tests, gsl/Errors/tests, stz/test-driver
Lower unoptimized package: gsl/GComplex/tests
Running pass Map Methods
Running pass Create Closures
Running pass Convert Mixes
Running pass Insert Guards
Running pass Elide Checks
Running pass Annotate Live
Running pass Convert Checks to Typeof
Running pass Box Mutables
Running pass Detect Loops
Running pass Simple Inline
Running pass Within Package Inline
Running pass Cleanup Labels
Running pass Simplify Typeof
Running pass Lambda Lift
Running pass Lift Objects
Running pass Resolve Matches
Running pass Simplify Typeof
Running pass Lift Closures
Running pass Lift Type Objects
[Function 1 of 5] Allocating registers for function 1001858 (   //Unnamed function)
[Function 2 of 5] Allocating registers for function 1001859 (   //Unnamed function)
[Function 3 of 5] Allocating registers for function 1001860 (   //Unnamed function)
[Function 4 of 5] Allocating registers for function 5 (   //Unnamed function)
[Function 5 of 5] Allocating registers for function 8 (   //Unnamed function)
External dependency: file "/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/runtime/driver.c" is up-to-date.
External dependency: file "/usr/local/lib/libgsl.a" is up-to-date.
External dependency: file "/usr/local/lib/libgslcblas.a" is up-to-date.
External dependency: file "src/GComplex.c" is up-to-date.
Call C compiler with arguments:
  "cc"
  "temp1690217952.s"
  "/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/runtime/driver.c"
  "/usr/local/lib/libgsl.a"
  "/usr/local/lib/libgslcblas.a"
  "src/GComplex.c"
  "-std=gnu99"
  "-lm"
  "-lpthread"
  "-ldl"
  "-fPIC"
  "-D"
  "PLATFORM_LINUX"
  "-I/mnt/c/Users/callendorph/Documents/AFT/Jitx/linux/lstanza/include"
  "-I/usr/local/include/gsl"
  "-DGSL_COMPLEX_LEGACY"
  "-o"
  "gsl-tests.exe"
/tmp/ccTXSx3h.o: In function `w_gsl_complex_polar':
GComplex.c:(.text+0x5d): undefined reference to `gsl_complex_polar'
/tmp/ccTXSx3h.o: In function `w_gsl_complex_arg':
GComplex.c:(.text+0x9b): undefined reference to `gsl_complex_arg'
collect2: error: ld returned 1 exit status
Delete temporary file "temp1690217952.s".

Files are still in the same order.

I will also note that now when I run the build a second time (after this fails to build like this) I get:

$> stanza build gsl-tests -verbose
Create temporary file "temp1945758733.s".
Build target gsl-tests is already up-to-date.

Which I think is a bug. The build fails and there is no gsl-tests.exe

[CuppoJava]

Pulling your repo now.

[CuppoJava]

Hi Carl,

Okay, I've reproduced the issue, and based upon the shortcomings you exposed, I will change the heuristic that I use to de-duplicate the compiler flags.

But even though the new heuristic is better, in the end, I realize now that it is impossible to solve this problem in its full generality. Here is a description of what was causing the error before.

Your program in the end depends upon these two Stanza packages (among others): gsl/ElementaryFuncs and gsl/GComplex. These packages are pulled in by the Stanza compiler automatically based upon your imports.

And they are declared to have these dependencies:

package gsl/ElementaryFuncs requires :
  ccfiles :
    "{GSL_LIBDIR}/libgsl.a"
    "{GSL_LIBDIR}/libgslcblas.a"

package gsl/GComplex requires :
  ccfiles:
    "src/GComplex.c"
    "{GSL_LIBDIR}/libgsl.a"
    "{GSL_LIBDIR}/libgslcblas.a"    

Based upon that, the system starts collecting a flat list of all the compiler files that are needed:

"{GSL_LIBDIR}/libgsl.a"
"{GSL_LIBDIR}/libgslcblas.a"
"src/GComplex.c"
"{GSL_LIBDIR}/libgsl.a"
"{GSL_LIBDIR}/libgslcblas.a"        

The system then de-duplicates these files by keeping only the first appearance of a file. So the de-duplicated set of files are now:

"{GSL_LIBDIR}/libgsl.a"
"{GSL_LIBDIR}/libgslcblas.a"
"src/GComplex.c"

which results in the incorrect ordering that you were seeing.

This heuristic can be improved for many common cases by changing the de-duplication algorithm. Instead of keeping the first appearance of a file, we can keep the last appearance of a file. This will result in:

"src/GComplex.c"
"{GSL_LIBDIR}/libgsl.a"
"{GSL_LIBDIR}/libgslcblas.a"        

which will work. So I'm going to push a new version of Stanza 0.17.40 that switches to this new heuristic.

Now, here's the counter-example that shows there may still exist cases where this heuristic fails.

Suppose you declared a third dependency like this:

package gsl/dummy-package requires :
  ccfiles:
    "src/GComplex.c"

Now the complete list of non-deduped files look like this:

"{GSL_LIBDIR}/libgsl.a"
"{GSL_LIBDIR}/libgslcblas.a"
"src/GComplex.c"
"{GSL_LIBDIR}/libgsl.a"
"{GSL_LIBDIR}/libgslcblas.a"        
"src/GComplex.c"

And if we apply the keep-last-appearance heuristic to this list, we will get the following:

"{GSL_LIBDIR}/libgsl.a"
"{GSL_LIBDIR}/libgslcblas.a"        
"src/GComplex.c"

And we'll be back to our initial incorrect ordering again.

Thank you for taking the time.

[callendorph]

So I think I understand now the heuristic you are using now - thanks for the detailed response.

You might consider using the --start-group and --end-group options:

https://stackoverflow.com/questions/5651869/what-are-the-start-group-and-end-group-command-line-options

This would alleviate the problem and allow you to keep your existing heuristic.

[CuppoJava]

Oh that's a great tip. Let me look into this.

Thanks!

[CuppoJava]

Stanza 0.17.40 has been pushed.

Thanks for the tip about start-group/end-group. I have opted to also keep the new heuristic because it affects more than just the linker. E.g. it also affects preprocessor directives, etc. The new heuristic does "the sensible thing" more often.

[callendorph]

Confirmed - I believe that is fixed in 0.17.40. Thank you!