[minor] we need a cross-platform way to set dll symbol visibility

[timotheecour]

dll symbol visibility is a bit messy and platform specific; would be nice to have a cross-platform way to do it.

example

proc fun1() {.exportc.} = discard
proc local_fun1() {.exportc.} = discard
proc global_fun1() {.exportc.} = discard
proc fun1_other() = discard
if false: fun1_other() # prevents optimizing it out (maybe there's a more robust way)

OSX

after trying a bit, the key is to use: --passc:-fvisibility=default --passl:-Wl,-exported_symbol,_fun1 (-exported_symbols_list filename could also be used) Nothing else worked. In particular, --version-script is not supported on OSX nor is -export-symbols-regex (note: to help debugging what gets executed, I had to use --listcmd --hint:exec:on)

nim c --app:lib --passl:-Wl,-exported_symbol,_fun1 main.nim
nm libmain.dylib | grep fun1
000000000000eba0 T _fun1
000000000000edb0 t _fun1_other__mxNciiSalhqvqXpk1aCNHw_2
000000000000ed40 t _global_fun1
000000000000ecd0 t _local_fun1

more docs: https://www.unix.com/man-page/osx/1/ld/

linux?

-export-symbols-regex ? --version-script ? We need to make sure it works across compilers

windows?

We need to make sure it works across compilers

note

• the point is to avoid (or have an alternative to) decorating individual symbols with things like __attribute__(("visibility" ("default))) (eg: {.pragma: lib, exportc, codegenDecl: """extern __attribute__((visibility("default"))) $# $#$#""".}), which is not good in many cases (eg maybe different libraries want to export a given symbol differently etc)
• briefly discussed in IRC by leorize

  leorize Araq: btw on *nix Nim isn't hiding unexported symbols when building dlls leorize should we just enforce -fvisibility=hidden then un-hide symbols like for windows?

proposal

• we add a nim specific cmdline flag that abstracts away platform specific differences to export what user wants.
• ideally, it should work even with nim's mangled symbols without needing to know the mangled symbol name; the compilers knows the mapping nimName=>mangledName so should be able to help
• it should work with non-nim code, eg C/C++ code emitted inside nim code
• we should have a simple way to list a number of symbols; ideally via a regex (the compiler can then do the magic to support OSX which doesn't have -export-symbols-regex)

links

• Support a cross-platform symbol export list for library targets · Issue #4298 · mesonbuild/meson : Support a cross-platform symbol export list for library targets · Issue #4298 · mesonbuild/meson
• Control over symbol exports in GCC - antek's tech blog : Control over symbol exports in GCC - antek's tech blog
• https://www.unix.com/man-page/osx/1/ld/
• https://flameeyes.blog/2011/01/20/hide-those-symbols/
• http://www.fortran-2000.com/ArnaudRecipes/sharedlib.html (lines 7 and 8 )

[Araq]

The way it is supposed to work, you use .exportc: "my_name_here", dynlib and then it ends up in the DLL/so. It's our job to ensure that this works indeed.

[timotheecour]

exportc works but that's now what I'm talking about here: oftentimes library writer doesn't want to decorate individual symbols (and hence change source code) just to tell what's exported / unexported in a library; different libraries (with overlapping source code) may choose to export different symbols; that's the whole point of --version-script / -export-symbols-regex / -exported_symbols_list / -exported_symbol.

case in point: right now if you run nim c --app:lib main.nim and then nm -g libmain.dylib you get a whole lot more that what you'd expect that gets exported:

nm -Ug libmain.dylib | wc -l
      83

including things (eg like _rawNewString etc) that could clash with other symbols (or other nim dlls), or expose un-wanted implementation details

instead with the OSX-specific command I showed above, you can export exactly what you specify on cmd line. Here's a minimal working dll that can be dlopen'd from C++:

nim c --app:lib --passl:-Wl,-exported_symbol,_NimMain main.nim
nm -Ug libmain.dylib # only 1 symbol instead of 83
000000000000eea0 T _NimMain

# now call the nim dll from C++, it works
clang++ -std=c++11 -o /tmp/z01 main2.cpp && /tmp/z01

#include <dlfcn.h>
int main (int argc, char *argv[]) {
  auto lib = dlopen("libmain.dylib", RTLD_NOW);
  auto fun = (void(*)()) dlsym(lib, "NimMain");
  fun(); fun();  return 0;
}

[Araq]

Ok but then I fail to see what we have to do, passC is available already for the people who know.

[timotheecour]

the goal is to abstract away platform differences; maybe a std lib proc is all we need:

# user code
import std/dynlib # or other module
setVisibleSymbols(@["foo", "bar"])

# std/dynlib.nim
proc setVisibleSymbolsImpl(s: seq[string)): string =
  doAssert(appType == "lib")
  for si in s:
    when defined(osx)
      result.add "-Wl,-exported_symbol," & si & "  " 
    else: # TODO: linux, windows
macro setVisibleSymbols*(s: static seq[string)) =
  let ret = setVisibleSymbolsImpl(s)
  quote do: {.passC: `ret`.}

[alaviss]

* briefly discussed in IRC by leorize

leorize Araq: btw on *nix Nim isn't hiding unexported symbols when building dlls leorize should we just enforce -fvisibility=hidden then un-hide symbols like for windows?

I don't think what I said was related to this issue. What I meant when I said that was because of this:

00000000000110a0 B NTI__d9aje06sbOkiVogdUrGOkfQ_
000000000000c01c T NimMain
000000000000c06c T NimMainInit
000000000000c010 T NimMainInner
000000000000bfbb T PreMain
000000000000bfaf T PreMainInner
000000000000c174 T _fini
0000000000002000 T _init
00000000000110e0 B allocatorStorage__9abMNjDlLVxVrfWZARGJwzw
00000000000115a0 B allocator__Het9ctRSeaVb4hPekZZNCJQ
0000000000011568 B allocs__T3KvbObbC5lve2bgh5sQDg
000000000000ac43 T chckNil
0000000000014898 B cmdCount
00000000000148a8 B cmdLine
0000000000003775 T cstrToNimstr
0000000000011150 B currException__9bVPeDJlYTi9bQApZpfH8wjg
0000000000011140 B errorMessageWriter__ZXnv0JyrWe3HTd07wpSr7A
0000000000011158 B excHandler__rqLlY5bs9atDw2OXYqJEn5g
0000000000011570 B framePtr__HRfVMH3jYeBJz6Q6X9b6Ptw
000000000000d280 R fsLookupTable__Gn52IZvqY4slyBTOYwGNRQ
00000000000148a0 B gEnv
0000000000011590 B gcFramePtr__ot48iojqko9aFxGhyjjjVaA
0000000000011148 B globalRaiseHook__JbO1ti4ULxrw54m4zNPbpA
0000000000011080 B localAllocator__Fo4FsHewuC4VzDNrMpf3Og
0000000000011580 B localRaiseHook__EIvMhANBvB9cp2Ezvt29cADg
000000000000944c T mulInt
000000000000a1aa T nimAddCharV1
000000000000b639 T nimDestroyAndDispose
0000000000014890 B nimInErrorMode__759bT87luu8XGcbkw13FUjA
00000000000033e5 T nimInt64ToStr
0000000000003910 T nimIntToStr
0000000000002ea7 T nimPrepareStrMutationV2
000000000000b4de T nimRawDispose
000000000000b756 T nimTestErrorFlag
0000000000011588 B nim_program_result
0000000000011578 B onUnhandledException__bFrawQlTKZhLweDD36j9b8g
0000000000011560 B outOfMemHook__kZNaA7u1MfSW5ZeoGvw8xg
00000000000099c7 T prepareAdd
00000000000098a9 T raiseDivByZero
000000000000a069 T raiseOverflow
00000000000034ae T raiseRangeError
000000000000ac75 T rawNewString
000000000000b9b5 T sched_getparam
000000000000bb75 T sched_getscheduler
000000000000bc90 T sched_setparam
000000000000be5e T sched_setscheduler
0000000000002d2b T setLengthStrV2
00000000000110c8 B sharedAllocator__14UnoPGASR4x3NgNJ5Y5lA
0000000000011160 B tempFrames__7nBYIr2UsDREpYylZK4fug
00000000000035c0 T toNimStr

A lot of Nim internal symbols was exported when I was building a dynamic library to replace certain libc functions. This is undesirable when I only exported 4 of them: https://github.com/alaviss/liblinux_sched/blob/hidden/linux_sched.nim

This probably have to do with the fact that N_LIB_PRIVATE was applied inconsistently by the compiler. My suggestion was that to use -fvisibility=hidden by default so that the compiler don't have to do the N_LIB_PRIVATE decoration, then add __attribute__((visibility("default"))) to N_LIB_EXPORT so that it works similar to how it's done for Windows (hide all show only specified). Might not be the best idea though.

[timotheecour]

seems at least somewhat related to this issue; in any case for your specific point, see https://github.com/nim-lang/Nim/pull/13136