search

nim-lang / RFCs

A repository for your Nim proposals.
136 stars 26 forks source link

`std/platformutils`: platform reflection at CT: test whether platform has some feature (C apis, intrinsics, include files, etc) #414

Open timotheecour opened 3 years ago

timotheecour commented 3 years ago

proposal

Add a low-level (usable in other low-level modules) module std/backendutils with APIs to test whether platform has certain features (eg RDTSC, __builtin_saddll_overflow, C include files/libraries etc). The API is implemented in the compiler by executing shell commands, typically by compiling and/or running C code and checking exit code or stdout/stdin or other means, depending on the use case.

This provides similar functionality to what other build tools provide, eg cmake which runs tests like this: https://github.com/libjpeg-turbo/libjpeg-turbo/blob/main/CMakeLists.txt producing those results: https://gist.github.com/timotheecour/8e11b7617ac332ba59f44ed829f7dc72

backendCompiles

# in std/platformutils
proc backendCompiles*(a: string, wrapInMain = false): bool {.compileTime.} =
 runnableExamples:
   assert backendCompiles("""#include <stdio.h>""")
   assert backendCompiles("""__int128 a;""", wrapInMain = true)
   assert backendCompiles("""NIM_STATIC_ASSERT(sizeof(NIM_BOOL) == 1, "");""", wrapInMain = true)

example 1:

avoid having to guess platforms on which to pass -d:nimEmulateOverflowChecks (and associated logic which spills over to nimbase.h via NIM_EmulateOverflowChecks)

instead, we'd write this code:

# lib/system/integerops.nim:
when backendCompiles("__builtin_ssubll_overflow(0, 0, NULL)", wrapInMain = true):
  proc nimSubInt64(a, b: int64; res: ptr int64): bool # implement using __builtin_ssubll_overflow
else:
  proc nimSubInt64(a, b: int64; res: ptr int64): bool = ... # fallback

example 2:

likewise, would simplify and make the detection logic more robust in bitops, and avoid the need for flags like -d:noIntrinsicsBitOpts

example 3:

check presence of RDTSC for std/cputicks (https://github.com/nim-lang/RFCs/issues/411)

when backendCompiles("__rdtsc()", wrapInMain = true):
  # use __rdtsc timestamp counter
else:
  # use fallback implementation using `clock_gettime` or other timestamp counter

example 4:

for https://github.com/nim-lang/RFCs/issues/399 std/int128s

when backendCompiles("""__int128 a;""", wrapInMain = true):
  # use __int128
else:
  # use emulation

other APIs

other APIs can be added in std/platformutils eg:

implementation

backendCompiles is implemented as a vmops which executes a shell command which compiles the provided code snippet and checks exit status, reusing the logic from extccomp.nim, in particular using same backend flags as would be used for compiling the main projects file.

This in particular is what makes this feature worth having implemented in compiler rather that letting user code call staticExec which would make it more difficult to pass the same backend flags or having to take care about caching, temporary file creation etc (the config logic shouldn't have to be duplicated by user).

caching

juancarlospaco commented 3 years ago

sysutils ?.

Varriount commented 3 years ago

I support this, provided the module is created as an external module first, and then later evaluated for inclusion into the standard library.

timotheecour commented 3 years ago

it needs compiler support to implement the vmops, that's the whole point, so it can be available at CT; I don't see how an external package would work.

Varriount commented 3 years ago

Then perhaps some thought should be given as to how the compiler can support this kind of functionality for external modules.

How would this work for cross-compilation? Even with access to header files for another system, it's not (usually) possible to test at compile-time whether a target system supports certain intrinsics. The best you can do is generate versions of a function that are specialized for certain target systems, and dispatch to them at runtime.

ringabout commented 3 years ago

std/sysrand can benefit from this proposal. This proposal allows urandom fall back to other APIs. Then urandom can work in VM and won't affect users.

Ref https://github.com/nim-lang/Nim/pull/17059 (which is already reverted)

timotheecour commented 3 years ago

Then perhaps some thought should be given as to how the compiler can support this kind of functionality for external modules.

that's precisely what {.vhmook.} is about, user-defined vmops, see https://github.com/timotheecour/Nim/issues/598; however in the case of this RFC, the feature should be a compiler vmops since it's general purpose and would improve existing code in stdlib.

How would this work for cross-compilation?

cross-compilation is always going to be trickier as assumptions on target platform must be specified somehow;