Documentation: confused as to how to proceed with larger than few headers libraries.

[mavavilj]

I was interested in trying this for some interesting C/C++ libraries, such as:

https://github.com/libLAS/libLAS

But the documentation is very limited in explaining what are the preferred approaches for libraries larger than a few files.

E.g.

Do I need a library that's "header-only"? Do I need to extract only headers from large libraries? Do they need to be designed as "header-only" or something? Do I need to build a dynamic library? Do I need a library that builds to a dynamic library?

[PMunch]

Yes the documentation is a bit light for now, the project was released in an early state and I wanted to get some feedback to see if there where anything major I should change before writing the documentation. But to answer your questions:

Do I need a library that's "header-only"?

No, as long as you can #include it in C you should be good

Do I need to extract only headers from large libraries? Do they need to be designed as "header-only" or something?

No, see above. If you're able to build C projects using the library then you should be able to build Nim projects using the same libraries with Futhark.

Do I need to build a dynamic library? Do I need a library that builds to a dynamic library?

This completely depends on the library you're building against. The example in the README compiles statically, but it could also be compiled to use a dynamic library. Again, if you can do it in C you should be able to do it in Nim with Futhark.

That being said there is one minor caveat for now. When importing headers with Futhark it won't recursively import other files. This is to avoid a problem where it starts importing all the system libraries and definitions starts to crash with Nim definitions making a huge mess. I plan on adding a way to whitelist folders which it can import from automatically to get the behaviour more similar to C, but in the meantime you need to import each .h file manually.

[PMunch]

Here is an example of how to wrap libLAS, this is a port of the apps/bigtest.c from their repository:

import futhark
import terminal, strutils

importc:
  absPath "/usr/lib/clang/12.0.1/include"
  absPath "/usr/include/liblas/capi/"
  "liblas.h"

proc dumpError(x: string) =
  echo x, "\n\tMessage: ", LASError_GetLastErrorMsg(), "\n\tMethod: ", LASError_GetLastErrorMethod()

var
  header: LASHeaderH = nil
  writer: LASWriterH = nil
  reader: LASReaderH = nil
  pt: LASPointH = nil
  err: LASError
# Limitation about seeking past 4GB output size.  At 20 bytes / record, we
# can successfully write 204 million records, but not 205.
const
  nMillionPoints = 205
  NPOINTS = 1024*1024*nMillionPoints
  OutputName = "Issue147.las".cstring

# Write a LAS file and after the points are in, update the header.
header = LASHeader_Create()
writer = LASWriter_Create(OutputName, header, LAS_MODE_WRITE)

echo "Starting"
for i in 0..<NPOINTS:
  if i mod 1000 == 0:
    stdout.eraseLine()
    stdout.write formatFloat(i / NPOINTS * 100.0, ffDecimal, 2) & "%"

  pt = LASPoint_Create()
  err = LASPoint_SetX(pt, 0)
  if err != LeNone: echo "For point ", i, ", failed to set point value X"
  err = LASPoint_SetY(pt, 0)
  if err != LeNone: echo "For point " , i, ", failed to set point value Y"
  err = LASPoint_SetZ(pt, 0);
  if err != LeNone: echo "For point ", i, ", failed to set point value Z"
  err = LASWriter_WritePoint(writer, pt)
  if err != LeNone: echo "For point ", i, ", failed to WritePoint"
  LASPoint_Destroy(pt)
err = LASHeader_SetPointRecordsCount(header, NPOINTS)
if err != LeNone: dumpError "Failed to LASHeader_SetPointRecordsCount"
err = LASWriter_WriteHeader(writer, header)
if err != LeNone: dumpError "Failed to LASWriter_WriteHeader"
LASWriter_Destroy(writer)
LASHeader_Destroy(header)

# Read the file we just wrote and check the header data.
reader = LASReader_Create(OutputName)
header = LASReader_GetHeader(reader)
var npoints = LASHeader_GetPointRecordsCount(header)
echo "\n\nWrote ", NPOINTS,", Read ", npoints, " (testing ", nMillionPoints," Million (1024 x 1024) Points)"

Compiled with nim c --passL:"-llas_c" bigtest.nim

[mavavilj]

Does this imply that one should always look for a "C API" in any given library?

Even when there's mention of C++ as well.

[PMunch]

Yes, Futhark only supports C at the moment

[mavavilj]

From your example, I do not exactly understand though how it knows that those functions exist. There's

import futhark
import terminal, strutils

importc:
  absPath "/usr/lib/clang/12.0.1/include"
  absPath "/usr/include/liblas/capi/"
  "liblas.h"

But beyond this, I don't understand how nim now understands that functions such as

header = LASHeader_Create()
writer = LASWriter_Create(OutputName, header, LAS_MODE_WRITE)

exist.

Or also, where exactly is futhark run?

[PMunch]

From your example, I do not exactly understand though how it knows that those functions exist.

That is the magic of Futhark, you don't have to care how it knows. But all joking aside, the way it works is that the Futhark macro sees that you want to import "liblas.h", it feeds this along with the paths and such to a helper program called Opir. This program uses libclang to parse the header file and outputs all the information in the header file as a more easily consumable JSON file. Then the Futhark macro converts this JSON into Nim AST which it then outputs into your program. So essentially it knows they exists because it read and understood the C files. If you look in your nimcache folder you will see that it caches the JSON and Nim output if you want to poke around in them.

Not entirely sure what you mean by where Futhark is run? importc is a macro defined in the Futhark library which does all the things I described above.

[mavavilj]

Not entirely sure what you mean by where Futhark is run? importc is a macro defined in the Futhark library which does all the things I described above.

That is because I confuse it with the standard library one:

https://nim-lang.org/docs/manual.html#foreign-function-interface-importc-pragma

I don't know. It would perhaps have been better to name the function differently, like "dofuthark" or "futhark_importc".

[mavavilj]

That is the magic of Futhark, you don't have to care how it knows.

Seems very promising indeed. This was one of the reasons for looking at nim as well, because I don't like the makefile etc. mess in C++ or the IDE mess in Java.

Interested in trying this for other libraries.