312 object loaded functions

[agentm]

This patch enables loading atom and database context functions from the persistent storage database directory under the "compiled_modules" directory. The expected use-case requires that the database owner copy the object file into that directory and then load it. Subsequent restarts of the server will also load the module from the same location. If a user attempts to load an object file from elsewhere, the load should fail with a security error.

@farzadbekran, would you be willing to test this branch? The object loading code can be tricky and I want to make sure nothing is specific to my environment.

resolves #312

[farzadbekran]

@agentm Sure, I'll let you know as soon as I got some results.

[farzadbekran]

@agentm I'm still getting seg faults. I'm using this file as a test. This is how I compile it:

stack exec -- ghc src/DB/ConstraintFunctions.hs -package project-m36 -package fuzzy

then I put the .o file inside compiled_modules folder and load it in tutd:

loadatomfunctions "DB.ConstraintFunctions" "constraintAtomFunctions" "ConstraintFunctions.o"

and then it dies with this error:

project-m36-server: shopper-db/compiled_modules/ConstraintFunctions.o: unknown symbol `monoidzmsubclasseszm1zi0zi1zmGVl1nDJj2pLCZZya9zzwWBfh_DataziMonoidziTextual_zdfTextualMonoidText0_closure'
./start-db.sh: line 3: 52430 Segmentation fault      (core dumped) stack exec -- project-m36-server -n shopper-db -d shopper-db --fsync

[agentm]

Ok, I see the issue. I'll update the documentation to include instructions on how to build a shared object file and test it with your example.

[agentm]

Hi @farzadbekran, I tested compilation of your module into a shared library and loading using it loadatomfunctions. I'm shoring up the documentation but here's the GHC incantation you'll need:

stack exec -- ghc src/DB/ConstraintFunctions.hs -dynamic -shared -fPIC -o ConstraintFunctions.so -package project-m36 -package fuzzy

[farzadbekran]

Hi @agentm, I'll run a test on my side and let you know how it goes. Thanks.

[farzadbekran]

@agentm after loading the .so file via tutd nothing strange seems to happen. But when I run :showatomfunctions I get this error in the server:

project-m36-server: internal error: SMALL_MUT_ARR_PTRS_FROZEN_DIRTY object (0x42016f33d0) entered!
    (GHC version 8.8.4 for x86_64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug

[agentm]

Very odd- it may be a GHC-version-specific issue. I'll try to narrow it down.

[farzadbekran]

I just checked with ghc-8.8.10, same error.

[agentm]

I tested this with cabal + GHC 8.10.4 and hit the same issue, so it's not related to stack or a specific GHC version. I'll trim down the atom functions to figure out what's going on.

[agentm]

Building and loading examples/DynamicAtomFunctions.so does work, so there is something specific to ConstraintFunctions.so which causes this case to fail.

[agentm]

Oh, it was something dumb. Project:M36 expects [AtomFunction], not a HashSet AtomFunction. After changing that in ConstraintFunctions.hs, I am able to load and use the atom functions.

TutorialD (master/main): loadatomfunctions "ConstraintFunctions" "constraintAtomFunctions" "/home/agentm/Dev/project-m36/ConstraintFunctions.so"
TutorialD (master/main): :showexpr relation{tuple{a fuzzySearch("y","why")}}
┌──────┐
│a::Int│
├──────┤
│1     │
└──────┘

I'll look into validating the type signature of the loading function- that shouldn't crash the server.

[agentm]

Since haskell doesn't keep type information around at runtime, I don't think there is a way to prevent the error.

[farzadbekran]

Just tested it. It works with the changes.

[agentm]

Great! I'll wrap this up and get it merged.

[agentm]

@farzadbekran, with this merged, you should be able to proceed on the master branch with profiling.