johnjcamilleri commented 3 years ago

@krangelov has mentioned at the summer school his ideas on completely reimagining the PGF format as a database. @anka-213 has expressed his interest in contributing, and I (@johnjcamilleri) also think I could be of use here. I'm creating this issue to be the home of that discussion.

krangelov commented 3 years ago

For the current status look at branch 'majestic':

https://github.com/GrammaticalFramework/gf-core/tree/majestic/src/runtime/c

The subfolder 'doc' contains documentation for the decisions that I made so far. My ambition is that the doc folder would develop into a living book documenting the internals. Hopefully, it will help new developers to jump in later.

What works so far:

the C runtime can read the abstract syntax of a .pgf and build an .ngf file from it. The .ngf is the actual database. Some of the abstract syntax functions in the API are implemented as well.
the old Haskell runtime is gone, the folder runtime/haskell contains the new Haskell binding. There are two top-level modules PGF and PGF2. PGF tries to emulate the API of the old Haskell runtime, while PGF2 should be compatible with the API of the C runtime. Everything that already works in the C library has also a binding in the Haskell runtime. There is also a testsuite that I use to test the Haskell binding together with the actual runtime.
most of the code in the Python binding is commented out. It compiles, but the only thing that you can do there is to load a grammar. I only needed that in order to test that it is possible to load the runtime dynamically.
the code for the compiler comes from the branch 'c-runtime'. In that branch the compiler runs on top of the C runtime instead of the Haskell one. In the new branch the compiler doesn't compile yet since not everything in the runtime is implemented yet. Part of the compiler must be updated anyway, since if we want dynamically changing grammars, the grammar building API would have to change anyway.

Current milestones:

Finish the rest of the readonly API for the abstract syntax
Add API that can add/remove abstract functions and categories
Implement proper transactions
Modify the compiler to make it possible to compile grammars with only abstract syntax
Modify the compiler to produce an extended version PMCFG which would not blow up in size so easily
Load the new PMCFG in the runtime
Continue with the different API functions for the concrete syntax. Perhaps linearization should be the first to start with.

If you want to join the effort then we should schedule a meeting and decide how to proceed.

On Mon, 9 Aug 2021 at 11:05, John J. Camilleri @.***> wrote:

@krangelov https://github.com/krangelov has mentioned at the summer school his ideas on completely reimagining the PGF format as a database. @anka-213 https://github.com/anka-213 has expressed his interest in contributing, and I @.*** https://github.com/johnjcamilleri) also think I could be of use here. I'm creating this issue to be the home of that discussion.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrammaticalFramework/gf-core/issues/130, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYFSZHT3E245XBNRLP5A2TT36K4HANCNFSM5BZSYMKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

johnjcamilleri commented 3 years ago

Wow, this seems a lot more advanced than you let on! I will take some time to better understand how it works.

odanoburu commented 3 years ago

Hi @krangelov, pretty cool work! I remember hearing something about using SQLite for this work, which would give you transactions for free and also a platform-independent format (I'm assuming it would take the role of the .ngf files). Have you decided against it? I'm just curious and would love to contribute (I don't think I can, but I'll follow the discussion and if it turns out I can I'll say something!)

krangelov commented 3 years ago

I tried using SQLite but it quickly becomes very cumbersome to work in this way. It means that every time when I must follow a pointer from one structure to another, I must open/close a cursor and do a few more function calls until I get what I want. In addition everything must be serialized/deserialized to a platform independent format.

I am not planning to use it anymore, but here is an interesting spin off project from the experiment:

https://github.com/krangelov/daison

It is a NoSQL database where you can use list comprehensions instead of SQL. Hopefully, something similar will be available for storing auxiliary information in grammars.

On Tue, 10 Aug 2021 at 13:23, bc² @.***> wrote:

Hi @krangelov https://github.com/krangelov, pretty cool work! I remember hearing something about using SQLite for this work, which would give you transactions for free and also a platform-independent format (I'm assuming it would take the role of the .ngf files). Have you decided against it? I'm just curious and would love to contribute (I don't think I can, but I'll follow the discussion and if it turns out I can I'll say something!)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrammaticalFramework/gf-core/issues/130#issuecomment-895948205, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYFSZHVMCVEN7KNYJIHQRDT4ED2JANCNFSM5BZSYMKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

aarneranta commented 3 years ago

Yes, it looks majestic to me, and the documents read well! Although the details about memory addresses are tricky.

I agree that linearization would be the first thing to make available, so that we can see if this is all we need of a linearization format in a large-scale setting.

Aarne

From: John J. Camilleri @.***> Sent: Tuesday, August 10, 2021 12:10:35 PM To: GrammaticalFramework/gf-core Cc: Subscribed Subject: Re: [GrammaticalFramework/gf-core] Next-generation PGF (#130)

Wow, this seems a lot more advanced than you let on! I will take some time to better understand how it works.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/GrammaticalFramework/gf-core/issues/130#issuecomment-895903811, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAWBQXNCW6VC573ASGXGO53T4D3JVANCNFSM5BZSYMKQ.

odanoburu commented 3 years ago

@krangelov I see, thanks for the explanation! and daison looks interesting indeed!

johnjcamilleri commented 3 years ago

I figure the best way I can start contributing to this effort is to be your first user! Which probably means I will have lots of annoying setup questions. To start with, I tried to build the runtime using the instructions from the old C runtime, which failed. I set up a CI workflow here which documents what I tried and also contains the current error I get. I'm happy to contribute to other kinds of documentation (as library docs and how-to guides) as I learn more.

Going forward, we should add whatever testsuite you have (mentioned earlier) to this workflow.

johnjcamilleri commented 3 years ago

To be explicit, I am trying:

autoreconf -i
./configure
make

and getting:

make[1]: Entering directory '/home/runner/work/gf-core/gf-core/src/runtime/c'
  CXX      pgf/db.lo
  CXX      pgf/text.lo
  CXX      pgf/pgf.lo
  CXX      pgf/reader.lo
make[1]: *** No rule to make target 'pgf/expr.cxx', needed by 'pgf/expr.lo'.  Stop.
make[1]: Leaving directory '/home/runner/work/gf-core/gf-core/src/runtime/c'

What am I doing wrong?

krangelov commented 3 years ago

I added expr.cxx to the repo, so it should compile now.

To run the testsuite, you should run:

cabal test

in runtime/haskell.

On Thu, 12 Aug 2021 at 13:07, John J. Camilleri @.***> wrote:

To be explicit, I am trying:

autoreconf -i ./configure make

and getting:

make[1]: Entering directory '/home/runner/work/gf-core/gf-core/src/runtime/c' CXX pgf/db.lo CXX pgf/text.lo CXX pgf/pgf.lo CXX pgf/reader.lo make[1]: *** No rule to make target 'pgf/expr.cxx', needed by 'pgf/expr.lo'. Stop. make[1]: Leaving directory '/home/runner/work/gf-core/gf-core/src/runtime/c'

What am I doing wrong?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrammaticalFramework/gf-core/issues/130#issuecomment-897548450, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYFSZGP7DCQCBVLQJY4LGLT4OTQ5ANCNFSM5BZSYMKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

johnjcamilleri commented 3 years ago

Thanks Krasimir, the CI workflow now builds everything and runs the tests successfully. I will keep poking around to understand things better. Probably the next thing I can work on is adding more tests.

johnjcamilleri commented 3 years ago

most of the code in the Python binding is commented out. It compiles, but the only thing that you can do there is to load a grammar. I only needed that in order to test that it is possible to load the runtime dynamically.

As far as I can see, nothing is commented out in src/runtime/python/pypgf.c?

I added a simple test to load the basic.pgf file from Python:

import pgf
import sys

sys.stdout.write("loading...")
sys.stdout.flush();
gr = pgf.readPGF("../haskell/tests/basic.pgf")
sys.stdout.write("\n")

but I get a segmentation fault. The CI output is here, and I get the same locally.

krangelov commented 3 years ago

Actually commented out means that it is disabled with:

if 0

endif

You can see that on line 42. Comments in C/C++ are not recursive and are not as nice as in Haskell.

Otherwise the Python binding used to work until I broke it while developing the Haskell binding. You can find the comment:

/TO BE FIXED/

on line 3516.

I am working on the coming Python course now, but I hope to return to the runtime soon.

On Tue, 24 Aug 2021 at 09:14, John J. Camilleri @.***> wrote:

most of the code in the Python binding is commented out. It compiles, but the only thing that you can do there is to load a grammar. I only needed that in order to test that it is possible to load the runtime dynamically.

As far as I can see, nothing is commented out in src/runtime/python/pypgf.c?

I added a simple test https://github.com/GrammaticalFramework/gf-core/blob/majestic/src/runtime/python/test.py to load the basic.pgf file from Python:

import pgfimport sys sys.stdout.write("loading...")sys.stdout.flush();gr = pgf.readPGF("../haskell/tests/basic.pgf")sys.stdout.write("\n")

but I get a segmentation fault. The CI output is here https://github.com/GrammaticalFramework/gf-core/runs/3408397541?check_suite_focus=true, and I get the same locally.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrammaticalFramework/gf-core/issues/130#issuecomment-904384489, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYFSZFRZ2FZ54U4FWVSWDDT6NBDXANCNFSM5BZSYMKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

johnjcamilleri commented 3 years ago

Ah, thanks for the clarification!

johnjcamilleri commented 3 years ago

Hi Krasimir, I'm not experienced in C++, but I think this work is so important to the future of GF that it is worth me taking time to learn C++ itself as well as the internal details of how this new runtime works. I might be slow initially, but perhaps you can think of some simpler tasks to give me now, to help me get up to speed?

krangelov commented 3 years ago

I would appreciate it if you could start updating the Python bindings. Before that give me about a week to finish and push some local changes that I have. After that we could have a chat about what needs to be done.

On Thu, 26 Aug 2021 at 08:58, John J. Camilleri @.***> wrote:

Hi Krasimir, I'm not experienced in C++, but I think this work is so important to the future of GF that it is worth me taking time to learn C++ itself as well as the internal details of how this new runtime works. I might be slow initially, but perhaps you can think of some simpler tasks to give me now, to help me get up to speed?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrammaticalFramework/gf-core/issues/130#issuecomment-906146145, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYFSZHWW5IJZ3DTV3NWYEDT6XQYTANCNFSM5BZSYMKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

krangelov commented 3 years ago

The Python binding works again. I also reached the milestone where all essential functions for the abstract syntax work.

There are a lot of low hanging fruits in the Python binding. These functions should be easy to add:

pgf_readNGF -- analogous to readNGF in the Haskell runtime, it can be built by modifying pgf_readPGF pgf_bootNGF -- analogous to bootNGF in the Haskell runtime, it can be built by modifying pgf_readPGF

These are also easy to restore from the old code:

PGF_getAbstractName PGF_getCategories PGF_getFunctions PGF_functionsByCat

The next bigger chunk is:

PGF_functionType -- get the type of a function PGF_getStartCat -- get the type for the start category Expr_repr -- to show an expression as a string pgf_readExpr -- read an expression from a string Type_repr -- to show a type as a string pgf_readType -- read an expression from a string

The last six functions require that the marshalling of abstract expressions and types is implemented for Python as well. I started writing the documentation about how this works.

On Thu, 26 Aug 2021 at 11:47, Krasimir Angelov @.***> wrote:

I would appreciate it if you could start updating the Python bindings. Before that give me about a week to finish and push some local changes that I have. After that we could have a chat about what needs to be done.

On Thu, 26 Aug 2021 at 08:58, John J. Camilleri @.***> wrote:

Hi Krasimir, I'm not experienced in C++, but I think this work is so important to the future of GF that it is worth me taking time to learn C++ itself as well as the internal details of how this new runtime works. I might be slow initially, but perhaps you can think of some simpler tasks to give me now, to help me get up to speed?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrammaticalFramework/gf-core/issues/130#issuecomment-906146145, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYFSZHWW5IJZ3DTV3NWYEDT6XQYTANCNFSM5BZSYMKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

johnjcamilleri commented 3 years ago

Great, I will start working on getting these implemented.

johnjcamilleri commented 3 years ago

Checklist for Python bindings:

Reading from file

[x] pgf_readNGF
[x] pgf_bootNGF

Restore from old code

[x] PGF_getAbstractName
[x] PGF_getCategories
[x] PGF_getFunctions
[x] PGF_functionsByCat

Require marshalling of abstract expressions

[x] PGF_functionType
[x] PGF_getStartCat
[x] Expr_repr
[x] pgf_readExpr
[x] Type_repr
[x] pgf_readType

johnjcamilleri commented 3 years ago

So I have managed to implement the simpler functions for getting categories/functions by restoring old code, but one thing I can't figure out is how the error handing is supposed to work. Namely, the GuExn* err parameter seems to have disappeared from functions such as pgf_iter_categories.

For now the old error-handling code is commented out like so:

static PyObject*
PGF_getCategories(PGFObject *self, void *closure)
{
    PyObject* categories = PyList_New(0);
    if (categories == NULL)
        return NULL;

    // GuPool* tmp_pool = gu_local_pool();
    //
    // Create an exception frame that catches all errors.
    // GuExn* err = gu_new_exn(tmp_pool);

    PyPGFClosure clo = { { pgf_collect_cats }, self, categories };
    pgf_iter_categories(self->pgf, &clo.fn);

    // if (!gu_ok(err)) {
    //     Py_DECREF(categories);
    //     gu_pool_free(tmp_pool);
    //     return NULL;
    // }
    //
    // gu_pool_free(tmp_pool);
    return categories;
}

Any pointers here?

krangelov commented 3 years ago

pgf_iter_categories & pgf_iter_functions doesn't really generate any errors. That is why the parameter has disappeared. Other functions that may fail now take PgfExn as a parameter. See pgf_read_pgf for example.

On Mon, 30 Aug 2021 at 23:29, John J. Camilleri @.***> wrote:

So I have managed to implement the simpler functions for getting categories/functions by restoring old code, but one thing I can't figure out is how the error handing is supposed to work. Namely, the GuExn* err parameter seems to have disappeared from functions such as pgf_iter_categories.

For now the old error-handling code is commented out like so:

static PyObjectPGF_getCategories(PGFObject self, void closure) { PyObject categories = PyList_New(0); if (categories == NULL) return NULL;
// GuPool* tmp_pool = gu_local_pool();
//
// Create an exception frame that catches all errors.
// GuExn* err = gu_new_exn(tmp_pool);

PyPGFClosure clo = { { pgf_collect_cats }, self, categories };
pgf_iter_categories(self->pgf, &clo.fn);

// if (!gu_ok(err)) {
//     Py_DECREF(categories);
//     gu_pool_free(tmp_pool);
//     return NULL;
// }
//
// gu_pool_free(tmp_pool);
return categories;
}

Any pointers here?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrammaticalFramework/gf-core/issues/130#issuecomment-908712830, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYFSZEZRYLD6LWDGRLBGXLT7PZ5JANCNFSM5BZSYMKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

johnjcamilleri commented 3 years ago

Ok, what about the cases below? It seems possible that PyString_FromString and PyList_Append can fail. Do you think it's safe to ignore those too?

static void
pgf_collect_funs(PgfItor* fn, PgfText* key, void* value)
{
    PgfText* name = key;
    PyPGFClosure* clo = (PyPGFClosure*) fn;

    PyObject* py_name = NULL;

    py_name = PyString_FromString(name->text);
    if (py_name == NULL) {
        // gu_raise(err, PgfExn);
        goto end;
    }

    if (PyList_Append((PyObject*) clo->collection, py_name) != 0) {
        // gu_raise(err, PgfExn);
        goto end;
    }

end:
    Py_XDECREF(py_name);
}

krangelov commented 3 years ago

These error should be handled. When the Python API reports an error it also raises a Python exception. We just have to detect that and stop the iteration. I changed the iterator API to receive PgfExn like in the old runtime. I also updated the Python code to compile but I didn't fix the error handling. You just have to set

err.type = PGF_EXN_OTHER_ERROR;

when a problem on the Python side is encountered. This will stop the iterator. At the end you should also release the reference to the partly constructed list.

Another issue is that you should not use PyString_FromString anymore. This function expects that the input string is null terminated, while the new API allows input strings that contain the zero character \0. Instead you can use:

PyString_FromStringAndSize(name->text, name->size);

This would be also more efficient since Python would not need to compute the length itself.

On Tue, 31 Aug 2021 at 08:33, John J. Camilleri @.***> wrote:

Ok, what about the cases below? It seems possible that PyString_FromString and PyList_Append can fail. Do you think it's safe to ignore those too?

static voidpgf_collect_funs(PgfItor fn, PgfText key, void value) { PgfText name = key; PyPGFClosure clo = (PyPGFClosure) fn;
PyObject* py_name = NULL;

py_name = PyString_FromString(name->text);
if (py_name == NULL) {
    // gu_raise(err, PgfExn);
    goto end;
}

if (PyList_Append((PyObject*) clo->collection, py_name) != 0) {
    // gu_raise(err, PgfExn);
    goto end;
}
end: Py_XDECREF(py_name); }

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrammaticalFramework/gf-core/issues/130#issuecomment-908941103, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYFSZATOLJXOLBBBW44QC3T7RZT3ANCNFSM5BZSYMKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

johnjcamilleri commented 3 years ago

Thanks Krasimir. On the subject of strings, does this look right to you?:

static PyObject*
PGF_functionsByCat(PGFObject* self, PyObject *args)
{
    const char* c;
    if (!PyArg_ParseTuple(args, "s", &c))
        return NULL;

    const size_t s = strlen(c);

    PgfText* catname = (PgfText*) alloca(sizeof(PgfText)+s);
    strcpy(catname->text, c);
    catname->size = s;

    ...

I need to read the parameter from args into a PgfText (previously PgfCId). But PyArg_ParseTuple wants to put it into a char*, so I have to do this allocation + copy. I'm guessing strlen also works on null-terminated strings, but maybe here this is the correct solution since the string in the arguments not coming from the runtime?

krangelov commented 3 years ago

I changed functionsByCat to use s# instead. In this way you get both the string and its length. This will work even if the function is called with a string containing the zero character.

Also instead of strcpy I used:

memcpy(catname->text, s, size+1);

Python always adds one zero character at the end and I copy that one as well. The idea is that although PgfText may contain strings containing '\0', most of the time that character is not used. If I append it at the end of the string then when I debug the runtime through gdb it correctly shows me the string as long as it doesn't contain '\0'.

Best Regards, Krasimir

On Tue, 31 Aug 2021 at 10:21, John J. Camilleri @.***> wrote:

Thanks Krasimir. On the subject of strings, does this look right to you?:

static PyObjectPGF_functionsByCat(PGFObject self, PyObject args) { const char c; if (!PyArg_ParseTuple(args, "s", &c)) return NULL;
const size_t s = strlen(c);

PgfText* catname = (PgfText*) alloca(sizeof(PgfText)+s);
strcpy(catname->text, c);
catname->size = s;

...
I need to read the parameter from args into a PgfText (previously PgfCId). But PyArg_ParseTuple wants to put it into a char*, so I have to do this allocation + copy. I'm guessing strlen also works on null-terminated strings, but maybe here this is the correct solution since the string in the arguments not coming from the runtime?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrammaticalFramework/gf-core/issues/130#issuecomment-909011524, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYFSZHJOLXCC6AZDHZX7CTT7SGH7ANCNFSM5BZSYMKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

johnjcamilleri commented 3 years ago

I've starting working on the marshalling in the Python bindings (see 3ecb93775). I think it's starting to make sense, but I'd like to run a few things by you.

What should TypeObject contain (see expr.h)? Currently I have just a PgfText* in there for the type, but it needs more structure for the hypotheses etc. But am I correct in that we do not need a PgfType* in there anymore?
Next, when building a type object to return in the unmarshaller (see dtyp in marshaller.c), I am allocating a TypeObject and then returning it by casting it to a PgfType. Is that right?
Then, when using something which is created by the unmarshaller (see pgf_readType in pypgf.c), I just cast it back to a TypeObject* again. I think this aligns with what you wrote in the documentation, but would appreciate if you you confirm this.

krangelov commented 3 years ago

What should TypeObject contain (see expr.h)? Currently I have just a PgfText in there for the type, but it needs more structure for the hypotheses etc. But am I correct in that we do not need a PgfType in there anymore?

In TypeObject, I would put something like:

typedef struct { PyObject_HEAD PyList hypos; PyString cat; PyList* exprs; } TypeObject;

In this way all fields use native Python types. The list hypos should contain tuples with binding type, variable and the type of the variable. The list exprs should contain objects of type ExprObject.

Next, when building a type object to return in the unmarshaller (see dtyp in marshaller.c), I am allocating a TypeObject and then returning it by casting it to a PgfType. Is that right?

Yes. That is right

Then, when using something which is created by the unmarshaller (see pgf_readType in pypgf.c), I just cast it back to a TypeObject* again. I think this aligns with what you wrote in the documentation, but would appreciate if you you confirm this.

Exactly!

johnjcamilleri commented 3 years ago

In TypeObject, I would put something like:
typedef struct {
    PyObject_HEAD
    PyList* hypos;
    PyString *cat;
    PyList* exprs;
} TypeObject;
In this way all fields use native Python types. The list hypos should contain tuples with binding type, variable and the type of the variable. The list exprs should contain objects of type ExprObject.

Understood! What about the member PyObject* master, is this still relevant? I only half understood what its purpose is/was.

krangelov commented 3 years ago

It is not relevant anymore since everything lives in the Python heap.

On Mon, 6 Sept 2021 at 08:34, John J. Camilleri @.***> wrote:

In TypeObject, I would put something like:

typedef struct { PyObject_HEAD PyList hypos; PyString cat; PyList* exprs; } TypeObject;

In this way all fields use native Python types. The list hypos should contain tuples with binding type, variable and the type of the variable. The list exprs should contain objects of type ExprObject.

Understood! What about the member PyObject* master, is this still relevant? I only half understood what its purpose is/was.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrammaticalFramework/gf-core/issues/130#issuecomment-913381608, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYFSZCRH5QMYI7DST5T6O3UAROGHANCNFSM5BZSYMKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

johnjcamilleri commented 3 years ago

The marshalling and types and expressions in the Python bindings is almost done. One thing in particular which I don't understand is the treatment of single quotes in identifier names.

In the Haskell tests you have:

TestCase (assertEqual "unicode names 2" (Just "ab") (fmap (showExpr []) (readExpr "'ab'")))

Which indicates that the outer quotes around the identifier are dropped when not needed. In Python I have this test as:

def test_readExpr_efun_str_unicode_2():
    assert pgf.showExpr([], pgf.readExpr("'аb'")) == "аb"

However this fails, but passes with ... == "'ab'". So in Python the outer quotes aren't being dropped. I would expect this behaviour to come from the runtime; we are both using pgf_print_expr and pgf_read_expr rather directly, and I don't see that you do anything in particular for quotes in the Haskell bindings. Any idea where this difference could come from?

krangelov commented 3 years ago

There is something really fishy. See what I get:

pgf.showExpr([], pgf.readExpr("'аb'")) "'аb'" pgf.showExpr([],pgf.readExpr("'ab'")) 'ab'

The only difference is the space after the comma. Do you get the same? I will try to investigate.

On Fri, 17 Sept 2021 at 14:07, John J. Camilleri @.***> wrote:

The marshalling and types and expressions in the Python bindings is almost done. One thing in particular which I don't understand is the treatment of single quotes in identifier names.

In the Haskell tests you have:

TestCase (assertEqual "unicode names 2" (Just "ab") (fmap (showExpr []) (readExpr "'ab'")))

Which indicates that the outer quotes around the identifier are dropped when not needed. In Python I have this test as:

def test_readExpr_efun_str_unicode_2():
assert pgf.showExpr([], pgf.readExpr("'аb'")) == "аb"
However this fails, but passes with ... == "'ab'". So in Python the outer quotes aren't being dropped. I would expect this behaviour to come from the runtime; we are both using pgf_print_expr and pgf_read_expr rather directly, and I don't see that you do anything in particular for quotes in the Haskell bindings. Any idea where this difference could come from?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrammaticalFramework/gf-core/issues/130#issuecomment-921745286, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYFSZATMPO5BY32DTH3MT3UCMVPXANCNFSM5BZSYMKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

johnjcamilleri commented 3 years ago

Hmm no I can't reproduce that:

>>> import pgf
>>> pgf.showExpr([], pgf.readExpr("'аb'"))
"'аb'"
>>> pgf.showExpr([],pgf.readExpr("'аb'"))
"'аb'"
>>> str(pgf.readExpr("'аb'"))
"'аb'"

krangelov commented 3 years ago

I think that when I copied your example there was an invisible character with code 0xd0 inside the string. After I manually removed the string and typed it again. It started working.

On Fri, 17 Sept 2021 at 14:48, John J. Camilleri @.***> wrote:

Hmm no I can't reproduce that:

import pgf pgf.showExpr([], pgf.readExpr("'аb'")) "'аb'" pgf.showExpr([],pgf.readExpr("'аb'")) "'аb'" str(pgf.readExpr("'аb'")) "'аb'"

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrammaticalFramework/gf-core/issues/130#issuecomment-921770306, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYFSZEVPCEYKUGGRYFYVOTUCM2LRANCNFSM5BZSYMKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

johnjcamilleri commented 3 years ago

Transaction interface in Python

I'm thinking about how the Python API for transactions should be designed. In Haskell, you create a transaction with createFunction. Although it is implemented in terms of pgf_create_function, the latter only gets executed when a transaction is run with modifyPGF, thanks to functionality and partial evaluation.

We can't do things this way in Python, so my idea is this. Calling createFunction in Python creates some new Transaction object, which is internally modelled as a list of tuples or whatever, but which crucially is not implemented in terms of pgf_create_function. It is only the implementation of modifyPGF which then reads the transaction object and converts it into calls to pgf_create_function and friends. Is this the way you imagined it working?

johnjcamilleri commented 3 years ago

Roadmap

I was talking with @harisont about her possibly contributing to this effort too. For this (and in general) I think it might be good to have a bit of a short-term roadmap of what should be worked on and with what priority. I guess the main goal is to get concrete linearisation working, and there are certainly some subgoals in order to reach that point. Another related question is what other bindings we want to have and whether it's worth working on them now. Specifically, I would like to eventually have bindings from Type/JavaScript and I know there are others who would like this too. But is it premature to start on this now?

Concretely we've been discussing these possibilities:

Use the new Haskell bindings in an actual application (which is currently using the old Haskell runtime). This will require at least linearisation, generation, and morphological analysis.
Split work on the Python bindings between me and her.
She continues on the Python bindings and I look into starting work on JavaScript bindings.

Any thoughts on this?

krangelov commented 3 years ago

The reason why the Haskell API looks as it is is because I need to preserve referential transparency. For example, if I have this:

let ty = functionType gr "f" in do modifyPGF gr (dropFunction "f") return ty

Then ty should still be the type of f although, at the time when it is actually evaluated, the function would have already been dropped from the main branch of the grammar. That is why I maintain persistent and transient branches of the same grammar in the database. Here modifyPGF takes the current master branch and creates a new branch where f is removed. The old branch would still exist as long as there is any client using it. In the particular example it will exist until the evaluation of ty is forced and until the garbage collector is run after that.

In all imperative languages we don't need to care about referential transparency since there are no pure functions there anyway. I imagine the Python API like this:

All readonly API calls always work with the current branch of the grammar.
When we want to make an update we do:

with gr.transaction() as t: createFunction t "f" ty

Here t holds a reference to a new transient branch. All changes are applied to the new branch. When the transaction is finished, the method t.close() should call pgf_commit_transaction which will replace the current master branch with the new transient branch. All subsequent calls to a readonly API would use the updated grammar immediately.

On Tue, 21 Sept 2021 at 10:13, John J. Camilleri @.***> wrote:

Transaction interface in Python

I'm thinking about how the Python API for transactions should be designed. In Haskell, you create a transaction with createFunction. Although it is implemented in terms of pgf_create_function, the latter only gets executed when a transaction is run with modifyPGF, thanks to functionality and partial evaluation.

We can't do things this way in Python, so my idea is this. Calling createFunction in Python creates some new Transaction object, which is internally modelled as a list of tuples or whatever, but which crucially is not implemented in terms of pgf_create_function. It is only the implementation of modifyPGF which then reads the transaction object and converts it into calls to pgf_create_function and friends. Is this the way you imagined it working?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrammaticalFramework/gf-core/issues/130#issuecomment-923745241, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYFSZC7HCJ2WN5ZTFERJ4LUDA5CTANCNFSM5BZSYMKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

krangelov commented 3 years ago

I am currently working on compiling the grammars to a new concrete syntax format. We can already compile and use abstract syntax grammars.

As part of the new compilation, I need to replace the current partial evaluator. The main issues are to make it work with metavariables and to do lazy pattern matching. Since I am already at the evaluator, I also want to make evaluation of variants work properly. Currently they are broken and often we get more variants than we should. I was looking at that problem for the last few days. Once the compilation is done then the next thing will be the linearization itself.

Random/exhaustive generation is a completely independent goal and can be done in parallel. Morphological analysis would be easy as soon as the compilation works. It is actually easier than linearization.

For the bindings we should make all existing bindings to work again. This means that we also need Java (more important) and C# (less important). JavaScript would be nice of course and if you need it then go ahead. I expect that the current API wouldn't change much.

On Tue, 21 Sept 2021 at 10:24, John J. Camilleri @.***> wrote:

Roadmap

I was talking with @harisont https://github.com/harisont about her possibly contributing to this effort too. For this (and in general) I think it might be good to have a bit of a short-term roadmap of what should be worked on and with what priority. I guess the main goal is to get concrete linearisation working, and there are certainly some subgoals in order to reach that point. Another related question is what other bindings we want to have and whether it's worth working on them now. Specifically, I would like to eventually have bindings from Type/JavaScript and I know there are others who would like this too. But is it premature to start on this now?

Concretely we've been discussing these possibilities:

Use the new Haskell bindings in an actual application (which is currently using the old Haskell runtime). This will require at least linearisation, generation, and morphological analysis.

Split work on the Python bindings between me and her.

She continues on the Python bindings and I look into starting work on JavaScript bindings.

Any thoughts on this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrammaticalFramework/gf-core/issues/130#issuecomment-923753561, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYFSZGIB4QCSXDWLT2VYRDUDA6MVANCNFSM5BZSYMKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

harisont commented 3 years ago

I'll try to help with the Java bindings then.

johnjcamilleri commented 3 years ago

The relationship between revision numbers and names is still a little unclear to me. Let me see if I understand this correctly.

pgf_clone_revision is used to create a new "branch" with a new revision number, and an optional name which is implicitly master if omitted.
Changes made with pgf_create_function etc. are made to a specific revision but not active until they are committed with pgf_commit_revision.
If a revision has many pending changes and for some reason the commit fails, then none of them are applied.
The API provides no way of knowing if a revision has uncommitted changes.
Since every API function takes a revision as an argument, you can easily use different versions of a PGF side by side if you keep track of the revision numbers.
pgf_checkout_revision gives you the revision number for a branch name. It is just a lookup and does not change the state of anything (like git checkout does).
pgf_free_revision will remove a revision from memory, such that if you try to use that revision number after freeing it you will get an error.

If two revisions are created from the same original one, and then both committed, both will be usable by their revision number, and the branch name will point to whichever was committed last. See pseudocode:

(db, rev0) = new_ngf(...)
rev1 = clone_revision(db, rev0)
rev2 = clone_revision(db, rev0)
create_function(db, rev0, "f0")
create_function(db, rev1, "f1")
create_function(db, rev2, "f2")
commit_revision(db, rev1)
commit_revision(db, rev2)
assert get_functions(db, rev0) == [] // f0 was never committed
assert get_functions(db, rev1) == ["f1"]
assert get_functions(db, rev2) == ["f2"]
assert checkout_revision("master") == rev2 // rev2 was committed after rev1

krangelov commented 3 years ago

You are mostly right. I should also write a new page in the documentation about revisions. Some comments follow:

On Wed, 22 Sept 2021 at 14:27, John J. Camilleri @.***> wrote:

The relationship between revision numbers and names is still a little unclear to me. Let me see if I understand this correctly.

pgf_clone_revision is used to create a new "branch" with a new revision number, and an optional name which is implicitly master if omitted.

Changes made with pgf_create_function etc. are made to a specific revision but not active until they are committed with pgf_commit_revision.

Changes are always active but only for the revision in which they are done. You should never make changes to a revision which is already committed, only to a freshly cloned one. The reason is that there might be other clients reading the database at the moment and they should not see the changes until they are complete. This happens when the writer calls commit. At that point if the readers call checkout then they will get access to the changes. If they don't call checkout they will keep working with the old revision.

If a revision has many pending changes and for some reason the commit fails, then none of them are applied.

Yes.

The API provides no way of knowing if a revision has uncommitted changes.

This is not hard to add but why do we need it?

Since every API function takes a revision as an argument, you can easily use different versions of a PGF side by side if you keep track of the revision numbers.

pgf_checkout_revision gives you the revision number for a branch name. It is just a lookup and does not change the state of anything (like git checkout does).

pgf_free_revision will remove a revision from memory, such that if you try to use that revision number after freeing it you will get an error.

If two revisions are created from the same original one, and then both committed, both will be usable by their revision number, and the branch name will point to whichever was committed last. See pseudocode:

(db, rev0) = new_ngf(...) rev1 = clone_revision(db, rev0) rev2 = clone_revision(db, rev0) create_function(db, rev0, "f0") create_function(db, rev1, "f1") create_function(db, rev2, "f2") commit_revision(db, rev1) commit_revision(db, rev2) assert get_functions(db, rev0) == [] // f0 was never committed assert get_functions(db, rev1) == ["f1"] assert get_functions(db, rev2) == ["f2"] assert checkout_revision("master") == rev2 // rev2 was committed after rev1

Currently there is no control and you can call create_function with rev0 but you should never do it. The effect is that you will get ["f0"]. Using rev1 and rev2 is fine and they will have different sets of functions.

In the future there should also be a way to merge revisions.

On Wed, 22 Sept 2021 at 14:27, John J. Camilleri @.***> wrote:

The relationship between revision numbers and names is still a little unclear to me. Let me see if I understand this correctly.

pgf_clone_revision is used to create a new "branch" with a new revision number, and an optional name which is implicitly master if omitted.

Changes made with pgf_create_function etc. are made to a specific revision but not active until they are committed with pgf_commit_revision.

If a revision has many pending changes and for some reason the commit fails, then none of them are applied.

The API provides no way of knowing if a revision has uncommitted changes.

Since every API function takes a revision as an argument, you can easily use different versions of a PGF side by side if you keep track of the revision numbers.

pgf_checkout_revision gives you the revision number for a branch name. It is just a lookup and does not change the state of anything (like git checkout does).

pgf_free_revision will remove a revision from memory, such that if you try to use that revision number after freeing it you will get an error.

If two revisions are created from the same original one, and then both committed, both will be usable by their revision number, and the branch name will point to whichever was committed last. See pseudocode:

(db, rev0) = new_ngf(...) rev1 = clone_revision(db, rev0) rev2 = clone_revision(db, rev0) create_function(db, rev0, "f0") create_function(db, rev1, "f1") create_function(db, rev2, "f2") commit_revision(db, rev1) commit_revision(db, rev2) assert get_functions(db, rev0) == [] // f0 was never committed assert get_functions(db, rev1) == ["f1"] assert get_functions(db, rev2) == ["f2"] assert checkout_revision("master") == rev2 // rev2 was committed after rev1

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrammaticalFramework/gf-core/issues/130#issuecomment-924882607, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYFSZHFWL2T4Z5KTHHGLSTUDHDU5ANCNFSM5BZSYMKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

johnjcamilleri commented 3 years ago

Changes are always active but only for the revision in which they are done.

Ok, so the only effect of pgf_commit_revision is to update the link between branch name and revision number. If you only use only revision numbers and not names then you can basically ignore the commit/checkout functions.

This is not hard to add but why do we need it?

If changes are always active then it's not needed, I was assuming that there was such a thing as inactive/pending changes on a revision.

krangelov commented 3 years ago

Ok, so the only effect of pgf_commit_revision is to update the link between branch name and revision number. If you only use only revision numbers and not names then you can basically ignore the commit/checkout functions.

Yes! I assume that Python/Java/C# will always work with the current revision number for a given name. In this way they will see the changes automatically. Haskell will have to call checkoutPGF manually to see the changes.

krangelov commented 3 years ago

I finished the first draft for the transactions documentation:

https://github.com/GrammaticalFramework/gf-core/blob/majestic/src/runtime/c/doc/transactions.md

I hope that it gives an idea about the design and the current status.

On Wed, 22 Sept 2021 at 15:23, Krasimir Angelov @.***> wrote:

Ok, so the only effect of pgf_commit_revision is to update the link between branch name and revision number. If you only use only revision numbers and not names then you can basically ignore the commit/checkout functions.

Yes! I assume that Python/Java/C# will always work with the current revision number for a given name. In this way they will see the changes automatically. Haskell will have to call checkoutPGF manually to see the changes.

johnjcamilleri commented 3 years ago

Thanks for all the documentation! One thing that still bothers me though is that you refer to revisions being transient or persistent, but there is no way to query that information in the API. To me it sounds like a revision is basically persistent if there is some branch name attached to it. But if I have just a revision number, I cannot find out if it is persistent or not. Another seeming omission in the API is that there is no way to get all the available branch names — this actually would allow me to find out if a revision was persistent, by calling pgf_checkout_revision on all branch names and comparing the revision numbers.

krangelov commented 3 years ago

I can easily add an API for querying both revision state and the set of available branches.

Btw. A revision can be transient and still have a name. When you call pgf_clone_revision then this always creates a new transient revision with the same name as the existing one. It will remain transient until you call pgf_commit_revision.

On Fri, 24 Sept 2021 at 08:08, John J. Camilleri @.***> wrote:

Thanks for all the documentation! One thing that still bothers me though is that you refer to revisions being transient or persistent, but there is no way to query that information in the API. To me it sounds like a revision is basically persistent if there is some branch name attached to it. But if I have just a revision number, I cannot find out if it is persistent or not. Another seeming omission in the API is that there is no way to get all the available branch names — this actually would allow me to find out if a revision was persistent, by calling pgf_checkout_revision on all branch names and comparing the revision numbers.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrammaticalFramework/gf-core/issues/130#issuecomment-926371264, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYFSZEA4AW5MDH4QHE7ZCLUDQIVRANCNFSM5BZSYMKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

johnjcamilleri commented 3 years ago

So immediately after this call:

r2 = clone(db, r1)

r1 and r2 are in a sense both linked to master because:

If I call checkout(db, "master") I get r1
If I call commit(db, r2) followed by checkout(db, "master"), then I get r2

What happens if you call clone on an already transient revision?

r2 = clone(db, r1)
r3 = clone(db, r2)

Will the runtime forbid it?

krangelov commented 3 years ago

On Fri, 24 Sept 2021 at 09:49, John J. Camilleri @.***> wrote:

So immediately after this call:

r2 = clone(db, r1)

r1 and r2 are in a sense both linked to master because:

If I call checkout(db, "master") I get r1

If I call commit(db, r2) followed by checkout(db, "master"), then I get r2

Yes.

What happens if you call clone on an already transient revision?

r2 = clone(db, r1) r3 = clone(db, r2)

Will the runtime forbid it?

It will not forbid it. You will just get two transient revisions and you can make changes to both. After that you can commit either of them or both. If you commit both then you should get a merge but currently you get the last revision that you committed.

johnjcamilleri commented 3 years ago

Since the introduction of ipc.cxx (bdd84f10f9a41f5903e5dedf0ff35b3ba26812cf) I am getting this error:

ImportError: /usr/local/lib/libpgf.so.0: undefined symbol: shm_open

krangelov commented 3 years ago

It should be fixed now.

On Fri, 24 Sept 2021 at 13:14, John J. Camilleri @.***> wrote:

Since the introduction of ipc.cxx (bdd84f1 https://github.com/GrammaticalFramework/gf-core/commit/bdd84f10f9a41f5903e5dedf0ff35b3ba26812cf) I am getting this error:

ImportError: /usr/local/lib/libpgf.so.0: undefined symbol: shm_open

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GrammaticalFramework/gf-core/issues/130#issuecomment-926544669, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYFSZGOUPDFQ72K6UGGEZTUDRMQLANCNFSM5BZSYMKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

johnjcamilleri commented 3 years ago

Confirmed, thanks!

johnjcamilleri commented 3 years ago

Some questions about the probability functions:

pgf_function_prob returns positive infinity for a non-existent function; wouldn't 0 or some error make more sense? Maybe the runtime doesn't need to change, but I feel like the bindings should at least handle this better (e.g. raising KeyError in Python)
pgf_category_prob seems to always return 0, regardless of whether the category exists or not. I added two (failing) test cases in Haskell for this.
In the Haskell probability tests you are just checking for equality with pi, but because of floating-point imprecision I am having to use math.isclose() in Python. In createFunction I am reading the probability with PyArg_ParseTuple(args, "f", &prob) into a prob_t and in functionProbability I am converting it back with PyFloat_FromDouble((double)prob), which seems logical to me. I don't understand why it works in Haskell without any loss of precision though.

GrammaticalFramework / gf-core

"Majestic" PGF runtime #130

if 0

endif

Transaction interface in Python

Roadmap