R7RS library status

[lassik]

(Continuing from #5)

What's the current status and plans for (define-library ...) and (import ...) support in STklos, and are there places where we could help?

[jpellegrini]

I think those can be built on top of STklos modules with some care. I'm not sure how much of a priority this is for @egallesio

As far as I can see, there are only two points in which STklos is not R7RS compliant -- the library system and hygienic macros.

[jpellegrini]

Also see issue #65

[lassik]

define-library would be a major milestone since it opens up the opportunity to run portable code in STklos (without making any changes to the code). Indeed, it probably won't be too hard to write basic define-library support by adapting define-module.

[egallesio]

Hi @lassik and @jpellegrini,

R7RS define-library is in a local branch for a long time (and I have just seen that the issue #5 is now two years old. TWO YEARS OLD!!:. That is crazy) At this time, I made an implementation that was not correct (I didn't understand, that you can write combined import-sets, and I didn't get back to correct my code).

This week, I took back my old code and corrected the embedded import-spec. If I understand well the R7RS document, we can have imports like :

(import  (prefix (rename (except (only (foo 1)
                                                            foo/1-1 foo/1-2 foo/1-3)
                                                    foo/1-2)
                                       (foo/1-1 new-foo/1-1))
                         prefix-))

They are now accepted. A lot of other things still need some work to have a nice integration with STklos modules. However, I should be able to commit something soon (less than two years, I promise). This first version will have problems with the import/export of syntaxes, but this is a problem that will be tackle when macros will be properly integrated.

[jpellegrini]

I took back my old code and corrected the embedded import-spec ... I should be able to commit something soon

That's great :) STklos is a really cool Scheme implementation, and having full R7RS support it'll be really amazing!

This first version will have problems with the import/export of syntaxes, but this is a problem that will be tackle when macros will be properly integrated

I'm sorry I'm a bit late on that. I'll get back to working on it. Anyway, importing and exporting syntax would likely be easy because they'll be stored as ordinary variables (of a specific type).

[lassik]

Great news! Looking forward to it.

[egallesio]

STklos is a really cool Scheme implementation, and having full R7RS support it'll be really amazing!

Thanks.

I'm sorry I'm a bit late on that. I'll get back to working on it. Anyway, importing and exporting syntax would likely be easy because they'll be stored as ordinary variables (of a specific type).

Don't worry about that.
However, this is not so simple, I think. In fact if we want to still have .ostk files we cannot rely on a table which exists only at runtime. That means that we need to have also a table maintained by the compiler that it uses even when the code is not loaded. Exported syntaxes need also to be extractable from a .ostk without loading it (this why we need to have them in a place which is different from where the bytecode is stored).

[jpellegrini]

In fact if we want to still have .ostk files we cannot rely on a table which exists only at runtime

I think I can adapt what I had done to actually have something that works with current ostk files...

Exported syntaxes need also to be extractable from a .ostk without loading it (this why we need to have them in a place which is different from where the bytecode is stored).

Ok, but... Why is it necessary to be possible to extract syntax from ostk files without loading them? (Just curious)

[egallesio]

Ok, but... Why is it necessary to be possible to extract syntax from ostk files without loading them? (Just curious)

Oops, didn't see this question.

It's a bit tricky: We need to extract the syntax from an ostk file, when compiling a file that is requiring another ostk file. Suppose, that we have the file foo.stk

;; File foo.stk

(export-syntax ++)

(define-macro (++ var)
  `(set! ,var (+ ,var 1)))

(printf "Executing file foo.stk\n")

(provide "foo")

and a file using it called using-foo.stk:

;; using-foo.stk
(require "foo")

(define v 100)
(++ v)
(print v)
(exit)

In a pure interpreted world, we have no problem:

$ stklos -l using-foo.stk
Executing file foo.stk
101

Suppose now that we want to compile using-foo, while foo.stk is not compiled.

stklos-compile -o using-foo.ostk using-foo.stk
using-foo.stk:5: warning: reference to undefined symbol ++
Compilation time 0.5ms

Compilation is OK, but you have a warning on the fact that ++ is unknown. And effectively running the produced ostk will fail.

$ stklos -l using-foo.ostk                      
Executing file foo.stk
**** Error while loading file "using-foo.ostk"
         Where: in try-load
        Reason: variable `++' unbound

  - <<let/call>>e
  - try-load
  - %try-load
  - %load
  - <<let/call>>
EXIT

However, if foo has already been produced, when the compiler sees the (require "foo""), it will find the definition of the ++ macro and will inline its expansion (instead of producing a call to the function ++).

$ stklos-compile -o foo.ostk foo.stk
Compilation time 0.7ms
$ stklos-compile -o using-foo.ostk using-foo.stk
Compilation time 0.602ms
$ stklos -l using-foo.ostk                      
Executing file foo.stk
101
$

This time we have something that is equivalent to the evaluated environment.

In fact, if we have compiled files, we cannot merge the macros in the standard environment, under penalty of producing function calls for macro calls. In other word, when we see a call (f x y z), we need to know if f is a macro (and its expansion code) or a function. Since f can be available from code that we have not executed yet, this information cannot be found in the current dynamic environment.

[egallesio]

Some news about the define-library implementation:

I have an implementation of define-library with support for everything that is defined in R7RS except cond-expand (should be easy) and include-library-declarations (but I have to figure out the difference with include). Export of macros is not functional and probably will not be while we don't have a better implementation of macros.

The import clause can possibly provoke the loading of a file, and .sld suffix can be used.

This implementation uses modules (but modules continue to implicitly import the STklos module, whereas libraries imports nothing by default). For now, imports and exports are not fully integrated. Ideally, I would like to have the same code for importing modules and libraries. It is not so simple, because modules are used during the bootstrap, and the internal representation of import/exports list are not the same. So I have to write some temporary code (that will be deleted when the bootstrap will be done) to manage the dual representation. I have finished, with the export clauses, but export do some controls at run time that could be done at compilation time. Deporting the controls that can be done at compile time (verify that export list are well formed and construction of possible renamings) mostly works. For now, I have problems when requiring compiled modules which uses Bigloo modules (Bigloo modules are used in the implementation of pattern matching).

The (base ...) libraries are implemented and some of the (srfi ...) (in fact, SRFIs are moved in their own directory and need to be modified a bit). Given, the number of supported SRFIs, some time will be needed to have their complete port.

I hope to have something usable soon. When the imports/exports clauses will be usable with libraries as well as modules, I will commit my branch here.

[jpellegrini]

In fact, if we have compiled files, we cannot merge the macros in the standard environment, under penalty of producing function calls for macro calls. In other word, when we see a call (f x y z), we need to know if f is a macro (and its expansion code) or a function. Since f can be available from code that we have not executed yet, this information cannot be found in the current dynamic environment.

@egallesio if global macros are stored as variables in modules, would that work?

• compiler sees (f a b c)
• if f is bound to an object of structure type <syntax>, so the compiler doesn't even try to apply it: I'd add a verification here. when the variable is bound to a <syntax> object, the expander is called and the code re-written.
• if f is not bound to a <syntax> object, then the compiler proceeds as usual.

Would that work? Maybe then the ostk files don't need to change. Can we extract the variables from the ostk files?

[egallesio]

• compiler sees (f a b c)
• if f is bound to an object of structure type <syntax>, so the compiler doesn't even try to apply it: I'd add a verification here. when the variable is bound to a <syntax> object, the expander is called and the code re-written.
• if f is not bound to a <syntax> object, then the compiler proceeds as usual.

Would that work?

Yes! In fact, this is what the compiler does now, except that it doesn't search syntax in the same place as variables. We could probably have such an approach. The problem is how to populate the module with expansions.

In the example before, requiring foo in using-foo.stk should place the expansion in the current-module and that would be easy.

Maybe then the ostk files don't need to change.

Probably not (or not a lot), but I think that we still need to have global syntaxes available in a dedicated place of the .ostk.

Can we extract the variables from the ostk files?

Not easily, and anyway their value is, of course, not available since it can only be known at runtime, whereas the expansion of a syntax must be known at compile time.

In the previous example, you see that the printf in foo.stk is not executed when compiling using-foo.stk, whereas, in a certain extent, the define-macro is static.

[egallesio]

Hi,

I have pushed a branch (called deflib) which implement most of the R7RS define-library. I suffered to fully bootstrap it, since it is used STklos modules and changed a bit their semantics. It is the main reason to let it in a separate branch for now, since it could break legacy code (import order is different, imported variable are now read-only as required by R7RS). Most of the points exposed previously hold.

Of course, it is possible to assume a different behavior for libraries and modules, but IMHO it would be confusing. For now,

• the library (a b c) is implemented by a module called a/b/c
• modules implicitly import the STklos module, which contains all the symbols defined by STklos runtime. Libraries import nothing by default
• Only the SRFIs 1 to 10 are implemented in the R7RS way (you just need to (import (srfi 1)) to load it implementation file and import it). There is still some work to adapt other SRFIs so that they can be uses using the old way with a (require "srfi-n") as well as a (import (srfi n)).
• STklos legacy modules can use the import/export clauses defined in R7RS (renaming, partial imports, ...)
• Documentation has not been updated yet.

That's all folks.

[jpellegrini]

I have pushed a branch (called deflib) which implement most of the R7RS define-library.

Great!!!

stklos> (define-library b (begin (define c 1) ))
;; b
stklos> c
**** Error:
%execute: variable `c' unbound
    (type ",help" for more information)
stklos> (in-module b c)
1

Yes!

Ok, I'll take a better look later. :)

One thing: I see that for now this branch needs -DSTK_DEBUG=1 in order to compile. (Not a problem, just an observation)

[jpellegrini]

@egallesio as I understand, the libraries are modules, and they will be searched for in the same places where modules would, right? So if I have a library named baz in foo/bar/baz.stk, I can add foo/bar/ in load-path and just (import baz)?

What about (import (quux baz)) -- will it try to load quux/baz.stk?

[egallesio]

One thing: I see that for now this branch needs -DSTK_DEBUG=1 in order to compile. (Not a problem, just an observation)

I have corrected this point, but obviously you were faster than me :smiley:

[egallesio]

@egallesio as I understand, the libraries are modules, and they will be searched for in the same places where modules would, right?

Yep!

So if I have a library named baz in foo/bar/baz.stk, I can add foo/bar/ in load-path and just (import baz)?

Yep too, and it works for modules too, since import implementation is shared.

What about (import (quux baz)) -- will it try to load quux/baz.stk?

That's correct. As you can see, there are a scheme and lib subdirectories now in lib to implement (srfi ...) and(scheme ...)` libraries.

[jpellegrini]

Using the branch deflib I was able to run the game of life example in section 5.6.2 of R7RS! (With a small change, since STklos will not work with (import (except (scheme base) set!) (because set! cannot be renamed in STklos), so I exported _set! from (example grid).

But it seems to be working nicely! :)

[egallesio]

Great. :smile:
We'll have to do something to let the redefinition of internals such as set! or let.

[jpellegrini]

Great. smile We'll have to do something to let the redefinition of internals such as set! or let.

The compiler could use uninterned symbols in its internal COND,

(case first
            ((if-1)                 (compile-if             e env tail?))
            ((define-1)             (compile-define         e env tail?))

so the user won't be able to override the primitive themselves, and we could wrap those with macros,

(if . args) -> (if-1 . args)
(define . args) -> (define-1 . args)

Now, the original ones should be available in a special unmutable environment returned by scheme-report-environment, and we could do that...

I was waiting for the new macros to be ready to them work on this.

What do you think?

[jpellegrini]

use uninterned symbols

But then some work would be required to rewrite the compiler case where it chooses based on first...

[jpellegrini]

Hi @egallesio !

I think I have made PRs for most of them, and these are missing:

• 0 (how do we do this one?)
• 41 This one is implemented in two parts, "primitive" and "derived". As far as I could see, nobody implements them separate (but I haven't checked all implementations), so we could as well just join the two files. What do you think?
• 51
• 59, vincinity (I think here's a procedure missing, BTW)
• 89
• 94
• 96 (SLIB prerequisites). I'm not sure how to handle this one
• 100
• 137
• 156
• 171
• 190

Some SRFIs (like 176) define procedures that are always present, even when defining an empty module, so I suppose these need no work. Right? --> Update: I think I was wrong, the new modules are actually empty! Sorry about this noise.

[jpellegrini]

Hi @egallesio ! I see you have transformed several SRFIs into R7RS libraries, so soon thy'll all be finished. I'll start working again in STklos later, can't do that right now because of lack of time...

[egallesio]

Hello @jpellegrini , Yep I'm trying to clean everything before integrating the SRFIs support you wrote. For now, I'm fighting with SRFI-35 which revealed some problems with libraries and re-exportation, the SCHEME module ...

I'll start working again in STklos later, can't do that right now because of lack of time...

No problem at all.

[jpellegrini]

Hi @lassik -- STklos now supports the R7RS define-library form, and all implemented SRFIs are either native or can be loaded with (import ...). Perhaps this issue could then be closed?

[lassik]

Great work! Let's close.