Interface proposal

[jcubic]

First thing that we need to agree before adding Scheme interpreter code and front-end REPL. In need to be generic and allow to change Scheme implementation and maybe also the REPL.

In my interpreter I have two ports stdout nad stdin in global scope that was used to write and read and also interpreter exposed eval and function that could be called from JavaScript. That was my main interface. In recent version I've hidden those ports because user was able to overwrite those names.

stdout, stdin and stderr need be hidden, but I'm not sure how this would look like in Gambit.

The other idea is to make eval and read JavaScript functions that call scheme functions.

There is one issue that I'm worried about (but it's working in gambit-in-browser) that read need to be async function. It can't block the thread like Emscripten default behavior (I'm not sure if it changed, but c read function invoked prompt ugly window). In my interpreter I've solved the issue by making everything is async. read return JavaScript promise and eval may or may not return a promise (exec main API always do that), but promises are hidden from the users. They are resolved automagically.

I was looking into gambit-in-the-browser intf.js file and I've seen some setTimeouts loops, I'm not sure if would be e

Just my ideas, you have any please write them below.

[feeley]

I just tested loading of third party code using a CORS proxy. It works!

> (load "/https://api.allorigins.win/raw?url=https://pastebin.com/raw/cbJHcFa0")
hello
"/https://api.allorigins.win/raw?url=https://pastebin.com/raw/cbJHcFa0"

I created the code here: https://pastebin.com/cbJHcFa0

It is not exactly as nice as an online code editor side-by-side with the REPL, but it is nice to know it works.

If scheme.org hosted a pastebin service then the CORS avoidance hoops would not be required. Of course it is rather risky to execute random code from the web, so perhaps that is a good reason to NOT have a pastebin on scheme.org !

[arthurgleckler]

On Wed, Jan 6, 2021 at 11:32 AM Marc Feeley notifications@github.com wrote:

If scheme.org hosted a pastebin service then the CORS avoidance hoops would not be required. Of course it is rather risky to execute random code from the web, so perhaps that is a good reason to NOT have a pastebin on scheme.org !

We should probably put an "at your own risk" warning somewhere on the page, just to make it clear.

[feeley]

By the way I'm thinking of simplifying the way Gambit handles URLs so the / prefix in front of http://... is not needed. This will require changing the way that Gambit handles paths so that any-valid-scheme:// in front of a path implies that the path is absolute (currently absolute paths must start with a /, or \ on Windows). I expect this to not interfere with the normal path syntax because the double slash in the path means the path is not normalized.

any-valid-scheme = ALPHA *( ALPHA / DIGIT / "+" / "-" / "." )

This change would make working with URLs easier and in particular the desktop version of Gambit could do "the right thing" with (with-input-from-file "http://..." read-line)... essentially an interface to curl in Scheme.

Any thoughts on this?

[lassik]

I agree with Arthur that loading Scheme code directly from URLs on other sites seems a bit risky. The contents of a URL can change at any time. Safer to verify with a cryptographic hash. Or to first load the code into a string, and then eval the string - at least there's an extra step there that the user has to do.

Treating whatever:// as absolute pathnames seems fine. URL syntax is universal nowadays, and some programs/languages unify the concepts or URL and pathname anyway.

[lassik]

However, it's not clear that procedures that can open local files, should be able to open URLs as well. We had a debate about this on one of the SRFI lists a few months back. I think opening a URL is a fundamentally different operation (it falls under the banner of "distributed computing", with more potential for failure and abuse than local-machine stuff).

[arthurgleckler]

On Wed, Jan 6, 2021 at 3:09 PM Lassi Kortela notifications@github.com wrote:

I agree with Arthur that loading Scheme code directly from URLs on other sites seems a bit risky. The contents of a URL can change at any time. Safer to verify with a cryptographic hash. Or to first load the code into a string, and then eval the string - at least there's an extra step there that the user has to do.

I'm okay with allowing it as long as there is a clear warning somewhere of the possible consequences. The value of being able to load code remotely is high, as long as one is careful, so it should be allowed.

[feeley]

@lassik The "local filesystem" is an illusion just like "RAM" is an illusion... especially in these modern days where you never know when something has been virtualized. Even an actual drive on your processor/IDE/USB bus must still be communicated with using messages. Disconnecting a USB drive by error is no different than losing access to your NAS because the network router's power failed.

Of course it is important to have a good error reporting subsystem that will allow the situation to be diagnosed and hopefully recovered from by attempting the operation again.

If you like the simplicity of using curl http://... from the shell, then you should also like (with-input-from-file "http://..." ...) in your favourite programming language.

To summarize, I think it would be bad to treat them differently "by design" because of a perceived reliability difference when in fact the implementors of the language and operating system are allowed to replace these things by devices with a different reliability than what is expected (such as using virtual memory that uses a hard drive to swap pages in and out of the physical RAM, or using a network connected drive on a server at the other end of the planet for a filesystem that pretends to be local).

[lassik]

But a (with-input-from-url "http://..." ...) would be as easy to use as with-input-from-file while making it clear that URLs may be used in addition to filenames. Indeed one can mount URLs into a Unix filesystem, so in principle they are equivalent to filesystem paths. But in practice, they have quite different security, performance and reliability characteristics most of the time. As one of many examples, sanitizing input filenames becomes much harder if URLs are allowed into any procedure that accepts filenames.

[lassik]

NFS is an example of bringing some of the reliability and performance characteristics of networking into the local file system, and it's one of the most disliked and insecure parts of Unix, usually considered a necessary evil. SFTP is well liked because it's clear that an unreliable, slow network connection is being used.

Fallacies of distributed computing is a classic list of the pitfalls of assuming local and remote are conceptually equivalent. This list was written before the current blending with virtualization and USB drives you describe, but the general character of things has changed only somewhat; I think the points generally still hold.

Many of the code portability / maintainability concerns are mitigated by the fact that once you have opened a file, whether local or remote, the same kinds of port objects can be used to read and write the contents. So most of the I/O code can still work the same for local and remote.

[feeley]

I prefer "the unix way" of having an interface which highlights the commonality of the different possible devices (files, ttys, ...) while allowing specialized operations for each type of device.

I dislike specialized interfaces that can't be easily composed. For example:

• (open-textual-input-file ...)
• (open-binary-input-file ...)
• (open-textual-output-file ...)
• (open-binary-output-file ...)
• (open-textual-input-output-file ...)
• (open-binary-input-output-file ...)
• (open-textual-url ...)
• (open-binary-url ...) etc etc

[lassik]

That highlights a further problem: what would open-textual-output-url do?

I agree that it's annoying to duplicate things like textual and binary for URLs. R6RS solved some problems like this this with a file-options object. I think they have the character encoding and things like that in it. Keyword arguments are another way.

[lassik]

The biggest worry I have about a mixed filename and URL opener, is that in a large program, any procedure call that opens a file can now connect to an arbitrary website, without input from the local system administrator (who has to explicitly mount particular servers into the Unix filesystem namespace if he wants to allow access to them). One can eve make network connections by accident if the syntax of any pathname happens to be in URL format instead of local. Scheme code would become difficult to audit.

[feeley]

I could go on with this discussion, but let's keep it for another thread (quickly: the open-output-url could generate a POST and ignore the response, and open-input-output-url could generate a POST and return the result).

[feeley]

Done!

> (current-directory)
"http://gambitscheme.org/try/"
> (path-expand "test.scm")
"http://gambitscheme.org/try/test.scm"
> (with-input-from-file "test.scm" read)
(pp "test.scm says hello!")
> (with-input-from-file "http://gambitscheme.org/try/test.scm" read)
(pp "test.scm says hello!")
> (load "http://gambitscheme.org/try/test.scm")
"test.scm says hello!"
"http://gambitscheme.org/try/test.scm"
> (load "https://api.allorigins.win/raw?url=https://pastebin.com/raw/cbJHcFa0")
hello
"https://api.allorigins.win/raw?url=https://pastebin.com/raw/cbJHcFa0"

[jcubic]

After this long discussion this want I have in my Scheme is that open-input-file use Node.js fs module to open local file, but in browser I do the same with BrowserFS (there are no yet open-output-file so in order to create a file I need to use command line).

BrowserFS is implementation of Node.js filesystem API, with the library for instance you can have git in browser.

And in my interpreter the only way to get load the url is to use load with url Maybe it would be good idea to never allow relative path in load so (load "/foo.scm"). But good things is that if load always fetch the file user can write hos own function that will do the same with local files (browserFS). Another idea is to check if local there is a file with fs.stat if there are no file like this then it will load file from url in same domain.

[jcubic]

I was finally trying try.scheme.org there is one thing missing that is kind of annoying. To start typing after you select the text you need to click exactly into the place after the prompt. It would be nice if you can just click on whole REPL.

The fix can be just detect in mouseup if there are any selection done by user, if selection is empty the code can give focus to the REPL. Also double click should probably be handled as well because someone can double click to select the text. This is for desktop the same should probably be done for mobile. I don't remember that I have in my terminal for this case, but it works fine on mobile.

I was also testing on mobile and there should be viewport meta tag. I can add that so the page is not zoomed out on mobile.

[jcubic]

I've added meta tag to git I've also generated cross-platform favicon (the same can be used on main site) I used realfavicongenerator.net.

[jcubic]

I was wrong about the focus the REPL is below the help screen that is outside you only need to click on the codemirror to make REPL in focus.

[feeley]

I just fixed a library loading bug introduced by my change to support URLs.

Let me explain my point of view on how file paths should be handled (and which is partly implemented currently in Gambit JS). Here are some examples with http://gambitscheme.org/try .

Each Scheme thread has a "current directory". When the main thread is started the current directory refers to the URL where the online REPL is located:

> (current-directory)
"http://gambitscheme.org/try/"

When paths are passed to open-input-file, with-input-from-file, load, etc the paths that are not absolute are "resolved" relative to the current directory. The Gambit procedure that does this operation is path-expand. It takes a path that is either relative or absolute and generates an absolute path:

> (path-expand "test.scm")
"http://gambitscheme.org/try/test.scm"
> (with-input-from-file "test.scm" read-line)
"(pp \"test.scm says hello!\")"
> (load "test.scm")
"test.scm says hello!"
"http://gambitscheme.org/try/test.scm"

A relative path is a path that is not absolute. A path is absolute if it starts with a / or a URI scheme and ://, so both /a/local/path and http://foobar are absolute paths. An initial / on a path can be defined to mean a path on the "local filesystem". With this, it is easy to bind the current directory to a path to the local filesystem to start using local files:

> (current-directory "/a/local/filesystem/path/")
> (path-expand "test.scm")
"/a/local/filesystem/path/test.scm"

The parameterize form can also be used to temporarily use the local filesystem:

> (parameterize ((current-directory "/another/local/filesystem/path/")) (path-expand "test.scm"))
"/another/local/filesystem/path/test.scm"

The Gambit JS runtime system doesn't yet use the local filesystem. However I have already prepared for the integration with BrowserFS... the global JS variable g_os_fs is bound to the nodejs fs module. So in principle just binding that to the BrowserFS module should allow all of this to work. I don't have experience with BrowserFS so maybe @jcubic should try this.

[feeley]

@jcubic By the way git-in-the-browser is really cool! The Gambit-C system (the normal distribution of Gambit) has a module system that can automatically install Scheme libraries from public git repositories using git. So if we can manage to implement a persistent local filesystem for the online REPL we could then use git-in-the-browser to install Scheme libraries on-demand. That would be really neat!

For a small demo with Gambit-C:

% gsi github.com/gambit/hello/demo
People customarily greet each other when they meet.
In English you can say: hello Bob, nice to see you!
In French you can say: bonjour Bob, je suis enchanté!
Demo source code: /Users/feeley/.gambit_userlib/github.com/gambit/hello/@/demo.scm

[jcubic]

Will try but I'm not that confident making change to the current REPL. To enable BrowserFS you need this code:

BrowserFS.configure({
  fs: "IndexedDB"
}, function(e) {
  if (e) {
    throw e;
  }
  window.g_os_fs = BrowserFS.BFSRequire('fs');
});

I was trying to run REPL locally but git don't have all required files, @feeley maybe you should add files that you use to git so anyone can run the app locally. One file that is missing is codemirror and Interpreter.min.js I've downloaded codemirror but I have no idea from where I should get that file. Is it the same as interp-barebones.js. The repo should have everything what's need to run the app.

Also adding Open Source License to the files would be good idea, unless you don't want anyone use the REPL outside of try.scheme.org.

[feeley]

All the files are in the try.scheme.org repository. Just run make in the try subdirectory and then open index.html. You do need to update Gambit first (with a cd gambit;git pull;make;make modules;sudo make install) because some things were fixed this morning.

[feeley]

For the license of the online REPL we can discuss which one, but I'm OK with a liberal one like MIT or BSD.

[lassik]

MIT license is the most traditional one in Scheme/Lisp communities, and also in the JavaScript community.

[lassik]

Is BrowserFS this library: https://github.com/jvilk/BrowserFS So it's a non-standard but convenient front-end that works the same way with many backends. What's the usual backend it uses on a web browser, LocalStorage?

[lassik]

BTW a handful of schemers would be interested in a Git implementation written from the ground up in pure Scheme. If we make one, Gambit could run it as well. But it will take all year, probably longer, to be usable, so good to start with a JS git client for try.scheme.org.

[feeley]

@jcubic I tried your code but got this error in browserfs. Any idea what the problem might be?

What does the "IndexedDB" refer to?

[jcubic]

I'm trying to compile the project but got different error:

I've run git pull in gambit repo built it again and compile interpreter. I'm on branch master of gambit.

[feeley]

@lassik Implementing git in Scheme is a major effort and I wonder if the motivation is merely just "hack value". There are so many other more pressing (and more valuable) things to do! If it every sees the day then I'm definitely interested in using it.

[feeley]

@jcubic make sure you clear your browser cache as your browser might be loading a cached, out of date, Interpreter.min.js .

The variable G_Device_console is defined like this:

if (g_os_web) {

  G_Device_console = function () {
    var dev = this;
    dev.wbuf = new Uint8Array(0);
    dev.rbuf = new Uint8Array(1);
    dev.rlo = 1;
    dev.encoder = new TextEncoder();
    dev.decoder = new TextDecoder();
    dev.echo = true;
    dev.read_condvar_scm = null;
  };
...

So make sure your changes haven't caused g_os_web to be false.

[jcubic]

Are you sure you use Gambit from master branch and not some fixed version that you have locally? I have option to disable cache when dev tools is open and I'm using dev tools all the time including on this.

Anyway I was not able to build the project but I've copied the code from try.scheme.org. Now I've tried to run this code:

(parameterize ((current-directory "/"))
  (let ((p (open-output-file "/test")))
    (write '(hello world) p)
    (close-output-port p)
    (let ((p (open-input-port "/test")))
      (display (read p))
      (newline))))

but it enter debugger and I don't know how to exit, ,q throw exception and exit scheme.

[feeley]

@jcubic concerning your first error, did you do make modules before the make install? You can also do make _gambit.js;sudo make install which is faster when only lib/_univlib.scm changes.

For the debugger... try ,help for the list of commands. You want ,t to return to the toplevel REPL, or you can enter ctrl-D (end-of-file "pops" one level of REPL).

[feeley]

OK I've managed to get a little further by using BrowserFS with "LocalStorage". I have pushed my changes on gambit master.

Now I get the following error. So it seems that BrowserFS is not 100% compatible with the nodejs fs module (some constants are missing):

[jcubic]

OK, I've build gamit and intepreter it's some kind of progress now I have error:

Uncaught ReferenceError: g_os_translate_flags is not defined

[feeley]

Also just pushed my changes to the try.scheme.org repo...

[feeley]

Can you explain in what situation you get that error? I just installed the very latest changes to http://gambitscheme.org/try and I don't get that error in the JS console.

[feeley]

The error I get when doing (open-input-file "/foobar") is the error I mention above.

TypeError: undefined is not an object (evaluating 'g_os_fs.constants.O_RDONLY')

[jcubic]

I use my snippet with parametrize but I've used BrowserFS 1.4 didn't know 2.0 exists jsdelivr I think I've just used old url that I've saved in by history.

[jcubic]

I've added commit with initalization of browserFS when you try now to use the code you will get this error.

[jcubic]

It seems that g_os_translate_flags need to be some kind of function but it's not defined.

[feeley]

You can't put the initialization of BrowserFS in the file Interpreter.scm because the JS code there is executed after the Gambit runtime system has been initialized. If you use my latest commit to Gambit you will see that I have initialized BrowserFS very early in the initialization of the runtime system. Here is the code in Gambit's lib/_univlib.scm:

(macro-case-target

 ((js)
  (##inline-host-declaration "

// Autodetect when running in a web browser (as opposed to nodejs).
g_os_web = (function () { return this === this.window; })();

// Autodetect availability of nodejs features.
if (typeof g_os_nodejs === 'undefined') {
  g_os_nodejs = !g_os_web;
}

if (g_os_nodejs) {
  os = require('os');
  vm = require('vm');
  process = require('process');
  child_process = require('child_process')
  if (typeof g_os_fs === 'undefined') {
    g_os_fs = require('fs');
  }
}

if (typeof g_os_fs === 'undefined') {

  g_os_fs = null;

  // Autodetect BrowserFS and use it if available for local filesystem.
  if (typeof BrowserFS !== 'undefined') {
    BrowserFS.configure({
      fs: 'LocalStorage'
    }, function (e) {
      if (e) {
        throw e;
      }
      g_os_fs = BrowserFS.BFSRequire('fs');
    });
  }
}
...

[jcubic]

Found in Gambit https://github.com/gambit/gambit/search?q=g_os_translate_flags I'm trying to define the g_os_fs = true because BrowserFS is async.

[feeley]

All of the nodejs file operations are done asynchronously, so it should work "as is" with BrowserFS. As I mention above, the problem is that fs.constants is not defined by BrowserFS.

[jcubic]

I mean g_os_fs is defined asynchronously, so if any code execute before it don't see that there is g_os_fs variable defined. when I've copied missing function to interpreter.scm I now get error about missing constants. There is issue created in 2018 by the author of BrowserFS https://github.com/jvilk/BrowserFS/issues/216

[feeley]

How can a dependency between JS <script>s be expressed in the index.html file? There must be a way to do this! It must be a fairly frequently occurring problem...

Copying the lib/_univlib.scm code into Interpreter.scm is not a long term solution...

For the constants, I guess we could create our own when using BrowserFS. I'll try it out...

[lassik]

How can a dependency between JS <script>s be expressed in the index.html file? There must be a way to do this! It must be a fairly frequently occurring problem...

https://en.wikipedia.org/wiki/Asynchronous_module_definition

New versions of the JS/EcmaScript language have some kind of native include/import statement.

[lassik]

@jcubic must know more about this, but there are module bundlers like https://www.rollupjs.org/ that handle the dependency resolution at build time. Kind of like a Unix/C linker.

[feeley]

I have pushed to gambit master changes to lib/_univlib.scm that define the missing constants. Also I have removed the BrowserFS initialization in Interpreter.scm. Now things don't "crash", but I do get an exception:

This is with BrowserFS with "LocalStorage".

[feeley]

I copied the constants from node's fs module. Maybe these are the wrong constants for BrowserFS...

[feeley]

That doesn't work because BrowserFS does not support fs.open with an integer flags parameter... it only accept strings (like "r", "w+", etc). So I'll have to redefine g_os_translate_flags to generate the appropriate flags strings...