jscl-project / jscl

A Lisp-to-JavaScript compiler bootstrapped from Common Lisp
https://jscl-project.github.io
GNU General Public License v3.0
876 stars 108 forks source link

Discuss interface of the JavaScript FFI #298

Open foretspaisibles opened 6 years ago

foretspaisibles commented 6 years ago

As stated here and there across code, issues or some PRs, we still need to work on a design for the JavaScript. I have started to experiment with this, see branch https://github.com/michipili/jscl/tree/explore/react. This branch is prefixed byexplore/, this is explorative programming and the organisation is really messy. :) The goal is not yet to prepare a PR but to experiment with ideas and gather feedback. My approach was motivated by the following requirements:

Foreign objects

  1. I would like to deal with foreign JavaScript objets pretty much as if they had been created by a CL defstruct, enhanced with functions to perform operations.

  2. I would like to reduce the boilerplate typing to the bare minimum and therefore use a declarative syntax, similar to what we know form defstruct or defclass.

For 2., here is an example of exposing the foreign JavaScript Arrays, just giving enough functions to handle them as a stack:

(define-foreign-js-class (js-array (:validate-class t) (:constructor |Array|))
  ((length
    :documentation "The length property of an object which is an instance of type Array sets or returns the number of elements in that array."
    :type number))
  ((push
    :documentation "The PUSH method adds one or more elements to the end of an array and returns the new length of the array.")
   (pop
    :documentation "The POP method removes the last element from an array and returns that element. This method changes the length of the array."))))

and here is a more extensive example, describing the Synthetic React-Dom events.

(defparameter *current-event* nil
  "The React event currently processed.")

(define-foreign-js-class
  (event (:implicit-instance *current-event*))
  ((bubbles
    :documentation "A Boolean indicating whether the event bubbles up through the DOM or not."
    :type boolean)
   (cancelable
    :documentation "A Boolean indicating whether the event is cancelable."
    :type boolean)
   (current-target
    :documentation "A reference to the currently registered target for the event. This is the object to which the event is currently slated to be sent; it's possible this has been changed along the way through retargeting."
    :type dom-event-target)
   (default-prevented
    :documentation "Indicates whether or not event.preventDefault() has been called on the event."
    :type boolean)
   (event-phase
    :documentation "Indicates which phase of the event flow is being processed."
    :type number)
   (is-trusted :type boolean)
   (native-event :type dom-event)
   (target
    :documentation "A reference to the target to which the event was originally dispatched."
    :type dom-event-target)
   (timestamp
    :documentation "The time at which the event was created, in milliseconds."
    :type number)
   (type
    :documentation "The name of the event (case-insensitive)."
    :type string))
  ((prevent-default
    :documentation "Cancels the event (if it is cancelable).")
   is-default-prevented
   (stop-propagation
    :documentation "Prevents further propagation of the current event in the capturing and bubbling phases.")
   is-propagation-stopped))

Note the use of an optional :implicit-instance which could also be used nicely to bind modules, giving the unique ability to rebind them locally, e.g. in a test-suite.

Satisfying these requirements means that the details of the foreign JavaScript layer are all contained in the argument to define-foreign-js-class. This avoid these details leaking away in other parts for the code, and in turns avoids tight coupling to JavaScript and JSCL of the programs using these definitions.

See https://github.com/michipili/jscl/blob/explore/react/react/ffi2.lisp#L52 to experiment with these definitions.

Defining JS Classes in Common Lisp

I used the Babel compiler (ES6 to portable JavaScript) to examine the implementation of inheritance in ES6 and tries to mimic this in JSCL. I have tried to define ahttps://github.com/michipili/jscl/blob/explore/react/src/compiler/compiler.lisp#L1437 builtin

(define-builtin create-class (class super props)
    `(var (,class
           (call
            (function (|superParameter|)
               (var (|props| (object ,@props)))
               (named-function ,class ()
                 (call-internal |classCallCheck| |this| ,class)
                 (var (|_this|
                        (method-call
                          (call-internal |possibleConstructorReturn|
                                         |this|
                                         (or (get ,class "__proto__")
                                             (method-call |Object| "getPrototypeOf" ,class)))
                          "call"
                          |this|)))
                 (var |i|)
                 (for ((= |i| 0) (< |i| (get |props| "length")) (post++ |i|))
                   (var (|descriptor| (property |props| |i|)))
                   (if (in "value" |descriptor|)
                       (if (=== (typeof (get |descriptor| "value")) "function")
                           (= (property |_this| (get |descriptor| "key"))
                              (method-call (get |descriptor| "value") "bind" |this|))
                           (= (property |_this| (get |descriptor| "key"))
                              (get |descriptor| "value")))))
                 (return |_this|))
               (call-internal |inherits| ,class |superParameter|)
               (call-internal |createClass| ,class |props|)
               (return ,class))
            ,super))))

which is quite close from working but my understanding of the compilers is for now too limited to actually get it working.

I would like a call to

(create-class |MyArray| |Array|
  (:reverse (lambda (this) <CODE-THAT-REVERSES-THE-ARRAY>
   :partition (lambda (this predicate) <CODE-THAT-PARTITIONS-THE-ARRAY>))

to expand adequately but my issue is that the symbols |MyArray| and |Array| should be interpreted literally while the code I generate tries to dereference them. How could I fix this?

For instance, the snippet

  (or (get ,class "__proto__")

should be expanded to

  (or (get <The JavaScript reference to MyArray> "__proto__")

but instead the symbol with name "MyArray" is bound to a local variable and its value cell is used in place of ,class.

See also: #294, where the discussion started.

foretspaisibles commented 6 years ago

Thanks to @davazp comments in #297 I could make some progresses towards a viable definition of a create-class compilation. It looks like this:

(define-compilation create-class (class super props)
  (let ((actual-super
          (cond
            ((stringp super)
             (make-symbol super))
            ((fboundp super)
             `(get ,(convert super) "fvalue"))
            ((boundp super)
             `(get ,(convert super) "value"))
            (t (error "~A: Invalid super class specifier." super))))
        actual-props)
    (do* ((tail props (cddr tail)))
         ((null tail))
      (push (list 'object "key" (car tail) "value" (convert (cadr tail) *multiple-value-p*)) actual-props))
    `(call-internal |js_to_lisp| (selfcall
      (var (,(make-symbol class)
            (call
             (function (|superParameter|)
                       (var (|props| ,(apply #'vector actual-props)))
                       (call-internal |inherits| ,(make-symbol class) |superParameter|)
                       (named-function ,(make-symbol class) ()
                                       (call-internal |classCallCheck| |this| ,(make-symbol class))
                                       (var (|_this|
                                             (call-internal |possibleConstructorReturn|
                                                            |this|
                                                            (method-call
                                                             (or (get ,(make-symbol class) "__proto__")
                                                                 (method-call |Object| "getPrototypeOf" ,(make-symbol class)))
                                                             "call"
                                                             |this|))))
                                       (var |i|)
                                       (for ((= |i| 0) (< |i| (get |props| "length")) (post++ |i|))
                                            (var (|descriptor| (property |props| |i|)))
                                            (if (in "value" |descriptor|)
                                                (if (=== (typeof (get |descriptor| "value")) "function")
                                                    (= (property |_this| (get |descriptor| "key"))
                                                       (method-call (get |descriptor| "value") "bind" |this|))
                                                    (= (property |_this| (get |descriptor| "key"))
                                                       (get |descriptor| "value")))))
                                       (return |_this|))
                       (call-internal |createClass| ,(make-symbol class) |props|)
                       (return ,(make-symbol class)))
             ,actual-super)))
      (return ,(make-symbol class))))))

It expects class name and super which could either be a string, such as "React.Component" or a symbol holding such a component (not tested yet). My construction is not totally good, indeed, the simple component definition

JSCL> (with-compilation-environment (compile-toplevel '(jscl::fset
 'my-todo-list
 (jscl::my-create-class
  "TodoList" "React.Component"
  ("render"
   (lambda (&rest react-internal)
      (jscl::/debug "my-todo-list#render")
     (create-element "h1" "Hello, World!")))))))

renders as

var l1 = internals.intern("MY-TODO-LIST");
var l2 = internals.intern("NIL", "COMMON-LISP");
var l3 = internals.make_lisp_string("my-todo-list#render");
var l4 = internals.make_lisp_string("h1");
var l5 = internals.make_lisp_string("Hello, World!");
var l6 = internals.intern("CREATE-ELEMENT");
l1.fvalue = internals.js_to_lisp(
  (function() {
    var TodoList = (function(superParameter) {
      var props = [
        {
          key: "render",
          value: function JSCL_USER_NIL(values) {
            var v1 = l2.value;
            var I;
            for (I = arguments.length - 1 - 1; I >= 0; I--) {
              v1 = new internals.Cons(arguments[I + 1], v1);
            }
            var v2 = this;
            console.log(internals.xstring(l3));
            return l6.fvalue(values, l4, l5);
          }
        }
      ];
      internals.inherits(TodoList, superParameter);
      function TodoList() {
        internals.classCallCheck(this, TodoList);
        var _this = internals.possibleConstructorReturn(
          this,
          (TodoList.__proto__ || Object.getPrototypeOf(TodoList)).call(this)
        );
        var i;
        for (i = 0; i < props.length; i++) {
          var descriptor = props[i];
          if ("value" in descriptor)
            if (typeof descriptor.value === "function")
              _this[descriptor.key] = descriptor.value.bind(this);
            else _this[descriptor.key] = descriptor.value;
        }
        return _this;
      }
      internals.createClass(TodoList, props);
      return TodoList;
    })(React.Component);
    return TodoList;
  })()
);

The generated code for the anonymous callback is wrong. The expression argument to return is l6.fvalue(values, l4, l5); where the values are not bound. When running this code triggers an exception in Chrome and the debugger shows that values is an undefined variable.

Using the alternative definition

JSCL> (with-compilation-environment (compile-toplevel '(jscl::fset
  'my-todo-list
  (jscl::my-create-class
   "TodoList" "React.Component"
   ("render"
    (lambda (&rest react-internal)
      (jscl::/debug "my-todo-list#render")
      (let ((answer (create-element "h1" "Hello, World!")))
        answer)))))))

leads to the generated code

var l1 = internals.intern("MY-TODO-LIST");
var l2 = internals.intern("NIL", "COMMON-LISP");
var l3 = internals.make_lisp_string("my-todo-list#render");
var l4 = internals.make_lisp_string("h1");
var l5 = internals.make_lisp_string("Hello, World!");
var l6 = internals.intern("CREATE-ELEMENT");
l1.fvalue = internals.js_to_lisp(
  (function() {
    var TodoList = (function(superParameter) {
      var props = [
        {
          key: "render",
          value: function JSCL_USER_NIL(values) {
            var v1 = l2.value;
            var I;
            for (I = arguments.length - 1 - 1; I >= 0; I--) {
              v1 = new internals.Cons(arguments[I + 1], v1);
            }
            var v2 = this;
            console.log(internals.xstring(l3));
            return (function(v3) {
              return v3;
            })(l6.fvalue(internals.pv, l4, l5));
          }
        }
      ];
      internals.inherits(TodoList, superParameter);
      function TodoList() {
        internals.classCallCheck(this, TodoList);
        var _this = internals.possibleConstructorReturn(
          this,
          (TodoList.__proto__ || Object.getPrototypeOf(TodoList)).call(this)
        );
        var i;
        for (i = 0; i < props.length; i++) {
          var descriptor = props[i];
          if ("value" in descriptor)
            if (typeof descriptor.value === "function")
              _this[descriptor.key] = descriptor.value.bind(this);
            else _this[descriptor.key] = descriptor.value;
        }
        return _this;
      }
      internals.createClass(TodoList, props);
      return TodoList;
    })(React.Component);
    return TodoList;
  })()
);

which renders appropriately in chrome. (Yeah!)

How can I adjust my handling of the props argument in my definition of create-class to avoid generating the problematic reference to values ?

davazp commented 6 years ago

I can try to examine your code to find the problem with values. But I don't like this approach too much. I see two other options that look more appealing:

The second approach looks more elegant from the implementation point of view, but it is not as effective I guess. For example, the generated code would be much larger. So I think I prefer the first one. The user is always able to go with the second approach if a different variation is desired.

foretspaisibles commented 6 years ago

What do you mean by embracing ES6? Generating ES6 code instead of plain old JavaScript code? That would actually be much easier but I would need to research about support and portability – the last time I checked ES6 was not supported anywhere without Babel, but it was quite a long time ago!

Look at the compiled code from babel and try to implement it as a macro, not directly within the compiler. If you can't, then we ca add a few other primitives to the compiler.

Actually this was my approach until now and it works but it has the downside that all components created bear the name of the local variable used when defining the component. My code is what Babel generated and look like

function _jsclReact_createComponent(componentName, actualSuperClass, props) {

    var subClass = function (superClass) {
        function subClass() {
            _jsclReact_classCallCheck(this, subClass);

            var _this = _jsclReact_possibleConstructorReturn(this, (subClass.__proto__ || Object.getPrototypeOf(subClass)).call(this));

            for (var i = 0; i < props.length; i++)
            {
                var descriptor = props[i];
                if("value" in descriptor) {
                    if(typeof descriptor.value === "function"){
                        _this[descriptor.key] = descriptor.value.bind(_this);
                    } else {
                        _this[descriptor.key] = descriptor.value;
                    }
                }
            }
            return _this;
        }

        _jsclReact_inherits(subClass, superClass);
        _jsclReact_createClass(subClass, props);

        return subClass;
    }(actualSuperClass);

    return subClass;
}

which leads debuggers (Chrome, Safari) to tag each component as subClass which is not very informative.

I will try to implement the first approach but I expect to run into the very same issue.

davazp commented 6 years ago

I'm not sure what we should do :-) I was just trying to give some ideas, but all options are open for now! Compatibility: https://caniuse.com/#feat=es6-class

which leads debuggers (Chrome, Safari) to tag each component as subClass which is not very informative.

I see. That is what I meant about adding primitives. Instead of adding classes directly, we could just add a way to bind a value to a given variable by name with a string (or many other approaches).

For example, I faced a similar problem when writing defun. I solved it by introducing named-lambda in the compiler. The minimum thing I needed. (For example, could we introduce named-object in the compiler?)

Again, you are probably more familiar with the problem that you are trying to solve that I am. Don't take my word as authoritative at all. If you think another approach is better, propose it :), but if possible I would like to keep the compiler as simple and general as possible.

foretspaisibles commented 6 years ago

Thanks a lot, as always your feedback is much appreciated!

Good news here: I made significant progresses on that and could build react components from the JSCL side as well as building minimal applications. This can be tried from the branch explore/react after bootstrapping JSCL, the command (jscl::react-1) will prepare the examples. Of course, it is still at a very early stage but substantial enough to be presented and discussed (IMHO).

I defined a %create-class primitive

(define-compilation %create-class (class super &rest methods)
  (let (compiled-methods)
    (dolist (method methods)
      (destructuring-bind (name ll &rest body) method
        (push (compile-method ll body :name name)
         compiled-methods)))
    `(class ,class ,super ,@compiled-methods)))

That can be used to define a JavaScript CLASS inheriting from SUPER (unless it is NIL) with the given METHODS:

(%create-class ("TaggedArray" "Array"
  ("tagObject" (tag-value)
    (setf (oget this "tag") value)))

The %create-class relies on a new compilation compile-method which is derived from compile-lambda and captures some specificities of JavaScript methods – when features are fixed it is an open question if and how this should be unifies with compile-lambda:

(defun compile-method (ll body &key name block)
  (multiple-value-bind (required-arguments
                        optional-arguments
                        keyword-arguments
                        rest-argument)
      (parse-lambda-list ll)
    (multiple-value-bind (body decls documentation)
        (parse-body body :declarations t :docstring t)
      (declare (ignore decls documentation))
      (let ((n-required-arguments (length required-arguments))
            (n-optional-arguments (length optional-arguments))
            (*environment* (extend-local-env
                            (append (ensure-list rest-argument)
                                    required-arguments
                                    optional-arguments
                                    keyword-arguments
                                    (ll-svars ll)))))

        `(method ,name
                 ,(mapcar (lambda (x)
                                       (translate-variable x))
                                     (append required-arguments optional-arguments))
                 ;; Check number of arguments
                 ,(lambda-check-argument-count n-required-arguments
                                               n-optional-arguments
                                               (or rest-argument keyword-arguments))
                 ,(compile-lambda-optional ll)
                 ,(compile-lambda-rest ll)
                 ,(compile-lambda-parse-keywords ll)
                 ,(when (string/= "constructor" name) (bind-this))
                 ,(let ((*multiple-value-p* nil))
                    (if block
                        (convert-block `((block ,block ,@body)) (string/= "constructor" name))
                        (convert-block body (string/= "constructor" name)))))))))

The most important variants are:

(define-raw-builtin super (&rest args)
  `(progn
     (call |super| ,@(mapcar #'convert args))
     ,(bind-this)))

This is hacky but I cannot think of a better approach – aside maybe editing the function body as I did for the React experimental and partial bindings.

A serious limitation is that parameters to the constructor are not supported!

I think sensible next steps on this work would be:

I am totally ignorant about webpack, do you have any short introduction to this tool you would recommend or suggest?

davazp commented 6 years ago

Cool. It's looking good.

(define-raw-builtin super (&rest args) `(progn (call |super| ,@(mapcar #'convert args)) ,(bind-this)))

How would you call the parent method of a class? Perhaps we can do that with another primitive super-method.

Do you think it's possible to wrap super in a normal JS function? Not sure how that would work with the JS semantics. The idea is to have super as a first-class value. That you could assign to any variable. If it's not, then just a built-in primitive is fine.

But we should make it work nicely together. Using super outside should give an error at compilation-time. The go special form does something similar. Only works inside a tagbody.

There are also many cases we would leave outside, like static methods... perhaps we can use CL declarations for that :-).

(%create-class ("TaggedArray" "Array"

The second argument can be any expression I think? Like in

class Bar extends calculatorMixin(randomizerMixin(Foo)) { }

It's a pity we can't use symbols for class and methods names. I understand there is a single namespace. Perhaps they should be string-designators... not convinced yet, strings sound good for now.

(%create-class string expression ...methods)

where is method is

("name" lambda-list ...body...)

like

(%create-class "componentName" #j:React:Component
  ("constructor" (props)
    (super props))

  ("render" ()
    :static nil    ;; optional, nil is the default
     )

I think we are getting close to have a nice first version for our %create-class special form. If you can research those things.. also feel free to open a PR with prototypes if you want.

mmontone commented 1 year ago

Hi,

I'm using this macro for dealing with objects and properties, that I think reduces a bit the boilerplate:

(defmacro jscl::with-object-properties (properties js-object &body body)
  "Access properties of JS-OBJECT and bind them in BODY.
Properties can both get accessed and assigned (they are bound using SYMBOL-MACROLET).
PROPERTIES is a list of symbols or (var-name property-name).
The string name of the actual properties is obtained from the symbol name.
So, the property name for a property 'some-property is \"some-property\".
For more control over the property name, use (var-name property-name) as binding.
For example, (some-property \"someProperty\").

Examples:
(with-object-properties (my-prop) some-object
  (print my-prop)
  (setf my-prop 'new-value))

(with-object-properties ((my-prop \"myProp\")) some-object
  (print my-prop))

Also create new objects with properties:
(let ((obj (jscl::new)))
  (with-object-properties (x y) obj
    (setf x 40)
    (setf y 50))
  obj)
"
  (let* ((obj (gensym))
     (bindings (mapcar
            (lambda (prop)
              (multiple-value-bind (var-name prop-name)
              (cond
                ((symbolp prop)
                 (values prop (string-downcase (symbol-name prop))))
                ((listp prop)
                 (values (the symbol (car prop)) (the string (cdr prop))))
                (t (error "Invalid property binding: ~s" prop)))
            (list var-name `(jscl::oget ,obj ,prop-name))))
            properties)))
    `(let ((,obj ,js-object))
       (symbol-macrolet ,bindings ,@body))))

Please let me know if you have any thoughts. Thanks.

mmontone commented 1 year ago

Also this utility for creating Javascript objects with properties more easily:

(defun jscl::new-with-properties (&rest props)
  "Create a new Javascript object with properties.
PROPS is a property list with property names and values.

Example:
(new-with-properties :a 33 :b 40)

it equivalent to the Javascript object:
{a: 33, b: 40}
"
  (let ((obj (jscl::new)))
    (do ((ps props))
    ((zerop (length ps)))
      (let ((key (pop ps))
        (val (pop ps)))
    (let ((key-name (cond
              ((symbolp key)
               (string-downcase (symbol-name key)))
              ((stringp key)
               key)
              (t (error "Invalid property name: ~s" key)))))
      (setf (jscl::oget obj key-name) val))))
    obj))
hemml commented 1 year ago

I'm using a bit more compact version in my library:

(defun make-js-object (&rest plist)
  (let ((obj (jscl::new)))
    (loop for (k v) on plist by #'cddr do (setf (jscl::oget obj (symbol-name k)) v))
    obj))

Also, a function to create DOM elements with hierarchical properties may be useful:

(defun js-string-split (path ch)
  "Split a string path by the character ch, much, much fater then JSCL function"
  (let* ((pos (position ch path))
         (p0 (subseq path 0 pos)))
    (cons p0 (if pos (js-string-split (subseq path (+ pos 1)) ch) (list)))))

(defun deep-oget-1 (el path)
  "Get a value of el.the.list.of.the.property.path"
  (if (cdr path)
      (deep-oget-1 (jscl::oget el (car path)) (cdr path))
      el))

(defun create-element (typ &rest args)
  "Create and return a DOM element type ``typ'' and specified attributes: (create-element \"A\" '|href| \"http://google.com\")"
  (let ((el ((jscl::oget (jscl::%js-vref "document") "createElement") typ)))
    (loop for (k v) on args by (function cddr) do
      (let* ((path (js-string-split (symbol-name k) #\.))
             (obj (deep-oget-1 el path))
             (fld (car (last path))))
        (jscl::oset v obj fld)))
    el))

It can be used like:

(create-element "div" :|style.border| "1px solid black" :|innerHTML| "Hello!")
mmontone commented 1 year ago

Nice. I think it would be great if we could agree on a set of FFI utils and include with JSCL.

Also improve documentation on FFI specially. Took me a while to find out that this the syntax to call JS methods:

((jscl::oget jscl/ilt::*websocket* "send") message)
vlad-km commented 1 year ago

Look at https://github.com/jscl-project/jscl/discussions/449 Welcome and join