Advanced widgets, any interest to include them in Deform?

[tdamsma]

I have created a few "advanced" widgets for my projects based on deform, and I would be more than willing to give them back to this repo but not sure if they would fit in as they are quite js heavy (and thus require lots of js to be put in the statics directory or links to a cdn).

Here is the list:

• Select2SortableWidget

  Based on http://finn.heemeyer.net/select2.sortable/, extension of select2 that allows drag and drop to change the order of the selected items

• JsonWidget

  Json editor based on https://ajv.js.org/. Validates against a jsonschema on both server and client side

• ReactJsonSchemaFormWidget

  Client side form generator based on https://github.com/rjsf-team/react-jsonschema-form. Validates against a jsonschema on both server and client side. (About 500kb of minified js)

• JExcelObjectArrayWidget

  Based on https://jexcel.net/v3, allows editing a List[Dict[str:Union[str,int,float,bool]], as an excel like table where every row corresponds to a dict in a list. (About 600kb of non-minified js/css)

Effectively these are all wrappers around existing js libraries. For my projects I just vendored it in, but especially for the react widget this is not trivial as it has a complete build toolchain. So my question is if there is interest in me trying to make these available as part of Deform. And if so, how to deal with the js dependencies.

[stevepiercy]

Wow. That is quite impressive. Thank you for your offer.

For widgets to make it into the Pylons Project officially supported Deform project, see contributing.md. JS and CSS dependencies are handled as described in https://docs.pylonsproject.org/projects/deform/en/latest/widget.html. All of that can be... a lot.

Alternatively we have been discussing how to share advanced widgets that use Deform, exactly as you bring up. We have already decided to create a new repo for this purpose. I don't have any details at this time, as I have not thought through how we would structure it. It might be as simple as a typical "awesome" repository, a community cookbook, a modified deformdemo, or something else. @sydoluciani might have some thoughts to share.

Whatever it is, it should be easy to maintain for the contributor and maintainer, and easy for the developer to adopt.

Anyway, to answer your question about the four widgets, I would say:

• Select2SortableWidget seems straightforward, and not much more difficult than adding the Select2Widget itself was.
• JsonWidget sounds reasonable, but I'd like to see a link to a demo first.
• ReactJsonSchemaFormWidget is useful, but I imagine it is way too difficult to bring it into Deform due to the React requirements. I would prefer to see this in a cookbook recipe first to see what is involved.
• JExcelObjectArrayWidget looks useful, and a lot better than an ugly "copy-paste a CSV file into a textarea" solution.

Please let me know. Thank you!

[tdamsma]

Ok great that there is interest. I'll try to make a standalone demo somewhere and get back to you

[sydoluciani]

@tdamsma Thank you, that is great if you can work on new widgets, specially if we could include React to start with.

I think it is best we leave Deform simple with no to minimum changes, and have most of work done in front end.

Deform depends on Colander for validations and Peppercorn for deflating form inputs, so we might need to write an adaptor/proxy or convertor to read or write to/from cstruct or pstruct either in React side or Deform side to have seamless integration.

@tdamsma what do you think ?

like what @stevepiercy suggested, we can start with github repository with a simple React form skelton and take it from there. not necessary starting with another form generator, just a simple React form to have structure right, then other contributors might be interested in working on it as well.

Please check Razzle for setup example.

This is a lot of work, and it is better if we focus only on React, start simple and bring it to production level.

[sydoluciani]

Razzle is using Webpack, to separate javascript code that needs to run in server side from javascript that needs to run in client side, and I think we need to have such structure for both Deform and Pyramid to have developers inline for future development. such structure removes the dependency from front end technolgies, while we we can focus on React as primary front end. I am not saying we should use Razzle, but we need to have structure for curent and future frontend technologies.

One important note is, we probably don't want to have another node.js server in front of Pyramid, like explained in this site, but instead, just a structure to produce the final javascript to be included in Deform or Pyramid. With such structure Deform will be using select2.js installed by npm instead of including it in source code.

such structure needs to be engineered before deployed, so lets start with simple React login form with both server side and client side validation in github, document it, and take it from there.

[tdamsma]

So as promised, here is a deformdemo fork with my widgets hacked in: https://github.com/tdamsma/deformdemo/tree/advanced-widgets. It includes a react jsonschema form that is build on top of (not instead of) the colander schema, though it is actually bypassing most functionality. Internally we actually have developed a jsonschema to colander translating function, but it only supports a subset of jsonschema (exactly the bit we use) and nothing more as the jsonschema spec is huge.

@sydoluciani I'm not sure I think rewriting deform into react is a good idea, if I understand you correctly. If I want a react app, I'll just make a separate frontend. For me the power of deform is that it works with a simple server that generates html forms. And when you need some extra functionality, you can use widgets that where needed. If you really need advanced interactive forms, serverside html with predefined widgets is probably bot the way to go. That is where I switch to a JS SPA using Vue or React.

If you want a statically compiled piece of react that you can instruct to generate any colander form, all you need a jsonschema representation of the data and then use https://github.com/rjsf-team/react-jsonschema-form to render it into a form. Great thing about that is that you get a fully documented json api almost for free, as each form has a jsonschema that exposes the validation rules in a structured way.

Anyways, if you clone my fork and serve it you should get this:

[tdamsma]

I added one more widget. I am aware the widgets would need some cleaning up etc, I just extracted them from an internal repo without refactoring properly. But hopefully this is good enough to advance the discussion:

Are these widgets useful enough to maintain them?

If yes, where to land the, In deform? In an example repo? I would prefer to have this go straight into deform, but I don't think I know all the implications of that

[stevepiercy]

@tdamsma thanks for the clone. This is a lot to consider. It'll take me a week or so to try it out and think it over.

Deform is "intertwingled" with Peppercorn and Colander as described in Serialization and Deserialization. I've been curious to see how React can fit into this picture. I do not yet see how React can replace these components. More importantly, I would not harm the users of Deform with a replacement of Peppercorn and Colander, as it would require a complete rewrite of their apps for de/serialization and validation.

[stevepiercy]

@tdamsma side note, you might be interested in https://github.com/Pylons/pyramid_openapi3/ for another fully documented json api almost for free. Demo app and Demo API. I'm not sure of what it uses instead Deform/Peppercorn/Colander.

[sydoluciani]

@tdamsma Thank you.

We were talking about having separate repository for any new custom widgets or templates instead of including them in Deform, but including it in Deformdemo the way you have done is way more practical.

As soon as I saw react-jsonschema-form in the subject, noticed we need some sort of adaptor or convertor but didn't think of Python JsonSchema

It's an incredibly good contribution.

There is a deformdemo/custom_widgets directory that I can't agree with it's name, but it is designated for new templates, have you tried loading templates there and removing extra lines from line 47 to line 55 ?

Maybe having custom_widgets directory for your extra_widgets.py, each widget in individual files, then having custom_templates directory to have templates matching the widgets ?

I was not suggesting to rewrite Deform, instead we should keep Deform as it is simple and efficient, but moving toward using npm or yarn package management instead of copying JavaScript modules manually in Deform and then including the module, we should install JavaScript modules using npm or yarn, and having other tools to copy it over to static directory and maybe having the requirements in yaml file generated by webpack !!!! . so next time we want to upgrade any of the JavaScript modules like select2, we just use npm, yarn, webpack rather than copying the module manually.

At some point we could move select2 module from Deform to Deformdemo the way @tdamsma has done ithere and here to reduce dependency of Deform to any unnecessary JavaScripts.

[sydoluciani]

@stevepiercy conversions are smartly done in serialize method of widget, and finally rendered using field.renderer method.

Deform code is unchanged only Deformdemo is updated with new widgets.

With a few tweaks in Deformdemo we should have all new widgets and templates demonstrated in Deformdemo.

[stevepiercy]

Deform has two sets of functional tests, one at deform/tests/test_functional.py which runs functional tests within Deform alone, and the other in deformdemo at deformdemo/test.py. Anything in either repo has "official" Pylons Project support, which means it has tests and documentation.

The directory deformdemo/custom_widgets/ contains the retail form as described in the docs and is demonstrated at https://deformdemo.pylonsproject.org/popup/. It has no tests, but has docs in Deform and its docstrings, and is included for demonstration purposes only. This violates our Pylons Project official support policy of lacking tests.

However deformdemo is a unique beast, in that it is both tests and a demonstration of an official project (Deform), but is not released to PyPI. It's purpose is similar to the marketing websites, https://trypyramid.com/, https://pylonsproject.org/, and https://webob.org/, in that they all support an official project and are not released to PyPI but lack full "official-ness". We are not setting a new precedent.

Anyway all that is just a long explanation to say that, after further consideration, a new repo is not needed and that deformdemo may be used for custom widgets contributed by the community.

Going forward, I would like to see separation between the widgets that are used in Deform's functional tests by virtue of deformdemo/test.py and custom widgets. Something like this, where everything in the root directory except custom_widgets is the core of deformdemo and custom_widgets is the community sandbox:

.
├── __init__.py
├── custom_widgets
│   ├── __init__.py
│   ├── popup_example
│   │   ├── popup_example.py
│   │   ├── static
│   │   └── templates
│   │       ├── modal.pt
│   │       └── popup_example.pt
│   ├── my_widget1
│   │   ├── my_widget1.py
│   │   ├── static
│   |   │   ├── css
│   |   │   ├── js
│   |   │   └── img
│   │   └── templates
│   │       └── my_widget1.pt
│   ...
│   └── my_widgetN
│   │   ├── my_widgetN.py
│       ├── static
│       │   ├── css
│       │   ├── js
│       │   └── img
│       └── templates
│           └── my_widgetN.pt
├── templates
│   ├── code.pt
│   ├── form.pt
│   ├── index.pt
│   ├── main.pt
│   └── mini.pt
├── test.py
├── validation.py
└── validation.txt

Is that close to what you imagine, @sydoluciani?

For the management of Deform JavaScript modules (and CSS, fonts, and images), I tolerate the current method of copy-pasting the latest then letting it die on the vine. I would prefer to use npm/yarn/webpack and having GitHub Dependabot nag me about vulnerabilities. I am not very good at setting it up, but I am OK at using it.

[tdamsma]

I understand the desire to put the widgets in deformdemo from a maintainers perspective, but how would it work for a user? Would one import SomeWidgt from deformdemo.widgets, or copy paste the implementation from the source code? It feels a bt wierd to have deformdemo as a dependency, but equally weird to have to copy paste some code that from then on is probably unmaintained (or one would regularly have to check the reference implementation in deformdemo).

Cleanest would be a separate repo/release per widget, but that would be a little bit overkill and add extra maintenance burden. Or a separate repo that extends deform with extra widgets that can be installed alongside deform But if that would live in pylons/deformextensions then what is the advantage over just putting it in the pylons/deform repo directly?

As for the js part, I would suggest to provide a dockerfile next to every widget that somehow collects/builds/updates/minifies the static js/css. Perhaps the static files can then be excluded from the repo and only be copied into the releases.

[sydoluciani]

@tdamsma I think what you have currently done is the best way to go, thank you, but if you can please follow @stevepiercy suggested directory structure for custom_widget and make a pull request to include your contribution to Deformdemo.

There are way more JavaScript modules that can be included in Deform, and adding these JavaScript modules to Deform eventually make the upgrade a difficult job. and having separate repository is going to have the same effect of having the widget included in Deformdemo and users are going to somehow transfer or copy and paste the code if they need to.

We can add the npm/yarn/webpack option to milestone until someone has time to implement it. I need to have a closer look at couple of other Pyramid/Python projects and see how they have automated their front end deployment.

[stevepiercy]

@tdamsma Deformdemo is a demo of Deform widgets. It is not intended to be a dependency. It is a showcase of how to implement Deform widgets.

I want to clarify what are Deform's core widgets and a custom widget in deformdemo only.

A core Deform widget should not be a burden for maintainers more than the Select2Widget, for example, where we grab the jQuery plugin or JavaScript dist and toss it into the static directory (which is currently at 2.0MB in size). Users can install Deform and the core widget just works.

Otherwise it would go into the custom widgets as a demo, and, hopefully, with enough information for how to implement it.

As much as I would like to make it easy for developers to install a custom widget via a package manager or similar means, we are not there yet. Dockerfiles introduce another requirement that I don't want to support. Copy-paste is unfortunately, I think, the best method we can offer to developers for implementing custom widgets at this time.

To add a custom widget to deformdemo, I would take the example of the sole existing custom widget, and improve its organization, using the suggested directory structure above or something close to it.

To move this forward, I would like to see one PR for each of the widgets that I think should make it into Deform core (as well as deformdemo for tests). Select2SortableWidget is useful and pretty small. I'm also OK with JsonWidget.

I think the ReactJsonSchemaFormWidget is too complex and too large for Deform core. I changed my mind on JExcelObjectArrayWidget, because it increases Deform's size too much for an uncommonly used widget (it's not really "core"). Both of these would be nice to see in deformdemo as custom widgets, if possible.

[tdamsma]

@stevepiercy I'll add separate PR's for the deform widgets first. First one is the Select2Sortable widget. Don't know how to link the deform and deformdemo PR's as they are interdependent

[stevepiercy]

Correct. PyCharm has a nifty feature to push commits to multiple repos, but there is no direct linkage. I usually copy-paste the URL of the issue/PR that I create across repos so that they at least show up in GitHub as related. GitHub also has a feature to link issues to PRs.

[tdamsma]

I only just managed to setup the selenium test environment on windows+wsl2, which was more challenging than it should be. It is a whole other issue, but has there been any discussion migrating the test setup to dokcer/ docker-compose? That way you can cleanly define the test setup and have the same setup working on linux/osx/windows, both for development as for CI.

[stevepiercy]

For Windows setup, I would accept any updates to the contributing.md to pave the way for future Windows users. Most developers are on Linux or macOS, and only recently have we got the setup in a state that works reliably.

There has been no discussion about using Docker or Docker Compose. What is the problem that using them would solve? Would their use establish a barrier to entry for contributors? Who would support Docker related issues? We should discuss in a separate issue to avoid hijacking this issue.

[sydoluciani]

I have created #471 for any thought or input.

[tdamsma]

GitHub also has a feature to link issues to PRs.

I would if I could, but I think I don't have permission to add links in this repo