search

Tangerine-Community / Tangerine

Digitize your offline data collection. Create your Forms online with Tangerine Editor, conduct them offline with the Tangerine Android App. All results you collect can be exported as a CSV file, easy for processing in a spreadsheet. Tangerine has been used in over 1 million assessments and surveys in over 60 countries and in 100 languages.
http://www.tangerinecentral.org/
GNU General Public License v3.0
47 stars 30 forks source link

Data collector can collect data with forms #232

Closed rjcorwin closed 6 years ago

rjcorwin commented 7 years ago

Architecture

Component Hierarchy influenced by QTI

Original research: https://github.com/Tangerine-Community/Tangerine/issues/89

We are moving from a Component Hierarchy that results in hierarchies such as...

To a hierarchy where a Workflow is called a Form separated out into a tree of Sections and things like Location Subtest and Question are at the same Element level and can be placed on a Page.

As an Entity Relationship Diagram, it looks like this.

screen shot 2017-05-05 at 4 29 38 pm

WIP ER Diagram: https://docs.google.com/drawings/d/19ej1NX3q2E5N9pe3yn9ETBKjZiFi6FDRb8SvAXBcWpg/edit

1 - Form Component

Goal: A Form Component as a Controller dictated by JSON for displaying a flow of Sections and Pages with Results being collected along the way.

Two directions to think about.

2 - Page Component

Goal: Given a JSON object (not a JS object), generate a page with any combination of Element Components as dictated by the JSON object.

The impression we got at NG Conf was that modifying Component templates during runtime is not very "Angulary". This is a paradigm shift for Tangerine from the Backbone/jQuery days where in a View we were able to, based on a JSON property, instantiate a user configured View Class into another View's template. There are a couple of approaches to this we've determined to be available in Angular.

  1. Pure AOT: Each Page as an actual Component that gets compiled on runtime. Editor would use a JSON to code generator but we would also have the flexibility to write custom Forms as individual Angular Components.
  2. "JIT" like: JSON based with a foreach with a switch in the template. See angular-json-form module.
  3. Not Angular: Use a different View library wrapped up in an Angular Component.
  4. Hacky Angular technique?: ComponentFactoryResolver https://angular.io/docs/ts/latest/cookbook/dynamic-component-loader.html

Libraries

NG Formly

Angular JSON Schema Form

NG2 Formio

SurveyJS

Teradata's Covalent Dynamic Forms Component

Custom Module based on the AOT Foreach Switch approach

rjcorwin commented 7 years ago

Consider RxJS for Result streams.

https://medium.com/@luijar/rxjs-pouchdb-persistent-data-flows-480f503ee41f#.3g1vv4jhz

rjcorwin commented 7 years ago

I think ng2-formio might be a wrapper around a React component. Question asked here https://github.com/formio/ng2-formio/issues/82

That architecture might not be a bad thing, especially if it's well maintained. If we're opening our JSON -> Form mechanism up to other frameworks, this really opens it up.

rjcorwin commented 7 years ago

So far the json-schema-form Angular module and associated component is the most Angular centric JSON to Form library I have seen.

https://github.com/dschnelldavis/angular2-json-schema-form/tree/master/src/library

It pulls in a bunch of other Angular components as seen here https://github.com/dschnelldavis/angular2-json-schema-form/tree/master/src/widgets

The Component hierarchy as seen in Augury, is interesting, especially when comparing Bootstrap vs Material.

screen shot 2017-04-05 at 5 45 07 pm screen shot 2017-04-05 at 5 46 56 pm
rjcorwin commented 7 years ago

More options:

rjcorwin commented 7 years ago

@evansdianga If you take a look at formio, I asked them about architecture and they provided this diagram https://github.com/formio/ng2-formio/issues/82#issuecomment-292030125

image

rjcorwin commented 7 years ago

@chrisekelley One of my favorite moments of Angular Conf was when I was describing to someone what we were trying to accomplish and they looked at me in the face and said, "You're describing a platform feature. Angular handles forms." While that wasn't the answer to the question I was trying to ask (pure JSON driven forms, not JS object driven forms), that's what sparked the idea for forms as pure code that get AOT compiled.

rjcorwin commented 7 years ago

@evansdianga Here's a good doc on using formio in Angular if you are interested in checking it out. https://github.com/formio/ng2-formio/wiki/1.%29-Form-Renderer

rjcorwin commented 7 years ago

Giving the formio component a try, I hit a blocker with getting the hello world started. I filed a bug in their issue queue https://github.com/formio/ng2-formio/issues/89.

rjcorwin commented 7 years ago

I'm blocked on getting angular2-json-schema-form working in a new project as well. There are numerous issues in the queue describing it's not working with Angular 4. There is a PR that claims to fix this but I'm having issues getting that PR working. See the PR and my comment here https://github.com/dschnelldavis/https://github.com/dschnelldavis/angular2-json-schema-form/pull/44/pull/44

rjcorwin commented 7 years ago

With no existing JSON-fed-form library at hand, I think I will start building the form player shell with a simple start to Grid Test component that is a form of checkboxes. This will get us going with flow and schema.

rjcorwin commented 7 years ago

ng-formly is another promising Angular Module for rendering a JSON but it also does not work with the most recent version of Angular. Bug filed https://github.com/formly-js/ng-formly/issues/411

rjcorwin commented 7 years ago

I was able to take an example of generating a form from JSON in the Angular Cookbook, implement it in a recently generated Angular App, modularize it into a discreet Angular Component, and then add a custom event emitter that signals when the form is submitted.

rjcorwin commented 7 years ago

We've had a nice response from the NG Formly maintainers. They pointed out that there is a beta where the issue is fixed, another maintainer even sent a pull request to my example code.

mohammedzamakhan commented 7 years ago

One of the NativeScript developer also contributed to ng-formly-nativescript https://github.com/formly-js/ng-formly-nativescript

Think about native support for Tangerine too 👍

evansdianga commented 7 years ago

The response was quick. Digging into the library deeply. The docs have referenced the changes. Please have a look.

rjcorwin commented 7 years ago

Thanks @mohammedzamakhan. We'll have a look.

rjcorwin commented 7 years ago

@evansdianga I added Covalent Dynamic Forms Component to our list in the description of this ticket. Go ahead and fill out the research there.

rjcorwin commented 7 years ago

I got the start of a basic Tangerine Form component going today. It takes JSON config and an @Input() property, stuffs that in NG JSON Form Component's own config property, waits until NG JSON Form emits it's done event, then emits its own done event spitting out the results. The App Component is an example of using Tangerine Form Component. Tomorrow starts the work of making it conform more to the schema proposal and step through sections/pages showing a series of NG JSON Form Components while collecting results along the way. At some point we may swap out NG JSON Form Component for another form library like Formly or Covalent's Dynamic Form Component.

code: https://github.com/rjsteinert/tangerine-form/tree/master/src/app

screen shot 2017-05-09 at 4 43 13 pm
rjcorwin commented 7 years ago

This is very preliminary proof of concept, but I now have a component called BinderComponent that accepts configuration that describe a series of forms, displays those forms in order as they are submitted, and then emits the array of results when the last form is submitted. The "binder" analogy is that you give a binder a bunch of pages that are forms and the binder is what holds the pages together so you can flip through.

  1. AppComponent binds an Array of NG JSON Schema Form Arrays to BinderComponent's config input.
  2. BinderComponent then takes the first entry in the config Array, set that to currentPage, then binds currentPage to NgJsonFormComponent's config input.
  3. NgJsonFormComponent displays the form.
  4. When NgJsonFormComponent emits the jsonFormSubmit event, BinderComponent is listening so it receives the form data in the event handler which it then stuff into BinderComponents's results Array.
  5. Then BinderComponent takes the next entry in its config array and sets that to currentPage which then triggers results in a the same chain of events that cause the form to be displayed.
  6. When BinderComponent sees that there is no more entries in the config Array, it emits a binderDone event and passes the results along with that event. AppComponent is listening for this event and when it hears it, receives the results in the event handler and prints them into a text box below the BinderComponent.

screen shot 2017-05-10 at 2 26 18 pm

https://github.com/rjsteinert/tangerine-form/tree/master/src/app

rjcorwin commented 7 years ago

The next step is to finish the pull request that starts to implement features from the current Tangerine v3 spec https://github.com/rjsteinert/tangerine-form/pull/1/files

rjcorwin commented 7 years ago

The Tangerine Form repository I have been working on now use a schema that is very similar to our schema proposal. The Binder Component is now able to walk though a tree of sections as opposed to a linear array of pages. Tomorrow I will copy this prototype into a new Tangerine 3.x.x branch, wire it up to build in the Tangerine Docker builds, and then tag it as Tangerine v3.0.0-alpha1. Perhaps I'll have enough time to replace our very own basic page library (NG JSON Form Component) with something more advanced like Formly.

The pull request the got us to this next level: https://github.com/rjsteinert/tangerine-form/pull/1/files

rjcorwin commented 7 years ago

Meet Tangerine v3.0.0-alpha1.

screen shot 2017-05-12 at 1 50 23 pm

In the v3.0.0 pull request you'll find...

rjcorwin commented 7 years ago

@evansdianga I posed the extensibility question to the covalent maintainers https://github.com/Teradata/covalent/issues/585

rjcorwin commented 7 years ago

Here's a couple more features we could add to Sections as described by QTI Section Schema. These should probably be split out into a couple more User Stories.

rjcorwin commented 7 years ago

Today I gave replacing ng-json-form with ng-formly and quickly ran into an issue where it looks like using Material with ng-formly isn't very well support https://github.com/formly-js/ng-formly/issues/335. My work on that is in the v3.0.0--formly branch, diff as seen in this PR https://github.com/Tangerine-Community/Tangerine/pull/304.

Switching gears, I gave Covalent's td-dynamic-form component a try. A bit more success with that library so far as I have the basic demo working via this pull request https://github.com/Tangerine-Community/Tangerine/pull/305.

Getting data out of a td-dynamic-form component was a challenge. It may seem obvious how to get the data out to some but there is no mention of it in the docs. I eventually found this issue in their queue that describes a couple of ways of doing it https://github.com/Teradata/covalent/issues/479. I was hoping they would have an event we could bind to and receive form output from that event but alas it involves the local variable method which gives you access to the built Angular FormGroup object. This kind of makes sense as to not reinvent the wheel but there is still something not right about the use of "local variable". My fear is that the use of "local variable" encourages an undocumented API for a Component... and this is a perfect example.

If we're going to continue with the use of Covalent's Dynamic From Component we'll need to get a module system into it soon otherwise we'll need to maintain a fork with our own custom element components. That is perhaps a lesser of two evils when compared to having to maintain the whole thing without an upstream repository we can pull and push code from.

rjcorwin commented 7 years ago

Binder has matured to have a working skip logic mechanism some other nice tweaks and helper methods as seen in this PR https://github.com/Tangerine-Community/Tangerine/pull/308/files

Below is what I'm thinking for next week, alpha2. This will be on a fresh v3.x.x branch and I'm going to cherry pick from the other experimental branches I've been working on.

rjcorwin commented 7 years ago

Some notes from playing around with SurveyJS.

Two hello worlds created. First one as a clone of their own hello world using Angular CLI but I added more features and the second is from using Angular CLI and generate from scratch.

The good:

The bad:

Areas where we could help improve SurveyJS:

rjcorwin commented 7 years ago

On that downside of SurveyJS using the React View library, it's interesting to see at Google IO, Angular described as much more than just a View library. In this case, Angular is the forklift that is assembling Web Components.

screen shot 2017-05-22 at 3 02 34 pm

Angular is great for structuring and organizing an application... Ex. CLI for scaffolding, building, shipping, and distributing or other tools within framework which are all about architecting, controlling flow...

... a lot of effort went into the Angular rewrite to make sure that Web Components are well supported.

But what is React? It's interesting to read what React is when discussed in the context of Web Components. Here is an excerpt from this React docs page.

React and Web Components are built to solve different problems. Web Components provide strong encapsulation for reusable components, while React provides a declarative library that keeps the DOM in sync with your data. The two goals are complementary. As a developer, you are free to use React in your Web Components, or to use Web Components in React, or both.

I think that's a really good description of React. It's also a good description of just ONE part of Angular. This helps us imagine the overhead cost of using Angular's and React's different ways of providing "... a declarative library that keeps the DOM in sync with your data".

rjcorwin commented 7 years ago

Form Library Analysis

If we want to join an existing ambitious community looking to develop out complex forms from JSON, SurveyJS or Form.io are fine options but it would require developing it separate from our our Client App and not without Angular's Components. SurveyJS's use of React and TypeScript sounds better for building a Form Player than Form.io's use of plain ES6 yet Form.io's Form Editing efforts are going to be much more collaborative given they already intend to build an MIT licensed Form Editor that is also Angular based. Meanwhile SurveyJS's editor requires a commercial license.

If we want to stick to Angular, there are no existing projects that are as ambitious as ours but Covalent's TdDynamicForm Component or Formly could be used as a Page component. Between the two I think Formly is a better fit because I think it is the more ambitious and dedicated of the two. Covalent's TdDynamicForm Component is really just one of many components in a library. While Formly doesn't currently have great Material support, I don't think it would be much effort to tie in the existing Angular Material Components.

mohammedzamakhan commented 7 years ago

Form elements are not currently extendable using additional modules, just one Module for all the form elements.

Can you create the issue in formly repo, because I want to understand more about this point. Thanks

rjcorwin commented 7 years ago

@mohammedzamakhan I wasn't originally understanding how to declare custom form components so we can use them in Formly but I think I understand now. In the introductory example, you use both the Bootstrap Form Component library and then declare some custom ones using FormlyModule.forRoot(...).

screen shot 2017-05-25 at 12 13 43 pm
rjcorwin commented 7 years ago

@chrisekelley @evansdianga After many attempts at creating a TangerineForm Component that feels right, I think I've finally nailed down something. The following is a structure for the TangerineForm Class that has a clear separation between state and configuration. It's something that I think the TangerineFormComponent can easily use to navigate around the TangerineForm configuration and represent the state. To help TangerineFormComponent maintain state, I see a lot of promise in the Redux pattern which uses streams of packets to populate the state by way of a reducer.

TangerineForm Class example in loose YML:

  ngFormObject: { ... }
  state:
    form:
      currentPage: '/b/1',
      markedDone: false
    sections:
      '/a':
        state: 'COMPLETE'
      '/b':
        state: 'IN_PROGRESS'
      '/c':
        state: 'SKIP'
    pages:
      '/a/1':
        status: 'VALID'
        model:
          variable1: 'foo'
          variable2: 'bar'
      '/a/2': 'skip'
        status: 'VALID'
        model:
          variable3: 'yar'
          variable4: 'baz'
      '/b':
        status: 'IN_PROGRESS'
      '/b/1':
        status: 'INVALID'
        model:
          variable5: 'fin'
          variable6: ''
      '/b/2':
        status: 'UNTOUCHED'
        model:
          variable7: ''
          variable8: ''
      '/c/1':
        status: 'SKIP'
        model:
          variable9: ''
          variable10: ''
  configuration:
    pages:
      '/a/1': { ... }
      '/a/2': { ... }
      '/b/1': { ... }
      ...
    pageOrder: [ '/a/1', ... ]
    sections:
      '/a': { ... }
      '/b': { ... }
      '/c': { ... }
    sectionOrder: [ '/a', '/b', '/c' ]

Example of a stream:

  action: 'USER_INPUT'
  id: '/b/1'
  payload:
    status: 'INVALID'
    model:
      variable5: 'fin'
      variable6: ''
rjcorwin commented 7 years ago

The last thing to do on this ticket is to make sure form sessions are saved to the database.

rjcorwin commented 7 years ago

@lachko Put this in review. Time to close it?

rjcorwin commented 6 years ago

Epic ticket. Closing now.