development timeframe & goals for interpreter [comment]

[100ideas]

Just wanted to say hi 👋 and register my interest in your project.

I'm working on an app that helps users create reusable workflows for validating and operating on blocks of semi-structured data from tabular sources such as csv / excel and json sources from web apis.

Currently I am using json-schema to represent a document of

• collections: schema & validation rules + refs to value store
• blocks: lists of actions + shape of input & output interface + dependencies on other blocks
• actions: list of computations or manual operations to execute on given data

I'd like to adapt an existing API or workflow definition/orchestration tool to my application, but I have yet to try adapting any of them. Most of the systems I've checked out are pretty complicated and I have hesitated over concerns about technical debt and the priority of other features I need to work on. (except for shortbus, it's simple!). That said, your vision for an abstract toolchain is appealing - I wish appML was mature, I'd probably be using it.

I'd like to know a little more about your vision for how users would use an 'alpha' version of appML, and in particular how you imagine users creating interpreters for appML so they can use it to drive their own APIs. If you give me a little direction there I could try some experiments with my own app.

Lastly, here are a couple of other open-source tools that involve complex user-defined data types, operations, their dependencies, and an execution engine. I know there are a bunch out there, these are ones I think are particularly interesting / relevant.

• antha-lang extension of go compiler to allow computations and scheduling of workflows comprised of data elements that represent real-world biology protocols. Elements themselves are represented as go classes, so they can have complex validation & inheritance.
• meshroom (python) graphical workflow builder that models interfaces and execution graph of variety of photogrammetry command-line tools; interestingly graph state is maintained as files & folders on disk. This latter feature is what I think is so interesting.
• eve (ts) reactive dataflow ide. I just discovered it uses chevrotain too. the compiler and DSL might be interesting as concrete examples of how to interpret/compile chevrotain ASTs.

[kristianmandrup]

Thanks for your interest. You might want to check out some of my other recent projects as well, trying to tie this all together, f.e, Json schema to ElasticSearch mappings and to Yup validation. Also have various mappers/converters for GraphQL. The idea is to use the appML language to declaratively describe an app and have the underlying engine, assembled from various adapters, generate the skeleton og the app and all tests. Then you can just hook onto this core app and add extra custom functionality, ideally cutting down the manual work and boilerplate by 90-95%.

Unlike your ideas, especially tools such as creating state machines or workflows using graphs or similar. Had the same vision of adding a Graph based GUI on top and have it sync with the language spec. Ideally a 3D VR graph that can be controlled via speech and hand gestures ;)

Let's get this rolling... I should write a few blog posts on this shortly when I find the time.

Cheers!

[100ideas]

I actually found your work via your other projects. I've been checking out a lot of projects related to json-schema, json-ld, json-api, openapi/swagger, common-workflow-language, tcomb, ajv, yup, etc. trying to figure out how to reasonably implement "type" system for for user-provided data tables (collections, above) that allows runtime type definition, inference, validation, and compatibility checking.

I get the general idea of appMl and have skimmed the code. I played around with your suggested demo in chevrotain. I read the chevrotain docs. And I looked through the eve source code to see how chevrotain interpreter is used in production.

I understand that appML has yet to implement (or already provides?) some kind of API that allows a user to hook their code-generation code into appML's chevrotain-based interpreter. (Or would a user specify this mapping in their appML langugage file directly?)

I just need specifics about how to hook my code-generation functions into appML, or how you imagine implementing that process, and I will play around. Thanks!

[kristianmandrup]

Sounds great! The current demo is not yet ready to be used for anything really. My idea was to divide the language into separate parts that can be merged/joined I to one, as you can see from the mono repo. I was lately investigating how to add VS code integration. You are most welcome to play with that first. Provided lots of links and docs and reference projects doing the same for Chevrotain based languages/DSLs.

Next step is to define a CST visitor that can output something useful such as a decorated JSON schema that can then be fed into various other generators like you mentioned. The workflow part could do a similar thing, perhaps generatinf a fully functional state machine.

[kristianmandrup]

I've also built an API/abstraction on top of Chevrotain to make it more useful and smoother API for what I wanted to do. See tests in chevrotain-* modules.