Data generation with schema

[giorgio-v]

In the readme you said that you want to provide the ability to generate data from a given schema.

Since I will need a feature like this in the near future I’d like to know if this is still a planned feature and, if so, whether or not you’d like to discuss its design here.

I’m willing to contribute some effort and spare time to implement this.

TIA

[w01fe]

Thanks for your interest in contributing. Yep, it's still planned. We have some prototypes in the works, (cc @davegolland ) and there are also some third-party things being worked on as well -- discussion here:

https://groups.google.com/forum/#!searchin/prismatic-plumbing/test.check/prismatic-plumbing/X9_ROgupzpI/8ilcuGfDdREJ

We don't have a concrete time frame for finishing it yet though, so it might be a little while without help. I think our current thoughts and such are collected at that link, and if you're interested in taking a crack at fleshing out what we have we can expose what we have as an experimental thing for now. Let me know what you think.

[giorgio-v]

I’ll try to recap here the discussion:

• https://github.com/sfx/schema-contrib has implemented new data types generators, which is very useful but it doesn’t integrate them with core.schema.
• https://github.com/cddr/integrity doesn’t have any generator stuff, so I don’t think is relevant in this context
• https://github.com/MichaelBlume/schema-gen seems the most promising and it seems Zeeshan Lakhani is working on the same direction, but I don’t see any link to his projects. Also I don’t know if this is your direction as well.

Is my summary correct enough?

[davegolland]

There is also a small gist where I tried to make some of the schemas most frequently used inside Prismatic (namely vectors and maps with required and optional keys) generative:

https://gist.github.com/davegolland/3bc4277fe109e7b11770

[w01fe]

That seems right . The main thing that's not reflected is this:

Ability to inject some constraints/hints into the generation process. I think a similar solution to the coercers will work here, but haven't thought much about what the API would look like yet.

Like I said there, I'm not sure how it will work, but I think it'll be an essential feature for makign this really useful.

On Sat, Jun 7, 2014 at 9:32 AM, Dave Golland notifications@github.com wrote:

There is also a small gist where I tried to make some of the schemas most frequently used inside Prismatic (namely vectors and maps with required and optional keys) generative:

https://gist.github.com/davegolland/3bc4277fe109e7b11770

— Reply to this email directly or view it on GitHub https://github.com/Prismatic/schema/issues/103#issuecomment-45414550.

[zeeshanlakhani]

Sorry for not responding to this thread sooner, but I finally made a few updates with a lot of @davegolland's code :) and multimethods => https://github.com/zeeshanlakhani/schema-gen.

In terms of the injection of constraints/hints into the generation process, we are doing something like this for a project at the company I work for, but it's--currently--pretty-darn specific to our data and testing needs. I hope to have more time to explore such a solution, or, at least, help @w01fe and @davegolland work on it :).

Nonetheless, I'm mostly putting the schema-gen stuff into a specific repo for anyone who wants to collaborate or whatnot.

[w01fe]

Awesome, thanks for the update! I've been thinking about this a bit and playing around with ideas, but haven't found the time to come up with anything concrete yet. Would be very interested to hear more about the approach you're using for constraint injection, since any real-world experience will be way more informed than my crazy untested ideas :). If you get a chance to work more I'd appreciate bumps on this thread, since I really do hope to get to this soon and would love to collaborate.

On Wed, Jul 9, 2014 at 3:23 AM, Zeeshan Lakhani notifications@github.com wrote:

Sorry for not responding to this thread sooner, but I finally made a few updates with a lot of @davegolland https://github.com/davegolland's code :) and multimethods => https://github.com/zeeshanlakhani/schema-gen.

In terms of the injection of constraints/hints into the generation process, we are doing something like this for a project at the company I work for, but it's--currently--pretty-darn specific to our data and testing needs. I hope to have more time to explore such a solution, or, at least, help @w01fe https://github.com/w01fe and @davegolland https://github.com/davegolland work on it :).

Nonetheless, I'm mostly putting the schema-gen stuff into a specific repo for anyone who wants to collaborate or whatnot.

— Reply to this email directly or view it on GitHub https://github.com/Prismatic/schema/issues/103#issuecomment-48453189.

[giorgio-v]

I’ve made some progress with predicates. I hope to post a gist in the next few days in order to discuss it or scratch it, if you will.

HTH

[w01fe]

Sweet, look forward to checking it out!

On Mon, Jul 14, 2014 at 11:54 PM, giorgio-v notifications@github.com wrote:

I’ve made some progress with predicates. I hope to post a gist in the next few days in order to discuss it or scratch it, if you will.

HTH

— Reply to this email directly or view it on GitHub https://github.com/Prismatic/schema/issues/103#issuecomment-48996659.

[zeeshanlakhani]

@giorgio-v looking forward to it as well.

[giorgio-v]

Here’s the gist:

https://gist.github.com/giorgio-v/cd4025963581ec3e0a12

Basically I’ve started from Dave’s one and extended it to cover as many case as possible. Know issues:

• Regexps and schema.core.Both are not as robust as I wish (translation: they’re a hack). I’d like to discuss with you how to improve these.
• I haven’t tackled yet recursive schemas.

HTH

[giorgio-v]

Any news on this?

[w01fe]

Not yet, sorry. I did some initial investigation of what it would look like to break up the schema protocol further, which would enable generation without so much custom code (and make constraint injection easier) but haven't found anything satisfactory yet.

[giorgio-v]

I see. Let me know if I can help on this in some way. Thanks a lot.

[w01fe]

1.0.0 includes experimental generation support, would love your feedback on that! Please open a new issue if it doesn't meet your needs.