Best Approach for Swizzling Examples against Tests

If you look at the python exercism - there is a directory called 'test' - and it has a script 'check-exercises.py' that from what I understand, will go through all the exercises and swizzle the example solution so that it can be run against the proposed tests.

I"m thinking we might want to do this a different way. I propose we have a single Exercism master project with tests and examples as packages - one for each example.

In exercism development (not as an end user), would be writing an example and building up tests for it (more tdd like) would be done in that way, but then then we have an ExercismGenerator that would iterate through all the examples (either in a tree, or by pragma) and would generate the Tonel examples in the exercises directory? If done this way - we might not even need an example.st file in each exercise?

What's not clear to me - is whether there are build hooks in exercism to do this kind of stuff - or if we do it ourselves and then just check in the output.

@macta I'm having a bit of trouble picturing what you mean. Let me know if I've misunderstood.

I"m thinking we might want to do this a different way. I propose we have a single Exercism master project with tests and examples as packages - one for each example.

It looks like this style is the convention for other language tracks. One big repo with all the exercises in sub-directories under the exercises directory.

Going by this guide here the example.st file (a reference solution) is just for verifying that the tests are correct, and nothing else. So if we can verify the correctness of tests through other means then we wouldn't need an example.st. However there might be some value in keeping it as it keeps convention with other language tracks. Maintainers from other language tracks might be surprised when they come to the Pharo repo and find no reference solutions where they expected them.

What's not clear to me - is whether there are build hooks in exercism to do this kind of stuff - or if we do it ourselves and then just check in the output.

I haven't gone looking yet, but there may be a more general discussion issue/forum somewhere in Exercism where we can ask this stuff. Plenty of maintainers here who have had to go through the problems we are facing now. It would be a great idea to leverage that.

@samWson as Pharo works with an image - its an illusion that there are directories with examples and tests in them, particularly as we don't have a huge amount of control on how they are written out and stored, and more importantly how we would run tests (we are hacking into this mechanism to give students an illusion of downloading files and working on them).

So my suggestion is (and refined based on my spike) - we have a normal Pharo Project & Package called "Exercism" which has exercise tags (they look like sub-packages, but Pharo seems to call them tags now) - 1 for each exercise, and they are each named how we want the "directory" to appear under "examples". This project will have a single stub Welcome (might change ideas on this - but its something that can appear when you launch Pharo after loading our Exercism baseline). In this Project there will also be a second package for the tools for submitting and fetching exercises (I've called it Xercism, in an attempt to move it visually away from Exercism so students don't get confused about where the exercises they are loading will appear). They don't really need to now about the tools, as they should appear in a menu item on packages.

I propose we then have a second development Pharo Project called XercismExercises, along with a package of the same name (? I'm thinking aloud here). In my Spike, the HelloWorld example should be moved over to here. Each of the examples sub-project (tag) will have a test class, Test, and a solution class (possibly some more supporting classes to). This mostly matches what you would expect. I also propose each test has a class comment which is the contents of a Readme.md (which we can extract later - see below). We will test drive these out - or convert them from other languages).

I then propose that we have a class ExercismGenerator, which will verify all the tests run and then walk all of the XercismExercises that pass (? could we have broken ones checked in under development) and it will write out the Test class and extract a ReadMe.md (need to think about this - as it also appears in a class comment, but I think the exercism website wants a readme.md file for each exercise) and stick them in a directory under exercises. Once complete - this is something we will need to checkin separately to update our track. (maybe we can osprocess a script - but again I'm not totally clear on what the build characteristics of exercism are - it looks like things run on Travis?).

We might also need to look at writing out solutions as well (the exercism.json has a regex to identify them) - so we may need to prefix them when we write them out (maybe this is where the Ex prefix might kick in, so users can have their solution and ours loaded at the same time).

I'm proposing that we don't write out a stub class on export - as I think we want to emphasise the Smalltalk way of loving your debugger - and let it create the starting class for you on the back of a failed test. I notice the Ruby track mentions this - but they have to create that class manually.

We would obviously expect contributors to load this second project and work on 1 or more examples, in a fork and branch and then they would PR them back for inclusion (which should be a simple merge).

A bit of a brain dump - but thats what I saw on the back of my spike.

On 4 July 2018 at 17:04, Tim Mackinnon notifications@github.com wrote:

@samWson https://github.com/samWson as Pharo works with an image - its an illusion that there are directories with examples and tests in them, particularly as we don't have a huge amount of control on how they are written out and stored, and more importantly how we would run tests (we are hacking into this mechanism to give students an illusion of downloading files and working on them).

So my suggestion is (and refined based on my spike) - we have a normal Pharo Project & Package called "Exercism" which has exercise tags (they look like sub-packages, but Pharo seems to call them tags now) - 1 for each exercise, and they are each named how we want the "directory" to appear under "examples". This project will have a single stub Welcome (might change ideas on this - but its something that can appear when you launch Pharo after loading our Exercism baseline). In this Project there will also be a second package for the tools for submitting and fetching exercises (I've called it Xercism, in an attempt to move it visually away from Exercism so students don't get confused about where the exercises they are loading will appear). They don't really need to now about the tools, as they should appear in a menu item on packages.

I propose we then have a second development Pharo Project called XercismExercises, along with a package of the same name (? I'm thinking aloud here). In my Spike, the HelloWorld example should be moved over to here. Each of the examples sub-project (tag) will have a test class, Test, and a solution class (possibly some more supporting classes to).

I'm not quite clear on the intent here, but XercismExercises makes me twitch, could XercismTesting or XercismSolutions be suitable?

cheers -ben

Yeah - wasn’t sure of a name either - the intent is this is the project dev’s would use to create exercises for students. So its the one we load up, do TDD in and get things working and then we would generate the files for the other “Exercism” project that the exercism client then checks out from.

So its an internal name - and totally not tied to anything. I would just thinking we should try and keep it at a distance from Exercism so that we don’t confuse ourselves if we have both sides loaded.

Tim

On 4 Jul 2018, at 14:24, Ben Coman notifications@github.com wrote:

On 4 July 2018 at 17:04, Tim Mackinnon notifications@github.com wrote:

@samWson https://github.com/samWson as Pharo works with an image - its an illusion that there are directories with examples and tests in them, particularly as we don't have a huge amount of control on how they are written out and stored, and more importantly how we would run tests (we are hacking into this mechanism to give students an illusion of downloading files and working on them).

So my suggestion is (and refined based on my spike) - we have a normal Pharo Project & Package called "Exercism" which has exercise tags (they look like sub-packages, but Pharo seems to call them tags now) - 1 for each exercise, and they are each named how we want the "directory" to appear under "examples". This project will have a single stub Welcome (might change ideas on this - but its something that can appear when you launch Pharo after loading our Exercism baseline). In this Project there will also be a second package for the tools for submitting and fetching exercises (I've called it Xercism, in an attempt to move it visually away from Exercism so students don't get confused about where the exercises they are loading will appear). They don't really need to now about the tools, as they should appear in a menu item on packages.

I propose we then have a second development Pharo Project called XercismExercises, along with a package of the same name (? I'm thinking aloud here). In my Spike, the HelloWorld example should be moved over to here. Each of the examples sub-project (tag) will have a test class, Test, and a solution class (possibly some more supporting classes to).

I'm not quite clear on the intent here, but XercismExercises makes me twitch, could XercismTesting or XercismSolutions be suitable?

cheers -ben — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/exercism/exercism-pharo-tooling/issues/4#issuecomment-402478331, or mute the thread https://github.com/notifications/unsubscribe-auth/AFS913l1p_8m9KXBGCjZ9M9AmZeTBjL_ks5uDMIGgaJpZM4U_JSr.

On 4 July 2018 at 16:01, Samuel notifications@github.com wrote:

@macta https://github.com/macta I'm having a bit of trouble picturing what you mean. Let me know if I've misunderstood.

I"m thinking we might want to do this a different way. I propose we have a single Exercism master project with tests and examples as packages - one for each example.

It looks like this style is the convention for other language tracks. One big repo with all the exercises in sub-directories under the exercises directory.

Going by this guide here https://github.com/exercism/docs/blob/master/language-tracks/launch/first-exercise.md#write-the-reference-solution the example.st file (a reference solution) is just for verifying that the tests are correct, and nothing else. So if we can verify the correctness of tests through other means then we wouldn't need an example.st. However there might be some value in keeping it as it keeps convention with other language tracks. Maintainers from other language tracks might be surprised when they come to the Pharo repo and find no reference solutions where they expected them.

Our example.st might just contain a comment explaining how we do it within the Image. Or maybe its a file to be run headless by CI which invokes TestRunner on the development unit tests.

cheers -ben

In this Project there will also be a second package for the tools for submitting and fetching exercises (I've called it Xercism, in an attempt to move it visually away from Exercism so students don't get confused about where the exercises they are loading will appear). They don't really need to now about the tools, as they should appear in a menu item on packages.

The other perspective is that the power of Smalltalk is its integration and easy access to the source code of the tools. Being able to explore how our Exercism system works could provide students a sense of mastery and the possibilities available from having such integration. The package could be called "ExercismTools" and our Welcome page would suggest students ignore it until the end of of exercises.

The best place to recruit contributors is from students who have just been using the tools. Some "students" will be very experienced devs who may want to extend themselves.

I propose we then have a second development Pharo Project called XercismExercises. ... So its an internal name - and totally not tied to anything. I would just thinking we should try and keep it at a distance from Exercism so that we don’t confuse ourselves if we have both sides loaded.

So students should never see this. I don't think it needs to be separated by distance in the list, and believe calling it ExercismDev should be sufficient. And the generator would be in ExercismDevTools.

So the full package structure could be:

Exercism-*
ExercismDev-*
ExercismDevTools
ExercismTools

where students would see only 1 & 4, with the wildcard subtree under 1 & 2 identical, one item per exercise, with 2 holding solutions. "ExercismGenerator" would be held by 3. "Exercism" class would be held by 4.

It sounds like we have a rough plan - we really need to get it up and running and try it out as it’s pretty easy to change, and we may reconsider when we have 30+ exercises

Sent from my iPhone

On 5 Jul 2018, at 01:18, Ben Coman notifications@github.com wrote:

In this Project there will also be a second package for the tools for submitting and fetching exercises (I've called it Xercism, in an attempt to move it visually away from Exercism so students don't get confused about where the exercises they are loading will appear). They don't really need to now about the tools, as they should appear in a menu item on packages.

The other perspective is that the power of Smalltalk is its integration and easy access to the source code of the tools. Being able to explore how our Exercism system works could provide students a sense of mastery and the possibilities available from having such integration. The package could be called "ExercismTools" and our Welcome page would suggest students ignore it until the end of of exercises.

The best place to recruit contributors is from students who have just been using the tools. Some "students" will be very experienced devs who may want to extend themselves.

I propose we then have a second development Pharo Project called XercismExercises. ... So its an internal name - and totally not tied to anything. I would just thinking we should try and keep it at a distance from Exercism so that we don’t confuse ourselves if we have both sides loaded.

So students should never see this. I don't think it needs to be separated by distance in the list, and believe calling it ExercismDev should be sufficient. And the generator would be in ExercismDevTools.

So the full package structure could be: 1 Exercism- 2 ExercismDev- 3 ExercismDevTools 4 ExercismTools where students would see only 1 & 4, and the wildcard subtree under 1 & 2 are identical, one item per exercise, with 2 holding solutions. "ExercismGenerator" would be held by 3. "Exercism" class would be held by 4.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

The next best thing to do then is to push a merge request with the proposed repo stubbed out, even if its just directories containing READMEs. Seeing it will make it much easier to understand and then suggest any improvements.

The generator is now written and in the main repo

exercism / exercism-pharo-tooling

Best Approach for Swizzling Examples against Tests #4