ponylang / pony-tutorial

:horse: Tutorial for the Pony programming language
http://tutorial.ponylang.io
BSD 2-Clause "Simplified" License
310 stars 111 forks source link

[4/?] Add ability to run code examples in the playground: Get pony snippets tested with ponyc at CI time #549

Open shaedrich opened 3 months ago

shaedrich commented 3 months ago

Part of #340

Chapter Snippet stdOut stdErr error line exitCode
Getting started ▸ Hello world hello-world-main.pony Hello, world! n/a n/a 0
Types ▸ Classes classes-swap-values-sugar.pony a = 1, b = 2
a = 2, b = 1
n/a n/a 0
Types ▸ Classes classes-swap-values.pony a = 1, b = 2
a = 2, b = 1
n/a n/a 0
Types ▸ Classes classes-wombat.pony Your default wombat's name is "Fantastibat"
Your hungry wombat's name is "Nomsbat"
n/a n/a 0
Types ▸ Primitives primitives-doors.pony Is door open? false
2 + 3 = 5
n/a n/a 0
Types ▸ Actors actors-sequential.pony This is printed first
This is printed last
n/a n/a 0
Types ▸ Traits and interfaces traits-and-interfaces-multiple-traits.pony Their name is "Bob" and they are not bald n/a n/a 0
Types ▸ Traits and interfaces traits-and-interfaces-nested-traits.pony Their name is "Bob" and they are not bald n/a n/a 0
Types ▸ Traits and interfaces traits-and-interfaces-nominal-and-structural-subtyping.pony Their name is "Bob"
Their name is "Larry"
n/a n/a 0
Types ▸ Traits and interfaces traits-and-interfaces-nominal-subtyping-in-pony.pony Their name is "Larry" n/a n/a 0
Types ▸ Traits traits-and-interfaces-open-world-typing.pony size: 2, space: 8
size: 2, space: 8
size: 3, space: 8
size: 3, space: 4
n/a n/a 0
Types ▸ Traits and interfaces traits-and-interfaces-private-methods.pony n/a interfaces can't have private methods, only traits can 9:11 1?
Types ▸ Traits and interfaces traits-and-interfaces-trait.pony Their name is "Bob" n/a n/a 0
Types ▸ Type aliases type-aliases-enumerations-apply.pony red: #FF0000FF
green: #FF00FF
blue: #FFFF
n/a n/a 0
Types ▸ Type aliases type-aliases-complex-types-trait.pony Bob, aged 42, feels "Great!" n/a n/a 0
shaedrich commented 3 months ago

That's the current output:

Errors: grafik

Successful: grafik

jemc commented 2 months ago

As discussed before and in other PRs, we're mainly concerned with the checking whether something compiles or not (and a regular expression match for the error message in the expected-not-to-compile case).

So that approach drastically reduces how much of this expectation information we need to track - we essentially don't need to track anything at all for examples which are expected to compile, and for those which are expected to have a compilation error, we only need a string / regular expression to match against the error output.

As for the question of where these expectations should be tracked, I think if they were in Pony docstring of the source files themselves, that would risk confusion for the reader of these snippets in the playground (they might likely ask themselves if these expectations were actually of a part of Pony the language). So that leaves basically two options:

  1. A central file somewhere with a list of expected-not-to-compile sample names, each mapped to a string with a pattern to check for in the compilation error.
  2. Decentralized files - each file having the same name as the sample it maps to, but with some extra suffix (such as .error.txt) and the content of the file is the expected error pattern to match.

We're not strongly opinionated about these two approaches, but I have a slight preference for the latter.