Consolidate our internal example apps (waspc/examples)

[Martinsos]

Current situation

Right now we have example apps in three different locations:

1. waspc/examples/
2. headless-test/waspc/examples/
3. examples/

• waspc/examples/todoApp is the primary one we use when developing waspc, as a playground, and we also always keep it up to date with the changes to the code. SO this is a "working app". We also check in CI that it compiles and builds (we don't test functionality though, no e2e tests).
• headless-test/waspc/examples/todoApp is a (now diverged) fork of waspc/examples/todoApp that has headless(e2e) tests defined on it, which we run in the CI.
• Apps in examples are here for Wasp users to look at them and learn about Wasp, so many of them are on purpose relatively simple to showcase a certain functionality.

This is a lot of example apps and it is quite hard to maintain them all, so we want to consolidate them and make it easier for us.

Lot of this is already covered by https://github.com/wasp-lang/wasp/issues/1976 . Part is also covered by https://github.com/wasp-lang/wasp/issues/2024 and https://github.com/wasp-lang/wasp/issues/2058 .

Requirements

1. Have example apps that allow people to learn about Wasp (external example apps).
2. Have example apps that are for us to test (locally and in CI) and play with during development (internal example apps).
3. We want e2e tests for all example apps. Especially internal ones, but we should probably also have them for external ones.
4. Have minimal possible number of example apps in total, for easier maintenance.

Questions / Ideas

1. Could we have just one set of example apps? So they serve both as internal and external apps? Maybe those two just don't go together: internal need to be messy and overloaded with features, while external ones we want to be simmple?
2. Can we write e2e tests setup just once and use it for all the apps? So we don't have to define that again and again for every app? I think we should be able to at least define it just once per a set of apps (e.g. once for internal example apps, once for external example apps). But do we want it? In real usage, we will usually want people to define e2e tests next to the app, kind of like we did it with open-saas, not have them "above the app".
3. When do we run which e2e tests in the CI? All of them every time? Or internal ones every time, and external ones we run upon cutting new release?
4. What about the test we have for waspc/examples/todoApp, that checks if it compiles and builds. Should we make that test universal and run it for every (internal) example app in the CI?
5. Do we need so many external example apps? We have quite a few at the moment. Maybe we could reduce the amount to only a few? One complex, one simple, and maybe one for some specific feature? We should keep in mind we do also have OpenSaas which serves as a big example app in a way. And other templates.
6. How many internal example apps do we need? Just todoApp, as one app which has everything in it? Or also some additional smaller apps that test specific features?
7. Some options in Wasp are mutually exclusive, for example email and username auth at the moment, then DB system choice, emailSender choice, ... . How do we test all of these, without having multiple apps that are all very similar? Could dope.sh (diffing) approach help with this?
8. Bonus question: What about those example apps we have in our golden e2e tests? That we use to check how output of the Generator changed? They are kind of not really a part of this, but maybe we should give them a quick thought, regarding how they fit into all this.

Possible solutions

It seems quite clear that we should merge waspc/examples/ and headless-test/examples.

What is not so clear is if we should go step further and also try to merge internal with external examples, and also what and when to test in the CI.

Here are some different proposals as to what this could look like:

1. We could have two sets of apps: waspc/examples and examples. In waspc/examples, we would have one "mega" app, TheTestApp, that would have all the stuff in it and would be the main one we use to test stuff. We would also have another, BasicApp, which would have just some very basics in it, making it somewhat easier to try out stuff. Maybe this BasicApp is not even needed. We would have a couple dope.sh powered apps that would test differen variations on top of TheTestApp. TheTestApp would be covered with e2e tests, and we would also have test to check that it compiles and builds. These would run in CI all the time. We would have a single e2e tests setup for all of these apps. In examples, we would have one bigger, more complex app, real world or something like that. We would also have our todo app tutorial app in there. And finally, maybe one or two more apps that showcase specific features -> but we should try to have as little of these as possible, and also consider just merging these specific features into one app if possible. These apps would be covered with e2e tests, and also tests to see if they compile and build. These would run only on release in CI. We would have a single e2e tests setup for all of these apps? Or is that bad example hm.

2. Similar like above (1), but all of them under examples. They could share e2e testing setup (is that good?). There would be one "frankenstein app", "TheTestApp", but we would make it clear this one is mostly for internal usage, and the rest would be external apps. This kind of simplifies the setup + also gives them TheTestApp to look at.

While I am tempted by (2), approach (1) feels like safer bet, it won't limit us too much, and we can see in the future if it makes sense to merge internal and external and go for (2).

[infomiho]

Nice initiative 👍 the rising number of example apps makes cutting a new release more work than needed, so I think this is a good thing to tackle.

I think we should have one dev app that's a "kitchen sink" (name commonly used for these types of apps) which has all the features. Ideally, we cover all of them with headless tests then. Which gives us confidence that everything works and everything is tested all the time. Some of the stuff like logging in with Google might not be possible with headless tests (I'm not sure, tbh) but that's a small subset of features.

About the external apps: the best external app we have now is the Open SaaS one. It's the most useful one. I guess the other apps had a stronger purpose before like to demonstrate you can build more complex apps with Wasp. Now, maybe we no longer need the Trello clone or the Tableau clone because we have Open SaaS.

What's useful for us?

• Being able to work on new stuff.
• Have automated tests.

What's useful for the users:

• Having working code that they can copy paste.
• Seeing which features are available and how they work (live demos).
• Seeing various integrations e.g. different UI libs or 3rd party APIs.

I'd merge the headless tests app and the dev app. I'd probably delete most of the external apps and then show off the dev app as the "kitchen sink" app for the users to see working code. But I'm less certain about the second thing.

[sodic]

Great analysis, I'll quote several points and say what I think and why.

Requirements

I'm on board with all the requirements you listed, except maybe this one:

We want e2e tests for all example apps. Especially internal ones, but we should probably also have them for external ones.

Depending on how many example apps we have and how we use them, having e2e tests for each one of them might be more trouble than it's worth.

I'm more in favor of one big e2e-tested app with all functionalities, and keeping the "public" example apps simple and untested, as if they were written by someone who's not on the Wasp team.

Which brings me to...

Questions/ideas

Could we have just one set of example apps? So they serve both as internal and external apps?

I'd rather keep them separate. The two sets of example apps have different responsibilities (showcasing features vs. testing and working wasp), and I feel merging them would cause problems. It's no accident we naturally evolved into our current state where these two sets of example apps are separate.

This ties into my first response - I prefer running e2e tests exclusively on internal apps.

Can we write e2e tests setup just once and use it for all the apps? So we don't have to define that again and again for every app?

You mean like having a e2e testing framework? I think it's cool, but also think we don't need it (since I am a strong supporter of only running e2e tests on a single app, in which case this becomes a moot point).

What about the test we have for waspc/examples/todoApp, that checks if it compiles and builds. Should we make that test universal and run it for every (internal) example app in the CI?

This we could do (for both public and private apps). It's simple enough.

Do we need so many external example apps? We have quite a few at the moment.

Probably not, we could remove some of them from the repo.

Some options in Wasp are mutually exclusive [...] Could dope.sh help?

Yeah, this is a problem. I would wait and see how dope.sh does with OpenSaas before extending it here. Maybe we should start by testing only a single of the mutually exclusive options? I think that should be enough.

Possible solutions

It seems quite clear that we should merge waspc/examples/ and headless-test/examples.

Yes, we all agree on this one :)

As for the solutions, I'd like to propose a third one (it's a modified version of your first proposed solution). I'll copy your text and modify it (the main differences are in bold):

Option 3

We have two sets of apps: waspc/examples and examples.

In waspc/examples, we would have one "mega" app, TheTestApp, that would have all the stuff in it and would be the only one we use to develop, test and e2e test stuff.

We would also have another, BasicApp, which would have just some very basics in it, making it somewhat easier to try out stuff. Maybe this BasicApp is not even needed.

The CI ensures that TheTestApp compiles, builds, and e2e tests it on every run. This is the only app that's e2e tested.

In examples, we would have one bigger, more complex app, real world or something like that. We would also have our todo app tutorial app in there. And finally, maybe one or two more apps that showcase specific features -> but we should try to have as few of these as possible, and also consider just merging these specific features into one app if possible. These apps wouldn't be covered with e2e tests, but we would tests to see if they compile and build. These would run only on release in CI.

[Martinsos]

@sodic this all makes sense to me, except for not wanting to test the external example apps. Why wouldn't we have e2e tests for them? For every release, we need to make sure that these apps work correctly. Isn't it much easier to have e2e tests that we can run then testing these apps manually? If we don't have too many example apps + we share the setup for e2e tests (which would be doable I think), then maintaing those e2e tests sounds better than manually testing all those apps each time to me.

[Martinsos]

@infomiho that makes sense to me, although I think there is some benefit to having example apps that are a bit smaller also -> a big one is a lot to grasp. Although TodoApp for tutorial is maybe also a good smaller example app? Hm. Maybe different use cases, for example one for live chat, one for ecommerce, one for dashboard, stuff like that. Yeah probably that would be most valuable. OpenSaas is one hand example app, yeah that is true. But it is also quite big which I think makes it harder to take a look and say "aha this is what Wasp works like". Hm. Ok I think we need to figure this out yet.

[sodic]

For every release, we need to make sure that these apps work correctly.

@Martinsos That's true. I just don't know how we'd write the kind of tests that would spare us from having to run it manually.

If we can do it and it doesn't take a lot of time, I'm in.

[Martinsos]

For every release, we need to make sure that these apps work correctly.

@Martinsos That's true. I just don't know how we'd write the kind of tests that would spare us from having to run it manually.

If we can do it and it doesn't take a lot of time, I'm in.

For every release, we need to make sure that these apps work correctly.

@Martinsos That's true. I just don't know how we'd write the kind of tests that would spare us from having to run it manually.

If we can do it and it doesn't take a lot of time, I'm in.

Just Playwright tests, like we have for open-saas and for that headless-test/examples/todoApp, that would run the app, do some clicking around, stuff like that. Hm I guess you are not sure if these can test those apps well enough, taking into account stuff can go wrong from the Wasp side? That is maybe a good question. Ok, but certainly worth trying, we can see how it goes, as you said.