Discuss: What complexity can/should be moved to this repo and moved out of dependants?

plocket commented 3 years ago

[architecture and security discussion]

I think we could make an executable for each script instead and then we can adjust what's in the executables here. It would mean less would need to be updated in the dependent repos.
I think we might also be able to make most of the workflow into an action somehow that really only needs the triggers and the BASE_URL. I am a bit concerned that it would have to be its own repo, though.

Regardless of what we can do, what should we do? The dependents can always break out and make their own changes. How complex do we want that to be? How much of a need do we think there will be for people to make their own versions?

niharikasingh commented 3 years ago

From what I can tell, the most common use case will probably be "people write a bunch of tests in the tests/feature directory, and want them to run (A) locally and (B) on github merge request pushes". I think we should try to make that process as easy as possible, and (A) is much more important than (B). Based on that, I think point (1) from above takes priority over point (2). Point (2) is Github-specific and Github-actions-specific, and people might have different preferences/setups there, while (1) is more generic and could be used more widely.

plocket commented 3 years ago

I've been finding that 2 does become relevant at times. For example, right now I'm looking at changing the script so that if the initial setup fails, the rest of the stuff doesn't run. On GitHub, that means getting people to edit their scripts somehow. It's not crucial, but it is still a bit of an obstacle.

I think we could make an action of our own that we 'publish'. It would probably have to be in yet another repo. The dependent's workflow file would still exist, it would just use that action. If they ever wanted to get more granular, they could copy/paste the code in that action and adjust it. I think actions like that are version controlled, so people could use the version to make sure major changes there don't break their tests.

On the other hand, maybe we'll think we introduced a non-breaking change, but it actually does break older repos and we don't hear about it. It might be hard to keep track of what is or isn't a breaking change. I suppose that's true for every part of this, I'm just worried this will be harder for devs and us to discover or track down.

Is there a way to make a script in node affect what actions do next? Then we could do the executable thing and that would solve it. Bleh.

[Edit: There are also things that executable scripts can't affect, like env. variables [that need to be encrypted user info] that we just recently improved in a non-breaking way.]

plocket commented 3 years ago

Also just realized that a possible problem with using executables - the versions of the different parts of this don't line up. There may be a non-breaking change we want to implement in the test framework while there's a breaking change we want to introduce to the executables. It could force a major version change instead of a minor or patch even though the code of the framework itself was staying the same.

Is that really a problem? Would a breaking change in those node scripts actually really be an appropriate time to change major versions? I can't think of actual permutations right now.