Minimal Integration Testing

[bollwyvl]

As the current state of #18 unit testing is rather minimal, we should likely consider investing more effort in establishing a reasonable, locally-replicable integration testing approach.

Options

• galata
  • pro:
  • official jupyterlab subproject
  • js/ts-extensible
  • builds on jest ecosystem
  • con:
  • about two weeks old
  • requires ts/js knowledge
  • no particularly viable way to write in lab (jupyterlab-lsp makes this.... better, but js/ts have lots of sharp edges)
• robotframework-jupyterlibrary
  • pro:
  • maintained over about two years
  • builds on robotframework ecosystem
  • python and robot extensible
  • plain-english test cases and user keywords
  • can be authored in-lab with robotkernel or xeus-robot
  • con:
  • single maintainer (me)
  • robot syntax is a little funny the first couple times

[bollwyvl]

Practically, if this task came to me to develop and maintain, I would, somewhat unsurprisingly, choose rfjl, and can get reports into the CI pipeline (#30) quite rapidly. I typically maintain at least the test environment in heavily-locked-down conda environments, since I don't like surprises, but this has no bearing on what the system-under-test runs in.

[bollwyvl]

An additional wrinkle: we don't have a real backend we can test. It may be worth considering an easily self-hostable, but real backend (a la gitea on sqlite) rather than mocking github/gitlab. gitea is also available from conda-forge, but somewhat old at this point.

[fcollonval]

galata is probably gonna be the integration tool in the long run. But I have no experience on neither galata or robotframework.

Regarding galata, I use playwright quite often lately and the most efficient way I found to create test was to use the code generation feature. So basically one will not start that from JupyterLab anyway...

Regarding third-party service mock, I experimented mock-server lately. It can create expectations from open-api specification or by recording usage.