search

fediverse-devnet / feditest

A testing framework for distributed, heterogeneous systems communicating with complex protocols, such as the Fediverse
https://feditest.org/
MIT License
27 stars 4 forks source link

Only ask for valid account URI once #200

Open mexon opened 3 weeks ago

mexon commented 3 weeks ago

Currently the test framework asks for a valid account URI for every test. This can easily be worked around by editing a JSON file, but it's a slightly unfriendly welcome to the system. It doesn't ask for a server address every time, so presumably it could treat account URIs the same way.

Extract of session:

(.venv) admin@friends:~/feditest-tests-fediverse$ feditest run --testplan examples/testplans/webfinger-server-imp-vs-saas-any.json --html myapp-webfinger-results.html
TESTER ACTION REQUIRED: Enter the hostname for "server": friends.holograms.test
TESTER ACTION REQUIRED: Enter the name of the app you just provisioned at hostname friends.holograms.test: Friendica
TESTER ACTION REQUIRED: Please enter the URI of an existing or new account at Node "server" (e.g. "acct:testuser@example.local" ): acct:jem@friends.holograms.test
2024-06-19T16:40:54Z [ERROR] feditest: Ended test webfinger.server.4_1__2_parameter_ordering_not_significant::parameter_ordering in Any SaaS application with Exception:
Traceback (most recent call last):
  File "/home/admin/.venv/lib/python3.11/site-packages/feditest/testrun.py", line 162, in run
    self.test_from_test_function.test_function(**args)
  File "/home/admin/feditest-tests-fediverse/tests/webfinger/server/4_1__2_parameter_ordering_not_significant.py", line 30, in parameter_ordering
    assert_that(
  File "/home/admin/.venv/lib/python3.11/site-packages/feditest/__init__.py", line 273, in assert_that
    _assert_match(actual=actual_or_assertion, matcher=matcher, reason=reason, spec_level=spec_level, interop_level=interop_level)
  File "/home/admin/.venv/lib/python3.11/site-packages/feditest/__init__.py", line 241, in _assert_match
    raise AssertionFailure(spec_level, interop_level, description)
feditest.AssertionFailure: WebFinger errors (3)
Accessed URI: "https://friends.holograms.test/.well-known/webfinger?resource=acct%3Ajem%40friends.holograms.test&rel=http%3A//webfinger.net/rel/profile-page&rel=something-else&rel=self".
0: Link rel value not absolute URI nor registered relation type: "salmon"
1: Link rel value not absolute URI nor registered relation type: "magic-public-key"
2: Link href not a URI: "data:application/magic-public-key,RSA.6UazJKhugKx0DKlZSYCnC60C0L2uNHJNt7zdtsZ6pT8usPsfiI9WP7RUBFF535zcdWkSMzpKYu8pts2pTDK2Pw.AQAB"
Expected: None
     but: was <WebFinger errors (3 sub-exceptions)>

TESTER ACTION REQUIRED: Please enter the URI of an existing or new account at Node "server" (e.g. "acct:testuser@example.local" ): 
...
TESTER ACTION REQUIRED: Please enter the URI of an existing or new account at Node "server" (e.g. "acct:testuser@example.local" ): acct:jem@friends.holograms.test
TESTER ACTION REQUIRED: Please enter the URI of an existing or new account at Node "server" (e.g. "acct:testuser@example.local" ): acct:jem@friends.holograms.test
jernst commented 5 days ago

I agree, but not now. We need to make more complex tests work first to make sure the framework as we have it can do it -- as we think it will, but proof is in the pudding! -- and then we can optimize specific work flows.