Test running the cli and import tests via hooks

Example run:

st run --experimental=openapi-3.1 --checks al
l --workers 2 --hypothesis-suppress-health-check=too_slow https://edrisobaric.k8s.met.no/api 
================================================== Schemathesis test session starts =================================================
Schema location: https://edrisobaric.k8s.met.no/api
Base URL: https://edrisobaric.k8s.met.no/
Specification version: Open API 3.1.0
Random seed: 40799976358416033973297932885321346683
Workers: 2
Collected API operations: 5
Collected API links: 0
API probing: SUCCESS
Schema analysis: SKIP

Landingpage %s tested OK https://edrisobaric.k8s.met.no/
E....

=============================================================== ERRORS ==============================================================
__________________________________________________________ GET /conformance _________________________________________________________
Runtime Error

OpenAPI version unknown and version in conformance http://www.opengis.net/spec/ogcapi-edr-1/1.2/req/oas31 doesn't match. See <https://docs.ogc.org/is/19-086r6/19-086r6.html#_cc7dd5e3-1d54-41ff-b5ba-c5fcb99fa663> for more info.

Add this option to your command line parameters to see full tracebacks: --show-trace

Need more help?
    Join our Discord server: https://discord.gg/R9ASRAmHnA
============================================================== SUMMARY ==============================================================

Performed checks:
    not_a_server_error                              106 / 106 passed          PASSED 
    status_code_conformance                         106 / 106 passed          PASSED 
    content_type_conformance                        106 / 106 passed          PASSED 
    response_headers_conformance                    106 / 106 passed          PASSED 
    response_schema_conformance                     106 / 106 passed          PASSED 
    negative_data_rejection                         106 / 106 passed          PASSED 
    ignored_auth                                    106 / 106 passed          PASSED 

Experimental Features:
  - OpenAPI 3.1: Support for response validation
    Feedback: https://github.com/schemathesis/schemathesis/discussions/1822

Your feedback is crucial for experimental features. Please visit the provided URL(s) to share your thoughts.

Tip: Use the `--report` CLI option to visualize test results via Schemathesis.io.
We run additional conformance checks on reports from public repos.

==================================================== 4 passed, 1 errored in 7.51s ===================================================

A similar error from main would look like this:

======================================================== test session starts ========================================================
platform linux -- Python 3.12.3, pytest-8.3.3, pluggy-1.5.0
rootdir: /home/larsfp/src/sedr
configfile: pytest.ini
plugins: subtests-0.7.0, hypothesis-6.112.1, anyio-4.6.0, schemathesis-3.36.1
collected 9 items                                                                                                                   

sedr/schemat.py ......F..                                                                                                     [100%]

============================================================= FAILURES ==============================================================
______________________________________________ test_edr_conformance[GET /conformance] _______________________________________________

    @schema.include(path_regex="^" + util.args.base_path + "conformance").parametrize()
>   @settings(max_examples=util.args.iterations, deadline=None)

sedr/schemat.py:94: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

case = Case()

    @schema.include(path_regex="^" + util.args.base_path + "conformance").parametrize()
    @settings(max_examples=util.args.iterations, deadline=None)
    def test_edr_conformance(case):
        """Test /conformance endpoint."""
        response = case.call()
        conformance_json = json.loads(response.text)

        if "conformsTo" not in conformance_json:
            spec_ref = "https://docs.ogc.org/is/19-072/19-072.html#_4129e3d3-9428-4e91-9bfc-645405ed2369"
            raise AssertionError(
                f"Conformance page /conformance does not contain a conformsTo attribute. See {spec_ref} for more info."
            )

        resolves, resolves_message = util.test_conformance_links(
            jsondata=conformance_json["conformsTo"]
        )
        if not resolves:
            raise AssertionError(resolves_message)

        requirementA2_2_A5, requirementA2_2_A5_message = edreq.requirementA2_2_A5(
            jsondata=conformance_json["conformsTo"]
        )
        if not requirementA2_2_A5:
            raise AssertionError(requirementA2_2_A5_message)

        requirementA2_2_A7, requirementA2_2_A7_message = edreq.requirementA2_2_A7(
            response.raw.version
        )
        if not requirementA2_2_A7:
            raise AssertionError(requirementA2_2_A7_message)

        requirementA11_1, requirementA11_1_message = edreq.requirementA11_1(
            jsondata=conformance_json["conformsTo"]
        )
        if not requirementA11_1:
>           raise AssertionError(requirementA11_1_message)
E           AssertionError: OpenAPI version 3.1 and version in conformance http://www.opengis.net/spec/ogcapi-edr-1/1.2/req/oas31 doesn't match. See <https://docs.ogc.org/is/19-086r6/19-086r6.html#_cc7dd5e3-1d54-41ff-b5ba-c5fcb99fa663> for more info.

sedr/schemat.py:128: AssertionError
============================================================== PASSES ===============================================================
________________________________________________________ test_openapi[GET /] ________________________________________________________
__________________________________________________ test_openapi[GET /conformance] ___________________________________________________
__________________________________________________ test_openapi[GET /collections] ___________________________________________________
__________________________________________ test_openapi[GET /collections/{collection_id}] ___________________________________________
_________________________________________ test_openapi[GET /collections/isobaric/position] __________________________________________
____________________________________________________ test_edr_landingpage[GET /] ____________________________________________________
______________________________________________ test_edr_collections[GET /collections] _______________________________________________
___________________________________ test_edr_position_extent[GET /collections/isobaric/position] ____________________________________
====================================================== short test summary info ======================================================
PASSED sedr/schemat.py::test_openapi[GET /]
PASSED sedr/schemat.py::test_openapi[GET /conformance]
PASSED sedr/schemat.py::test_openapi[GET /collections]
PASSED sedr/schemat.py::test_openapi[GET /collections/{collection_id}]
PASSED sedr/schemat.py::test_openapi[GET /collections/isobaric/position]
PASSED sedr/schemat.py::test_edr_landingpage[GET /]
PASSED sedr/schemat.py::test_edr_collections[GET /collections]
PASSED sedr/schemat.py::test_edr_position_extent[GET /collections/isobaric/position]
FAILED sedr/schemat.py::test_edr_conformance[GET /conformance] - AssertionError: OpenAPI version 3.1 and version in conformance http://www.opengis.net/spec/ogcapi-edr-1/1.2/req/oas31 doesn't ma...
==================================================== 1 failed, 8 passed in 7.15s ====================================================

metno / sedr

Test running the cli and import tests via hooks #8