numirias / pytest-json-report

🗒️ A pytest plugin to report test results as JSON
MIT License
146 stars 39 forks source link

Report object without .report.json file #34

Closed Formartha closed 5 years ago

Formartha commented 5 years ago

Hello, I'm using pytest.main with providing it the plugin directly. e.g. JSONReport(). however, I would like to avoid saving the .report.json file before I hit the object (or at least, provide it some info not to store it on the disk).

The reason for my request is that I would like to modify the results based on a pattern I need (adding some extra fields and removing some).

How can I do that?

numirias commented 5 years ago

You can use --json-report-file none to prevent saving. WIth invocation from code, it would look like this:

import pytest
from pytest_jsonreport.plugin import JSONReport

plugin = JSONReport()
pytest.main(['--json-report-file', 'none', 'test_foo.py'], plugins=[plugin])

See https://github.com/numirias/pytest-json-report#direct-invocation for how to access and save the report yourself afterwards.

Note that, depending on your use case, it might be a bit more elegant to just use the provided hook to alter the report as shown here: https://github.com/numirias/pytest-json-report#modifying-the-report

Formartha commented 5 years ago

Thank you @numirias . I tried it in the past, but when doing that - it removes the content of the report object itself.

numirias commented 5 years ago

If you want to give a specific example that doesn't work as expected, I'm happy to look into it and try to reproduce.

Formartha commented 5 years ago

sure, an example:

I'm using pytest_sessionfinish hook. In that hook, I'm trying to fetch information from the report object in order to construct data structure which fit my needs.

trying to access config._json_report is not avilable while using "none" value in report file as you mentioned.

try the following in your conftest:

def pytest_sessionfinish(session):
        def _makehash():
            return collections.defaultdict(_makehash)

        build_json_rep = _makehash()

        report = session.config._json_report
        build_json_rep["Results"] = report.report["summary"]
        for key, value in report._json_tests.items():
            build_json_rep["Test_Data"][key] = value["outcome"]

        rep = {}
        rep[str(uuid.uuid4())] = json.loads(json.dumps(build_json_rep))
Formartha commented 5 years ago

hi, the issue is as the following (keep in mind, I'm a novice in whatever related to python).

for some reason, the plugin failed to register itself, due to it - there is an error related to it. the w/a is to delete: if not config.option.json_report: return

however, I do not know what it will impact.

def pytest_configure(config): if not config.option.json_report: return if hasattr(config, 'workerinput'): Plugin = JSONReportWorker else: Plugin = JSONReport plugin = Plugin(config) config._json_report = plugin config.pluginmanager.register(plugin)

Formartha commented 5 years ago

@numirias , did you had the chance to look into it? I tried sending a pull request, but it failed on some tests so I don't want to mess anything up :)

numirias commented 5 years ago

@Formartha Sorry for getting back late on this. And thanks for your contribution.

Unfortunately, I am still having trouble reproducing your report. Your example above actually works fine for me.

The line

if not config.option.json_report:
    return

means "If the user is not using the option --json-report, don't load the plugin." This ensures that the plugin is only loaded if pytest is called with the --json-report parameter. Your PR would reverse that logic and automatically create a report every time.

Are you sure you are calling pytest with --json-report? If so, could you give the exact command you are running pytest with?

numirias commented 5 years ago

I will close this issue for now.

Feel free to reopen it with info about how exactly you're calling pytest and on what file. Also note that the problem might be related to #37 which has since been fixed.