Jc2k / pytest-markdown

Run tests in your markdown
11 stars 2 forks source link

The plugin breaks numerous test suites by enabling itself on them #9

Open mgorny opened 2 years ago

mgorny commented 2 years ago

The pytest-markdown plugin seems to enable itself automatically when installed. This means that whenever it is installed, all test suites unintentionally get their code snippets from *.md run.

In the best case, this means a lot of unintended test failures. For example, editables are broken because the snippets include pseudocode:

ERROR README.md - NameError: name 'BuildAWheel' is not defined
ERROR docs/source/usage.md - NameError: name 'EditableProject' is not defined

rfc3339-validator is also broken. As is brotlicffi. None of these packages intended code snippets from their documentation to be run literally.

In the worst case, this could cause actual damage. Just imagine someone put a potentially destructive code snippet as an example of something. That person didn't expect the snippet to be run via pytest. Now, some user has pytest-markdown installe and boom, destructive action is run without any warning.

Please fix the plugin to process documentation only when explicitly requested by the caller (e.g. using a command-line switch).

Jc2k commented 2 years ago

Hi! Sorry you are having a hard time using pytest-markdown. Can you please confirm how you have it installed? Are you using a global install of pytest (rather than a venv) by any chance?

mgorny commented 2 years ago

We have it installed because ansible-compat needs it to run tests. Yes, it's a global install.

Jc2k commented 2 years ago

I didn't realise anything used it besides my personal stuff, people will run anything they find online won't they! I use venv's exclusively which is why this has not come up before. I'd generally push people to do that for a multitude of reasons, but you are right if people go and install this globally they are in for a very sad time.

I'd take a patch to add a command line switch , but it might be a few days before I can do it myself.

mgorny commented 2 years ago

Thanks for the understanding. There's no hurry, I've just forced -p no:markdown in Gentoo for now.