dandi / dandi-archive

DANDI API server and Web app
https://dandiarchive.org
13 stars 10 forks source link

Make tox and straight `python -m pytest` "turnkey" #299

Closed yarikoptic closed 3 years ago

yarikoptic commented 3 years ago

ATM local testing seems to require first to do some configuration etc to get a contributor going:

tox -e test ```shell $> tox -e test GLOB sdist-make: /home/yoh/proj/dandi/dandi-api/setup.py test inst-nodeps: /home/yoh/proj/dandi/dandi-api/.tox/.tmp/package/1/dandiapi-0.0.1+119.g64264c7.zip test installed: amqp==5.0.6,appdirs==1.4.4,asgiref==3.3.4,attrs==20.3.0,backcall==0.2.0,billiard==3.6.4.0,blessings==1.7,boto3==1.17.66,botocore==1.20.66,Brotli==1.0.9,celery==5.0.5,certifi==2020.12.5,cffi==1.14.5,chardet==4.0.0,ci-info==0.2.0,click==7.1.2,click-didyoumean==0.0.3,click-plugins==1.1.1,click-repl==0.1.6,colorama==0.4.4,commonmark==0.9.1,configparser==5.0.2,coreapi==2.3.3,coreschema==0.0.4,coverage==5.5,cryptography==3.4.7,dandi==0.16.0,dandiapi @ file:///home/yoh/proj/dandi/dandi-api/.tox/.tmp/package/1/dandiapi-0.0.1%2B119.g64264c7.zip,decorator==5.0.7,defusedxml==0.7.1,diskcache==5.2.1,distlib==0.3.1,dj-database-url==0.5.0,dj-email-url==1.0.2,Django==3.2.1,django-admin-display==1.3.0,django-allauth==0.44.0,django-composed-configuration==0.16.0,django-configurations==2.2,django-cors-headers==3.7.0,django-debug-toolbar==3.2.1,django-extensions==3.1.3,django-filter==2.4.0,django-girder-style==0.5.0,django-girder-utils==0.8.0,django-guardian==2.3.0,django-minio-storage==0.3.10,django-oauth-toolkit==1.5.0,django-s3-file-field==0.1.1,django-storages==1.11.1,djangorestframework==3.12.4,djangorestframework-yaml==2.0.0,dnspython==2.1.0,drf-extensions==0.7.0,drf-yasg==1.20.0,email-validator==1.1.2,etelemetry==0.2.2,factory-boy==3.2.0,Faker==8.1.2,fasteners==0.16,filelock==3.0.12,fscacher==0.1.4,girder-client==3.1.4,gunicorn==20.1.0,h11==0.12.0,h5py==3.2.1,hdmf==2.5.1,httpcore==0.13.2,httpx==0.18.1,humanize==3.5.0,idna==2.10,importlib-metadata==4.0.1,inflection==0.5.1,iniconfig==1.1.1,ipython==7.23.1,ipython-genutils==0.2.0,itypes==1.2.0,jedi==0.18.0,jeepney==0.6.0,Jinja2==2.11.3,jmespath==0.10.0,joblib==1.0.1,jsonschema==3.2.0,jwcrypto==0.8,keyring==23.0.1,keyrings.alt==4.0.2,kombu==5.0.2,MarkupSafe==1.1.1,matplotlib-inline==0.1.2,minio==6.0.2,numpy==1.20.2,oauthlib==3.1.0,packaging==20.9,pandas==1.2.4,parso==0.8.2,pexpect==4.8.0,pickleshare==0.7.5,pluggy==0.13.1,prompt-toolkit==3.0.18,psycopg2==2.8.6,ptyprocess==0.7.0,py==1.10.0,pycparser==2.20,pycryptodomex==3.10.1,pydantic==1.8.1,Pygments==2.9.0,PyJWT==2.1.0,pynwb==1.4.0,pyout==0.7.0,pyparsing==2.4.7,pyrsistent==0.17.3,pytest==6.2.4,pytest-cov==2.11.1,pytest-django==4.2.0,pytest-factoryboy==2.1.0,pytest-mock==3.6.0,python-dateutil==2.8.1,python3-openid==3.2.0,pytz==2021.1,PyYAML==5.4.1,requests==2.25.1,requests-oauthlib==1.3.0,requests-toolbelt==0.9.1,rfc3986==1.4.0,rich==10.1.0,ruamel.yaml==0.17.4,ruamel.yaml.clib==0.2.2,s3transfer==0.4.2,scipy==1.6.3,SecretStorage==3.3.1,semantic-version==2.8.5,sentry-sdk==1.0.0,six==1.16.0,sniffio==1.2.0,sqlparse==0.4.1,tenacity==7.0.0,text-unidecode==1.3,toml==0.10.2,tox==3.23.0,tqdm==4.60.0,traitlets==5.0.5,typing-extensions==3.10.0.0,uritemplate==3.0.1,urllib3==1.26.4,versioneer==0.19,vine==5.0.0,virtualenv==20.4.5,wcwidth==0.2.5,whitenoise==5.2.0,zipp==3.4.1 test run-test-pre: PYTHONHASHSEED='3797650013' test run-test: commands[0] | pytest Traceback (most recent call last): File "/home/yoh/proj/dandi/dandi-api/.tox/test/bin/pytest", line 8, in sys.exit(console_main()) File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/_pytest/config/__init__.py", line 185, in console_main code = main() File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/_pytest/config/__init__.py", line 143, in main config = _prepareconfig(args, plugins) File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/_pytest/config/__init__.py", line 318, in _prepareconfig config = pluginmanager.hook.pytest_cmdline_parse( File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/pluggy/hooks.py", line 286, in __call__ return self._hookexec(self, self.get_hookimpls(), kwargs) File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/pluggy/manager.py", line 93, in _hookexec return self._inner_hookexec(hook, methods, kwargs) File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/pluggy/manager.py", line 84, in self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall( File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/pluggy/callers.py", line 203, in _multicall gen.send(outcome) File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/_pytest/helpconfig.py", line 100, in pytest_cmdline_parse config: Config = outcome.get_result() File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/pluggy/callers.py", line 80, in get_result raise ex[1].with_traceback(ex[2]) File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/pluggy/callers.py", line 187, in _multicall res = hook_impl.function(*args) File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1003, in pytest_cmdline_parse self.parse(args) File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1283, in parse self._preparse(args, addopts=addopts) File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1191, in _preparse self.hook.pytest_load_initial_conftests( File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/pluggy/hooks.py", line 286, in __call__ return self._hookexec(self, self.get_hookimpls(), kwargs) File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/pluggy/manager.py", line 93, in _hookexec return self._inner_hookexec(hook, methods, kwargs) File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/pluggy/manager.py", line 84, in self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall( File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/pluggy/callers.py", line 208, in _multicall return outcome.get_result() File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/pluggy/callers.py", line 80, in get_result raise ex[1].with_traceback(ex[2]) File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/pluggy/callers.py", line 187, in _multicall res = hook_impl.function(*args) File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/pytest_django/plugin.py", line 322, in pytest_load_initial_conftests dj_settings.DATABASES File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/django/conf/__init__.py", line 82, in __getattr__ self._setup(name) File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/django/conf/__init__.py", line 69, in _setup self._wrapped = Settings(settings_module) File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/django/conf/__init__.py", line 170, in __init__ mod = importlib.import_module(self.SETTINGS_MODULE) File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1030, in _gcd_import File "", line 1007, in _find_and_load File "", line 986, in _find_and_load_unlocked File "", line 664, in _load_unlocked File "", line 627, in _load_backward_compatible File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/configurations/importer.py", line 182, in load_module reraise(err, "Couldn't setup configuration '{0}'".format(cls_path)) File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/configurations/importer.py", line 164, in load_module cls.setup() File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/configurations/base.py", line 127, in setup setup_value(cls, name, value) File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/configurations/values.py", line 15, in setup_value actual_value = value.setup(name) File "/home/yoh/proj/dandi/dandi-api/.tox/test/lib/python3.9/site-packages/configurations/values.py", line 105, in setup raise ValueError('Value {0!r} is required to be set as the ' ValueError: Couldn't setup configuration 'dandiapi.settings.TestingConfiguration': Value 'DANDI_SCHEMA_VERSION' is required to be set as the environment variable 'DJANGO_DANDI_SCHEMA_VERSION' ERROR: InvocationError for command /home/yoh/proj/dandi/dandi-api/.tox/test/bin/pytest (exited with code 1) _________________________________________________________________________________________________________________________________________________________________________________ summary _________________________________________________________________________________________________________________________________________________________________________________ ERROR: test: commands failed tox -e test 6.79s user 0.89s system 97% cpu 7.843 total ```

For dandi-cli @jwodder established a fixture which takes care about starting up a dockerized instance. I do not see why it could or should not be done for dandi-api as well.

dchiquito commented 3 years ago

Have you tried running docker-compose run --rm django tox as mentioned in the documentation? That will run tox inside the django container, which has all the right environment variables already specified, and as a bonus it will spin up the necessary backing services as well. That sounds like the "turnkey" solution you are asking for here.

yarikoptic commented 3 years ago

unrelated interesting in the scope of #222 observation is that versioneer seems to fail to provide correct version in that case as well:

$> docker-compose run --rm django tox
...
test inst: /opt/django-project/.tox/.tmp/package/1/dandiapi-0+unknown.zip

overall - the docker-compose invocation seemed to run but fails quite badly on v0.1.3-15-g0373d72 state of things:

tried on smaug (a different server) - the same there -- 9 failed, 215 passed, 133 warnings in 42.66s

But overall -- it is great that there is a way to run tests, but if we are to standardize on tox to be our testing execution helper, the original intent and purpose was to be able to run tests uniformly across components, e.g. via tox -e test. As number of components grows it becomes tedious to remember and/or RTFM every time I need to run tests in a component X. dandi-cli implements such approach via fixtures, and that is what I was advocating here.

waxlamp commented 3 years ago

Can you point to an example of what you mean in dandi-cli? "Fixtures" and "tox environments" are not objects of the same type.

dchiquito commented 3 years ago

I fixed the error you were running into in #442.

yarikoptic commented 3 years ago

Can you point to an example of what you mean in dandi-cli? "Fixtures" and "tox environments" are not objects of the same type.

I would be happy to, and even wondering why I have not done that originally: https://github.com/dandi/dandi-cli/blob/master/dandi/tests/fixtures.py#L158

waxlamp commented 3 years ago

Thanks @yarikoptic. I'm going to close this issue since the immediate problems are solved, and if we want to take on the task of doing a more comprehensive cross-component testing strategy, we should open a new issue to focus on that.

Right now, I don't see a need for it in dandi-api: the testing strategy we have now works as intended. The text fixture you showed is interesting and creative, and maybe there's a need to promote that strategy among more repos but let's explore that in a separate issue.