Closed nedbat closed 7 years ago
Original comment by Loic Dachary (Bitbucket: dachary, GitHub: dachary)
Following the instructions and fixing the problems I found on the way, I ran this:
#!bash
Exception Type: ProgrammingError at /api/v1/testcov Exception Value: relation "core_buyer" does not exist LINE 1: INSERT INTO "core_buyer" ("id", "first_name", "last_name") V...
I did not try to fix this last error. My advice is that you provide and test detailed instructions to create a reproducer for this bug. The proposed reproducer is a good step forward but it fails in a number of ways and cannot be used effectively.
Original comment by Reetesh Ranjan (Bitbucket: reeteshranjan, GitHub: reeteshranjan)
Post fixing the above, I collected the coverage data and I get the problematic data. Here is a description:
(coverage) reeteshranjan@test:~/coverage-testapp$ coverage run --source="." ./manage.py runserver
Performing system checks...
System check identified no issues (0 silenced).
June 14, 2016 - 15:41:10
Django version 1.9.7, using settings 'covtest.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
Then I make an API call
reeteshranjan@test:~/coverage-testapp$ curl "http://127.0.0.1:8000/api/v1/testcov"
All well!
Logged by the test server run by coverage.py
[14/Jun/2016 15:41:13] "GET /api/v1/testcov HTTP/1.1" 200 9
Which executes this piece of code in api/views.py
def test_api(request):
location = Location(title = 'My Test Location', latitude = '10.1311141', longitute = '75.0114141')
business = Business.create(location = location, title = 'My Business')
buyer = Buyer.objects.create(first_name = 'Reetesh', last_name = 'Ranjan')
return HttpResponse('All well!')
You see the output 'All well!' on the prompt that is returned by this API. Here is the binding of the URL to this API:
covtest/urls.py
urlpatterns = [
url(r'^admin/', admin.site.urls),
url(r'^api/v1/' , include('api.urls')),
]
api/urls.py
urlpatterns = [
url(r'^testcov$', views.test_api),
]
Now I stop the running server
(coverage) reeteshranjan@test:~/coverage-testapp$ coverage run --source="." ./manage.py runserver Performing system checks...
System check identified no issues (0 silenced).
June 14, 2016 - 15:53:51
Django version 1.9.7, using settings 'covtest.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
[14/Jun/2016 15:53:54] "GET /api/v1/testcov HTTP/1.1" 200 9
[14/Jun/2016 15:53:56] "GET /api/v1/testcov HTTP/1.1" 200 9
[14/Jun/2016 15:53:56] "GET /api/v1/testcov HTTP/1.1" 200 9
[14/Jun/2016 15:53:57] "GET /api/v1/testcov HTTP/1.1" 200 9
[14/Jun/2016 15:53:58] "GET /api/v1/testcov HTTP/1.1" 200 9
^C(coverage) reeteshranjan@test:~/coverage-testapp$
I see the coverage data file in the directory:
(coverage) reeteshranjan@test:~/coverage-testapp$ ls -la
total 72
drwxrwxr-x 9 reeteshranjan reeteshranjan 4096 Jun 14 21:24 .
drwxr-xr-x 9 reeteshranjan reeteshranjan 4096 Jun 14 21:23 ..
-rw-rw-r-- 1 reeteshranjan reeteshranjan 1672 Jun 14 21:24 .coverage
drwxrwxr-x 8 reeteshranjan reeteshranjan 4096 Jun 14 21:13 .git
-rw-rw-r-- 1 reeteshranjan reeteshranjan 47 Jun 14 06:47 .gitignore
drwxrwxr-x 2 reeteshranjan reeteshranjan 4096 Jun 14 05:42 .setup
drwxrwxr-x 2 reeteshranjan reeteshranjan 4096 Jun 14 21:23 api
drwxrwxr-x 4 reeteshranjan reeteshranjan 4096 Jun 14 06:02 conf
drwxrwxr-x 3 reeteshranjan reeteshranjan 4096 Jun 14 21:09 core
drwxrwxr-x 2 reeteshranjan reeteshranjan 4096 Jun 14 21:23 covtest
drwxrwxr-x 2 reeteshranjan reeteshranjan 4096 Jun 14 04:03 log
-rwxrwxr-x 1 reeteshranjan reeteshranjan 250 Jun 14 21:05 manage.py
-rw-rw-r-- 1 reeteshranjan reeteshranjan 236 Jun 14 06:57 requirements.txt
-rwxrwxr-x 1 reeteshranjan reeteshranjan 296 Jun 14 06:04 reset-db.sh
-rwxrwxr-x 1 reeteshranjan reeteshranjan 12895 Jun 14 06:08 setup.sh
And I collect the coverage report which shows 0% coverage in api/views.py
(coverage) reeteshranjan@test:~/coverage-testapp$ coverage report
Name Stmts Miss Cover
-----------------------------------------------------
api/__init__.py 0 0 100%
api/admin.py 1 0 100%
api/apps.py 4 4 0%
api/models.py 2 0 100%
api/tests.py 1 1 0%
api/urls.py 3 3 0%
api/views.py 8 8 0%
core/__init__.py 0 0 100%
core/admin.py 1 0 100%
core/apps.py 4 4 0%
core/migrations/0001_initial.py 7 7 0%
core/migrations/__init__.py 0 0 100%
core/models.py 19 0 100%
core/tests.py 1 1 0%
core/views.py 1 1 0%
covtest/__init__.py 0 0 100%
covtest/settings.py 19 0 100%
covtest/urls.py 3 3 0%
covtest/wsgi.py 4 4 0%
manage.py 6 0 100%
-----------------------------------------------------
TOTAL 84 36 57%
(coverage) reeteshranjan@test:~/coverage-testapp$
Original comment by Reetesh Ranjan (Bitbucket: reeteshranjan, GitHub: reeteshranjan)
I found the issue. It was some messy conflict between package names. I create the django project with the name 'coverage' and it was conflicting with the package name that coverage.py uses. Changed my django project name to 'covtest' and things are fine. I have pushed the changes to the sample app repository. Please pull the latest.
Original comment by Reetesh Ranjan (Bitbucket: reeteshranjan, GitHub: reeteshranjan)
Adding output of coverage debug sys
in case that helps:
(coverage) reeteshranjan@test:~/coverage-testapp$ coverage debug sys
-- sys -------------------------------------------------------
version: 4.1
coverage: /home/reeteshranjan/.virtualenvs/coverage/local/lib/python2.7/site-packages/coverage/__init__.pyc
cover_dirs: /home/reeteshranjan/.virtualenvs/coverage/lib/python2.7/site-packages/coverage
pylib_dirs: /usr/lib/python2.7
tracer: CTracer
plugins.file_tracers: -none-
config_files: .coveragerc
setup.cfg
configs_read: -none-
data_path: /home/reeteshranjan/coverage-testapp/.coverage
python: 2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2]
platform: Linux-3.19.0-25-generic-x86_64-with-Ubuntu-14.04-trusty
implementation: CPython
executable: /home/reeteshranjan/.virtualenvs/coverage/bin/python
cwd: /home/reeteshranjan/coverage-testapp
path:
/home/reeteshranjan/.virtualenvs/coverage/lib/python2.7
/home/reeteshranjan/.virtualenvs/coverage/lib/python2.7/plat-x86_64-linux-gnu
/home/reeteshranjan/.virtualenvs/coverage/lib/python2.7/lib-tk
/home/reeteshranjan/.virtualenvs/coverage/lib/python2.7/lib-old
/home/reeteshranjan/.virtualenvs/coverage/lib/python2.7/lib-dynload
/usr/lib/python2.7
/usr/lib/python2.7/plat-x86_64-linux-gnu
/usr/lib/python2.7/lib-tk
/home/reeteshranjan/.virtualenvs/coverage/local/lib/python2.7/site-packages
/home/reeteshranjan/.virtualenvs/coverage/lib/python2.7/site-packages
environment: -none-
command_line: /home/reeteshranjan/.virtualenvs/coverage/bin/coverage debug sys
source_match: -none-
source_pkgs_match: -none-
include_match: -none-
omit_match: -none-
cover_match: /home/reeteshranjan/.virtualenvs/coverage/lib/python2.7/site-packages/coverage
pylib_match: /usr/lib/python2.7
Original comment by Reetesh Ranjan (Bitbucket: reeteshranjan, GitHub: reeteshranjan)
@nedbat : I have created a test application that runs using a virtual environment. Please get it here. Please use the README to set the project. I have included a script setup.sh
that does everything e.g. virtual environment creation, system packages and python package installations, creating Cassandra and PostgreSQL databases and users, migrating the django models etc. so it should be very easy to get the app up and running. Please refer the README for more.
However; I am running into a new problem and not able to get to the coverage step. This is the issue I am facing:
My app runs fine when started in the usual way:
(coverage) reeteshranjan@test:~/coverage-testapp$ ./manage.py runserver
Performing system checks...
System check identified no issues (0 silenced).
June 14, 2016 - 01:51:17
Django version 1.9.7, using settings 'coverage.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
Please note that I am inside my virtual environment. The state of my virtual environment is in sync with the requirements.txt
I have included in the repository.
Calling the test API also works fine:
reeteshranjan@test:~/coverage-testapp$ curl "http://127.0.0.1:8000/api/v1/testcov"
All well!(coverage) reeteshranjan@test:~/coverage-testapp$
However; when I try to run coverage.py, I get into the following:
(coverage) reeteshranjan@test:~/coverage-testapp$ which coverage
/home/reeteshranjan/.virtualenvs/coverage/bin/coverage
(coverage) reeteshranjan@test:~/coverage-testapp$ coverage run ./manage.py runserver
Traceback (most recent call last):
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/home/reeteshranjan/.virtualenvs/coverage/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
utility.execute()
File "/home/reeteshranjan/.virtualenvs/coverage/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 302, in execute
settings.INSTALLED_APPS
File "/home/reeteshranjan/.virtualenvs/coverage/local/lib/python2.7/site-packages/django/conf/__init__.py", line 55, in __getattr__
self._setup(name)
File "/home/reeteshranjan/.virtualenvs/coverage/local/lib/python2.7/site-packages/django/conf/__init__.py", line 43, in _setup
self._wrapped = Settings(settings_module)
File "/home/reeteshranjan/.virtualenvs/coverage/local/lib/python2.7/site-packages/django/conf/__init__.py", line 99, in __init__
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named settings
Somehow the django settings cannot be loaded, and I am not sure why. Is there something different needed to make coverage.py work with django while running in a virtual environment? I did not have this issue earlier when I was running things globally (not in a virtual environment). I tried to look up; but could not find much on this.
Once this works, I would be able to hopefully reproduce the issue.
@reeteshranjan do you have any more information about this?
Let's start from the beginning: double-check all of the versions of code, perhaps with a fresh virtualenv. Show the output of pip freeze, and also "coverage debug sys". If it still doesn't work, try using --timid
on the coverage run line.
Original comment by Reetesh Ranjan (Bitbucket: reeteshranjan, GitHub: reeteshranjan)
Please ignore my last comment. The new version is 4.0.3. I assumed it's the same as the older one, when it's not, as the earlier one is 4.0.2.
However; still wanted to understand if the current version 4.0.3 has this fix, or I am still running a version without the fix.
Original comment by Reetesh Ranjan (Bitbucket: reeteshranjan, GitHub: reeteshranjan)
I just checked in the initial report above that the coverage version was 4.0.2 and when I ran pip install --upgrade coverage
today, it did compile some new files and indicated that older coverage was uninstalled. However; after the upgrade the version stays at 4.0.2. Is that OK?
Also, this is where coverage runs from on my server:
$ which coverage
/usr/local/bin/coverage
Fixed in 231bad38c241 (bb).
Possible fix for #445 and #420
The line that seems to break #445 is the import of weakref, even if we never use it. Delaying the import until we need it seems to fix #445.
→ <<cset 231bad38c241 (bb)>>
Seems to be this commit: 2ff50c9162ce (bb) , which is also implicated in #420.
It seems that whatever is causing the problem, it happened between 4.0a1 and 4.0a2.
Thanks for these instructions, they are perfect! I have reproduced the issue. Now to dig in to figure out what's going wrong...
Original comment by Bowen Song (Bitbucket: bowensong, GitHub: bowensong)
Hi @nedbat,
Here's the steps to create a minimal test environment, and reproduce the issue:
1, Create a virtualenv (I'm using virtualenvwrapper
):
mkvirtualenv test
2, Activate the venv:
workon test
3, Install packages:
pip install django django-cassandra-engine coverage
4, Create a django project:
django-admin startproject coverage_cassandra_test
5, Edit the django settings.py
file
django_cassandra_engine
to the INSTALLED_APPS
list:INSTALLED_APPS = (
'django_cassandra_engine',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
)
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
},
'cassandra': {
'ENGINE': 'django_cassandra_engine',
'NAME': 'system',
'HOST': 'localhost',
'USER': 'myuser',
'PASSWORD': 'mypass',
},
}
Note: you need to have Cassandra installed and running on your test environment.
6, Run python manage.py
, output should be something like this:
Type 'manage.py help <subcommand>' for help on a specific subcommand.
Available subcommands:
[auth]
changepassword
createsuperuser
[django]
check
compilemessages
createcachetable
dbshell
diffsettings
dumpdata
inspectdb
loaddata
makemessages
makemigrations
runfcgi
shell
showmigrations
sql
sqlall
sqlclear
sqlcustom
sqldropindexes
sqlflush
sqlindexes
sqlmigrate
sqlsequencereset
squashmigrations
startapp
startproject
test
testserver
validate
[django_cassandra_engine]
flush
migrate
runserver
sync_cassandra
syncdb
[sessions]
clearsessions
[staticfiles]
collectstatic
findstatic
7, Run coverage run manage.py
, result:
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/home/bowens/.virtualenvs/test/lib/python2.7/site-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
utility.execute()
File "/home/bowens/.virtualenvs/test/lib/python2.7/site-packages/django/core/management/__init__.py", line 328, in execute
django.setup()
File "/home/bowens/.virtualenvs/test/lib/python2.7/site-packages/django/__init__.py", line 18, in setup
apps.populate(settings.INSTALLED_APPS)
File "/home/bowens/.virtualenvs/test/lib/python2.7/site-packages/django/apps/registry.py", line 108, in populate
app_config.import_models(all_models)
File "/home/bowens/.virtualenvs/test/lib/python2.7/site-packages/django/apps/config.py", line 198, in import_models
self.models_module = import_module(models_module_name)
File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/home/bowens/.virtualenvs/test/lib/python2.7/site-packages/django_cassandra_engine/models.py", line 12, in <module>
cassandra_connection.connect()
File "/home/bowens/.virtualenvs/test/lib/python2.7/site-packages/django_cassandra_engine/base/__init__.py", line 94, in connect
self.connection = CassandraConnection(**settings)
File "/home/bowens/.virtualenvs/test/lib/python2.7/site-packages/django_cassandra_engine/connection.py", line 53, in __init__
self.setup()
File "/home/bowens/.virtualenvs/test/lib/python2.7/site-packages/django_cassandra_engine/connection.py", line 60, in setup
connection.setup(self.hosts, self.keyspace, **self.connection_options)
File "/home/bowens/.virtualenvs/test/lib/python2.7/site-packages/cassandra/cqlengine/connection.py", line 131, in setup
session = cluster.connect()
File "cassandra/cluster.py", line 820, in cassandra.cluster.Cluster.connect (cassandra/cluster.c:11444)
with self._lock:
File "cassandra/cluster.py", line 846, in cassandra.cluster.Cluster.connect (cassandra/cluster.c:11266)
raise
File "cassandra/cluster.py", line 840, in cassandra.cluster.Cluster.connect (cassandra/cluster.c:11146)
self.control_connection.connect()
File "cassandra/cluster.py", line 2088, in cassandra.cluster.ControlConnection.connect (cassandra/cluster.c:36955)
self._set_new_connection(self._reconnect_internal())
File "cassandra/cluster.py", line 2123, in cassandra.cluster.ControlConnection._reconnect_internal (cassandra/cluster.c:37811)
raise NoHostAvailable("Unable to connect to any servers", errors)
NoHostAvailable: ('Unable to connect to any servers', {'localhost': OperationTimedOut('errors=Timed out creating connection (5 seconds), last_host=None',)})
Environment information: Running on CentOS 7 with Python 2.7.5
Python packages:
cassandra-driver==2.7.2
coverage==4.0.2
Django==1.8.6
django-cassandra-engine==0.5.2
futures==3.0.3
six==1.10.0
wheel==0.24.0
I hope that's enough for you, and let me know if you need more information.
@bowensong Can you provide me with detailed instructions on how to reproduce the problem? Perhaps a simple program that shows the same issue?
Sorry, I'm not sure what else to try, unless you can give me access to the code, and detailed instructions about how to reproduce the problem.
Original comment by Reetesh Ranjan (Bitbucket: reeteshranjan, GitHub: reeteshranjan)
I have verified and to the best of my knowledge the django test web server (what I used above) has no concurrency enabled.
Original comment by Reetesh Ranjan (Bitbucket: reeteshranjan, GitHub: reeteshranjan)
So in the above comment, I have not used gunicorn. It's the basic web server that django provides for testing. I am not sure if that uses any concurrency; but I'll find out and revert. Thanks!
What kind of gunicorn worker are you using? Can you try the --concurrency
switch that corresponds?
Original comment by Reetesh Ranjan (Bitbucket: reeteshranjan, GitHub: reeteshranjan)
Even if this worked and I was able to get some coverage output, it is incorrect. I ran my tests that made the code flow across all the functions in my Django views.py; but the coverage shown is 0%! Here is the process I used:
$ coverage run --source='.' manage.py runserver
Then I ran my testcases that make several http calls for exercising several APIs in the code including all possible branches of code.
Then I used ctrl-C to make the server quit which created the .coverage file, and I got the output using coverage report/coverage html.
Is there anything wrong in this process?
Original comment by Reetesh Ranjan (Bitbucket: reeteshranjan, GitHub: reeteshranjan)
I have not directly imported gevent, eventlet, or greenlet. However; using the --timid flag worked. Could you please let me know what that could have done in this case?
Original comment by Reetesh Ranjan (Bitbucket: reeteshranjan, GitHub: reeteshranjan)
Here is the output of pip freeze:
Django==1.8.3
Markdown==2.6.2
MySQL-python==1.2.5
PAM==0.4.2
Pillow==2.3.0
Pygments==2.0.2
South==0.8.2
Twisted-Core==13.2.0
Twisted-Web==13.2.0
adium-theme-ubuntu==0.3.4
amqp==1.4.6
anyjson==0.3.3
apns==2.0.1
apt-xapian-index==0.45
argparse==1.2.1
billiard==3.3.0.20
blist==1.3.6
bpython==0.12
cassandra-driver==2.6.0
cassandra-pylib==0.0.0
celery==3.1.18
chardet==2.0.1
colorama==0.2.5
command-not-found==0.3
compizconfig-python==0.9.11.3
coverage==4.0.2
debtagshw==0.1
defer==1.0.6
diff-match-patch==20121119
dirspec==13.10
django-ajax-selects==1.3.3
django-braces==1.2.2
django-cassandra-engine==0.4.0
django-cors-headers==1.1.0
django-debug-toolbar==1.0.1
django-discover-runner==0.4
django-extensions==1.3.3
django-filter==0.10.0
django-import-export==0.2.7
django-model-utils==1.5.0
django-redis==4.2.0
django-suit==0.2.13
django-taggit==0.14.0
djangorestframework==3.1.3
duplicity==0.6.23
figleaf==0.6.1
futures==3.0.3
gunicorn==17.5
gyp==0.1
html5lib==0.999
httplib2==0.8
jsonfield==1.0.3
kombu==3.0.26
lockfile==0.8
logutils==0.3.3
lxml==3.3.3
msgpack-python==0.4.6
oauthlib==0.6.1
oneconf==0.3.7
pexpect==3.1
piston-mini-client==0.7.5
psycopg2==2.4.5
pyOpenSSL==0.13
pycrypto==2.6.1
pycups==1.9.66
pycurl==7.19.5.1
pygobject==3.12.0
pyserial==2.6
pysmbc==1.0.14.1
python-apt==0.9.3.5ubuntu1
python-debian==0.1.21-nmu2ubuntu2
pytz==2015.4
pyxdg==0.25
redis==2.10.3
reportlab==3.0
requests==2.2.1
sessioninstaller==0.0.0
six==1.9.0
software-center-aptd-plugins==0.0.0
sqlparse==0.1.15
ssh-import-id==3.21
stevedore==0.14.1
system-service==0.1.6
tablib==0.10.0
ubuntu-tweak==0.8.7
unity-lens-photos==1.0
urllib3==1.7.1
virtualenv==1.11.4
virtualenv-clone==0.2.4
virtualenvwrapper==4.1.1
wheel==0.24.0
wsgiref==0.1.2
xdiagnose==3.6.3build2
zope.interface==4.0.5
Also, can you try the --timid
flag?
Interesting report! Can you provide some more diagnostics? What is the output of "pip freeze" for you? Are you using gevent, eventlet, or greenlet? If so, try providing the --concurrency
flag.
Original comment by Reetesh Ranjan (Bitbucket: reeteshranjan, GitHub: reeteshranjan)
I just tried figleaf and it is able to start the runserver with $ figleaf ./manage.py runserver
Originally reported by Reetesh Ranjan (Bitbucket: reeteshranjan, GitHub: reeteshranjan)
We have a django app that talks with 2 DBs - postgresql and cassandra. Though the django app's runserver starts up fine and work (with ./manage.py runserver) as well as we have a gunicorn+nginx setup that works fine, when I try to measure coverage using a command like this:
coverage run --source='.' manage.py runserver
things don't work as the django app bails out with connection failure with cassandra.
Nothing changes across running with coverage or running normally. Cassandra server has not stopped across these 2 ways of running the server. Is there some issue with authentication that cannot be taken care when running with coverage. All the auth is anyway in the django settings, and there is no hack or anything that goes beyond django structure.
Here is the trace of the failure:
And here is how it works fine without coverage:
System check identified no issues (0 silenced). November 12, 2015 - 10:52:50 Django version 1.8.3, using settings 'sweatt.settings' Starting development server at http://127.0.0.1:8000/ Quit the server with CONTROL-C.
I have installed the coverage tool today and I can see the version to be 4.0.2 as in /usr/local/bin/coverage, /usr/local/bin/coverage2 and /usr/local/bin/coverage-2.7.
I am using ubuntu 14.04 and python 2.7.