Some imports in Python 3 break flame graph rendering

[sims1253]

I ran vprof -c cmh -s "evaluate.py -o 7496a0471ba8259926b36028b79b5a0e62ecf03c -d 510 -f 260 -s 25" and no results were presented in any way. If I understand correctly there should appear new tabs in my browser but that didn't happen.

Also: I had to pass the hash without single quotation marks as vprof would put another pair around them. Is this intended?

Platform: python3.5:

Python 3.5.2 (default, Jun 28 2016, 08:46:01) 
[GCC 6.1.1 20160602] on linux

Chromium Version 53.0.2785.116 (64-bit)

I am running Antergos linux

[nvdv]

I ran vprof -c cmh -s "evaluate.py -o 7496a0471ba8259926b36028b79b5a0e62ecf03c -d 510 -f 260 -s 25" and no results were presented in any way.

How long does your script run without profiling ? Can you run it just with -c c option?

If I understand correctly there should appear new tabs in my browser but that didn't happen.

Yep. that's right.

I had to pass the hash without single quotation marks as vprof would put another pair around them. Is this intended?

Can you elaborate?

Thanks!

[sims1253]

How long does your script run without profiling ? Can you run it just with -c c option? Depending on the git repository it can go up to a few hours. Its an evaluation script for my thesis so I usually run it overnight. Just started it with only the c config for a small repo so it should be done in half an hour or so.

Can vprof open chromium or does it have to be open already?

Can you elaborate? I always passed strings with '' around them using argparse. just tried it without and it seems to work. When I had 'hash' instead of just hashin the parameter list using vprof i got this: error: pathspec ''7496a0471ba8259926b36028b79b5a0e62ecf03c'' did not match any file(s) known to git. so there was an extra pair added. But now I think I just don't need them anyway

Thank you for the fast response :)

Edit: Oh and it didn't feel like profiling was slowing down the script big way but that could just be me not noticing it. Might time it later if you are interested.

[nvdv]

Depending on the git repository it can go up to a few hours. Its an evaluation script for my thesis so I usually run it overnight. Just started it with only the c config for a small repo so it should be done in half an hour or so.

c options has lowest overhead, m and h overhead is larger. Plus your script is run every time for every option. For now vprof is not very suitable for "big picture" profiles.

Can vprof open chromium or does it have to be open already?

vprof opens web browser automatically, but only after all data was collected.

I always passed strings with '' around them using argparse. just tried it without and it seems to work.

vprof treats string in double quotes as single argument to be able to process scripts with arguments.

[sims1253]

Ok now the new tab opens but the loading animation is the only thing I get.

Edit: Ok tried all three options alone and for the flame graph I just get one single chunk that says ~:0(<built-in method builtins.exec>) For memory I get loading screen. Heatmap seems to work as intended.

maybe I should add that the script is very memory hungry. 20+ GB are easily possible

[nvdv]

Thanks!

Edit: Ok tried all three options alone and for the flame graph I just get one single chunk that says ~:0()

This is interesting. Can you make screenshot (if possible, of course)?

For memory I get loading screen. maybe I should add that the script is very memory hungry. 20+ GB are easily possible

Seems that your script generates a lot of data 😄 As I've written above, handling such amounts of data in vprof can be improved in future versions.

EDIT: Is your script Python 2 compatible? If yes, can you run it under Python 2? Thanks!

[sims1253]

I also reduced the memory usage to a few GB but still only the heat map seems to work

how would I tell it to run with python2? looks like vprof uses the system default which is 3 for me I guess

[nvdv]

This looks interesting.

how would I tell it to run with python2? looks like vprof uses the system default which is 3 for me I guess

If your script is Python 2 compatible then you can install vprof with pip for Python 2. Usually it's accessible under pip2. Then you can run it as python -m vprof ...

[sims1253]

well it looks like it doesn't like python 2 :/

[nvdv]

Can you share some details about script you run (libraries you use and so on)? It also might be the reason you get such results.

[sims1253]

https://gitlab.com/sims1253/bachelor_thesis/blob/master/evaluate.py is the repository. Its hacky thesis code though.

In essence i get the git show for a repo via subprocess, for a number of commits, parse the diffs and rank files/lines by the number of fixes they received. See Bugspots for basic idea.

imports:

from __future__ import division
from contextlib import contextmanager
from scipy.integrate import trapz
import numpy as np
import linespots #this is the actual algorithm imported from the same directory
import re
import matplotlib.pyplot as plt
import os
from operator import itemgetter
from timeit import default_timer as timer
from collections import defaultdict

[nvdv]

Thanks, I'll take a look.

[nvdv]

Seems that importing numpy, scipy and matplotlib breaks flame graph rendering in some cases in Python 3.

[nvdv]

Looks like it's fixed in 0.34.