search

Tribler / dispersy

The elastic database system. A database designed for P2P-like scenarios, where potentially millions of computers send database updates around.
http://www.tribler.org
86 stars 43 forks source link

Objects that do a database call in their init #484

Open lfdversluis opened 8 years ago

lfdversluis commented 8 years ago

I have run the unit tests of dispersy and profiled them with yappi. I exported the output to callgrind format and exported this into a dotfile using gprof2dot.

Using this code I parsed the dotfile to find inits that call the database eventually.

I noticed that a lot of database functions are not covered by the tests such as executemany. So this list is most definitely incomplete.

Below the results from this run: dispersy/database.py:database:250:DispersyDatabase.execute dispersy/crypto.py:crypto:253:M2CryptoPK.__init__ dispersy/util.py:util:146:wrapper

edit: I think the code can be improved. Doing that now

lfdversluis commented 8 years ago

Just realized that this whole approach isn't going to work at all. If a method uses a decorator that is also used for database operations then it will think that it (may, and thus will when doing a dfs) do database operations eventually while this may not be the case.

In any case here is the output that lead me to this conclusion:

dispersy/crypto.py:crypto:253:M2CryptoPK.__init__ -> dispersy/util.py:util:146:wrapper -> dispersy/database.py:database:250:DispersyDatabase.execute
dispersy/crypto.py:crypto:328:M2CryptoSK.__init__ -> dispersy/util.py:util:146:wrapper -> dispersy/database.py:database:250:DispersyDatabase.execute
debugcommunity/node.py:node:33:DebugNode.__init__ -> dispersy/member.py:member:73:Member.__init__ -> dispersy/crypto.py:crypto:146:ECCrypto.key_to_bin -> dispersy/util.py:util:146:wrapper -> dispersy/database.py:database:250:DispersyDatabase.execute
dispersy/member.py:member:73:Member.__init__ -> dispersy/crypto.py:crypto:146:ECCrypto.key_to_bin -> dispersy/util.py:util:146:wrapper -> dispersy/database.py:database:250:DispersyDatabase.execute
dispersy/message.py:message:321:Implementation.__init__ -> dispersy/util.py:util:146:wrapper -> dispersy/database.py:database:250:DispersyDatabase.execute

Everywhere is the wrapper involved, which is the attach_runtime_statistics decorator method.

lfdversluis commented 8 years ago

Plan 2: Analyze the call stack at the database function level, if it contains an __init__ throw an exception and fix the case. Rinse and repeat. (Thread switching kills this approach though)