Supy gets stuck shutting down

[weedy]

INFO 2013-02-08T06:45:17 Removing driver SocketDriver(Irc object for rizon).
INFO 2013-02-08T06:45:17 Removing driver SocketDriver(Irc object for irchighway).
INFO 2013-02-08T06:45:17 Removing driver SocketDriver(Irc object for wsynet).
DEBUG 2013-02-08T06:45:17 Outgoing message (freenode): QUIT :SmugLeaf
INFO 2013-02-08T06:45:17 Driver for Irc object for freenode dying.
INFO 2013-02-08T06:45:17 Irc object for freenode dying.
INFO 2013-02-08T06:45:17 Driver for Irc object for freenode dying.

Then nothing for minutes/hours/days/etc. strace

munmap(0xb70dc000, 4096)                = 0
stat64("/home/weedy/supybot/data/tmp/Seen.db.664dce4bd8866b4b4d5760b4a8421f8ccb08be43", {st_mode=S_IFREG|0644, st_size=3640469, ...}) = 0
stat64("/home/weedy/supybot/data/Seen.db", {st_mode=S_IFREG|0644, st_size=3640550, ...}) = 0
stat64("/home/weedy/supybot/data/Seen.db", {st_mode=S_IFREG|0644, st_size=3640550, ...}) = 0
open("/home/weedy/supybot/data/Seen.db", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=3640550, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb70dc000
fstat64(3, {st_mode=S_IFREG|0644, st_size=3640550, ...}) = 0
_llseek(3, 3640550, [3640550], SEEK_SET) = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=3640550, ...}) = 0
close(3)                                = 0
munmap(0xb70dc000, 4096)                = 0
stat64("/home/weedy/supybot/data/Seen.db", {st_mode=S_IFREG|0644, st_size=3640550, ...}) = 0
rename("/home/weedy/supybot/data/tmp/Seen.db.664dce4bd8866b4b4d5760b4a8421f8ccb08be43", "/home/weedy/supybot/data/Seen.db") = 0
brk(0xc6e4000)                          = 0xc6e4000

The I hit ctrl-c

--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
sigreturn() (mask [])                   = -1217588448
stat64("/home/weedy/supybot/lib/python2.7/site-packages/supybot/drivers/__init__.py", {st_mode=S_IFREG|0644, st_size=7950, ...}) = 0
open("/home/weedy/supybot/lib/python2.7/site-packages/supybot/drivers/__init__.py", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=7950, ...}) = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=7950, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb70dc000
read(3, "###\n# Copyright (c) 2002-2004, J"..., 8192) = 7950
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0xb70dc000, 4096)                = 0
stat64("/home/weedy/supybot/lib/python2.7/site-packages/supybot/drivers/Socket.py", {st_mode=S_IFREG|0644, st_size=13842, ...}) = 0
open("/home/weedy/supybot/lib/python2.7/site-packages/supybot/drivers/Socket.py", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=13842, ...}) = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=13842, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb70dc000
read(3, "###\n# Copyright (c) 2002-2004, J"..., 8192) = 8192
read(3, "ve up and replace the offending "..., 4096) = 4096
read(3, " select.select([], [self.conn], "..., 4096) = 1554
read(3, "", 4096)                       = 0
close(3)                                = 0

[weedy]

bump

[progval]

Ànd the bot exits after the last Ctrl-C?

[weedy]

Yes it does.

[weedy]

Now it segfaults after the last ctrl-c

[progval]

try unloading some plugins which are not provided with Limnoria

[weedy]

What are the default plugins? On 3 Sep 2013 03:59, "Valentin Lorentz" notifications@github.com wrote:

try unloading some plugins which are not provided with Limnoria

— Reply to this email directly or view it on GitHubhttps://github.com/ProgVal/Limnoria/issues/450#issuecomment-23695605 .

[weedy]

I unloaded basically everything except Admin, Channel, Config, Ctcp, Network, Owner, Plugin, and User. Starts up and shuts down fine. Then I loaded Seen and everything went to shit. Are my databases too big?

-rw-r--r-- 1 weedy users 18M Sep 11 09:44 data/Seen.any.db -rw-r--r-- 1 weedy users 3.7M Sep 11 09:44 data/Seen.db

[progval]

Hmm… delete these files (keep a backup somewhere), and see if your problem is fixed

[weedy]

Obviously nuking the database fixes it. I was trying to point out that the database format or implementation is so terrible that i'll probably have to cronjob a database nuke once a year.

[progval]

What database engine do you use? (it is configurable)

weedy notifications@github.com a écrit :

Obviously nuking the database fixes it. I was trying to point out that the database format or implementation is so terrible that i'll probably have to cronjob a database nuke once a year.

---

Reply to this email directly or view it on GitHub: https://github.com/ProgVal/Limnoria/issues/450#issuecomment-24247705

Sent from my Android phone with K-9 Mail. Please excuse my brevity.

[weedy]

I'm failing to see where it could be configured. Seen.init just does CSV, and the files are CSV.

[progval]

On 11/09/2013 17:15, weedy wrote:

I'm failing to see where it could be configured. Seen.init just does CSV, and the files are CSV.

---

Reply to this email directly or view it on GitHub: https://github.com/ProgVal/Limnoria/issues/450#issuecomment-24248694 It is configured in supybot.databases (it is global to all plugins).

[weedy]

@config supybot.databases weedy: sqlite3 anydbm cdb flat pickle

[progval]

weird… sorry, I don't know what to do

weedy notifications@github.com a écrit :

@config supybot.databases weedy: sqlite3 anydbm cdb flat pickle

---

Reply to this email directly or view it on GitHub: https://github.com/ProgVal/Limnoria/issues/450#issuecomment-24265718

Sent from my Android phone with K-9 Mail. Please excuse my brevity.

[weedy]

Can the plugin be updated to like sqlite or something faster then comma separated junk?

[progval]

If you have set sqlite3 in supybot.databases, then the plugin should be using it.

Try to install python-sqlite3 (but if it is not installed, it should not be appearing in supybot.databases…)

[Mikaela]

Has anything new happened with this issue?

[weedy]

I wrote a watchdog that deals with supy locking up.

Which is to say, no supy still cries when it loads my seen database.

[Mikaela]

Did you ever install python3-sqlite and add it to supybot.databases? config setdefault databases should be able to do the latter.

Also which version are you using? Have you upgraded after 2013?

[weedy]

11:08:09 <ph34r> sqlite3 anydbm dbm cdb flat pickle

Python 2.7.6 (default, Apr  8 2014, 16:22:42) 
[GCC 4.6.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> 

I run master, I've always had sqlite support.

[Mikaela]

supybot --version?

[weedy]

Like I said, master. 0.83.4.1+limnoria 2014.06.26 607e5e0a680df385772ba8cad7d08006c9cd2f3e

[Mikaela]

I just wanted to verify that you are running the latest master as master can mean very old version.

[weedy]

Today I tested deleting my Seen databases and letting the bot start fresh.

It still uses CSV.

supybot/plugins/Seen/plugin.py

class SeenDB(plugins.ChannelUserDB):
    IdDict = IrcStringAndIntDict
    def serialize(self, v):
        return list(v)

supybot/plugins/__init__.py

class ChannelUserDB(ChannelUserDictionary):
    def __init__(self, filename):
        ChannelUserDictionary.__init__(self)
        self.filename = filename
        try:
            fd = codecs.open(self.filename, encoding='utf8')
        except EnvironmentError as e:
            log.warning('Couldn\'t open %s: %s.', self.filename, e)
            return
        reader = csv.reader(fd)

/home/weedy/supybot/lib/python2.7/site-packages/supybot/plugins/Herald/plugin.py:class HeraldDB(plugins.ChannelUserDB):
/home/weedy/supybot/lib/python2.7/site-packages/supybot/plugins/ChannelStats/plugin.py:class StatsDB(plugins.ChannelUserDB):
/home/weedy/supybot/lib/python2.7/site-packages/supybot/plugins/ChannelStats/plugin.py:        plugins.ChannelUserDB.__init__(self, *args, **kwargs)
/home/weedy/supybot/lib/python2.7/site-packages/supybot/plugins/Seen/plugin.py:class SeenDB(plugins.ChannelUserDB):
/home/weedy/supybot/lib/python2.7/site-packages/supybot/plugins/__init__.py:class ChannelUserDB(ChannelUserDictionary):
/home/weedy/supybot/plugins/repos/ProgVal/Biography/plugin.py:class BiographyDB(plugins.ChannelUserDB):
/home/weedy/supybot/plugins/repos/supybot-plugins/WordStats/plugin.py:class WordStatsDB(plugins.ChannelUserDB):
/home/weedy/supybot/plugins/repos/supybot-plugins/WordStats/plugin.py:        plugins.ChannelUserDB.__init__(self, *args, **kwargs)
/home/weedy/supybot/plugins/repos/supybot-plugins/WordStats/plugin.py:            plugins.ChannelUserDB.close(self)

There are many places where we use CSV and you might want to think about the performance implications of continuing to use it when you have bots that are years+ old.