Snitches not going off at all. Server might just need a restart.

[ProgrammerDan]

Which server?

[kujukuju]

The shard is Aleph. The server address is mc.civcraft.co.

[awoo-civ]

It's happening to me and multiple other people in my group as well, in Bet. We have snitches in Bet and Gimmel, and can't see reports from either of the shards while being in Bet.

The problem first started after I'd logged in at 2016-03-11 ~20:52 UTC and it's been fairly inconsistent in when it happens. For example, the latest server restart on 2016-03-12 ~16:55 has fixed it, but I've stopped receiving snitch reports after 2016-03-13 00:46:42

Additionally, when snitches are bugged, going from Bet to Gimmel unbugs Bet snitches for me as long as I stay in Gimmel. I'm not sure if it unbugs Gimmel snitches however.

[ProgrammerDan]

Thanks guys, I'll see what I can do to track this down.

[kujukuju]

I haven't confirmed this but I think snitches work again after you log out and back in. Someone should verify and report back.

[awoo-civ]

@TrentWDB Nope, logging in/out, even multiple times, doesn't fix it.

[ProgrammerDan]

Yeah, it won't ... I think I found the issue. Working on a fix right now. Thanks again for the reports, folks.

[awoo-civ]

The issue is still there. The most recent server restart has fixed it for a couple of minutes, but then it bugged out again.

[ProgrammerDan]

Yeah. I have to dig deeper. Further into the matrix ....

[ProgrammerDan]

@Lcth and @TrentWDB Still occurring? Any better?

Please let me know the name of the groups your snitches are on, that aren't working.

[awoo-civ]

@ProgrammerDan No change as far as I can tell. I'll PM you the group names on reddit.

[ProgrammerDan]

Thank you, that'll be helpful.

[ProgrammerDan]

@Lcth Still no luck?

Can you measure for me approximately how long they work before they "stop" working? Does changing shards start them working again?

[kujukuju]

Maybe it's relevant that this bug isn't per group, its per person. A few days ago Syril couldn't see the Meridian snitches while I (Bunnub) who just logged on could see them. I haven't noticed this glitch happening the last day or two but that being said it's not very common.

Off topic but check modmail please - jo_rey and I were mysteriously banned, lol.

[ProgrammerDan]

Sorry, modmail queue is very long. We're sorting through it.

I think I have an idea what's causing this issue, thanks -- I had also begun to suspect that it was a per-user problem, because of certain errors being thrown and (now) captured.

[awoo-civ]

@ProgrammerDan I found out something interesting while looking through my chat logs:

On the day snitches started bugging out the server suddenly shut down, in what I'm assuming was a crash. 20 minutes after it came back online, snitches broke for the first time. Maybe the crash corrupted data of users who were online at the time?

Timeline:

2016-03-11 (UTC) 16:12 - Restart 16:45 - Restart (no warning, likely a crash) 17:44 - Last snitch notification

2016-03-12 16:54 Restart 17:38 Last snitch notification

2016-03-13 18:54 Restart 19:23 Last snitch notification

2016-03-14 01:51 Restart 02:22 Last snitch notification

These are the only clear restart/last notification pairs i could find.

As for shard borders, it's still the same. If I go from Bet to Gimmel while snitches are bugged, I start receiving snitch notifications from Bet. However, it bugs out again once I come back to Bet.

[ProgrammerDan]

So... this definitely helps tie things together for me. I've added all sorts of null traps and introspection; and I know the groups that are causing things to fail.

This would explain why those groups were screwed up, potentially.

[ProgrammerDan]

@Lcth or @TrentWDB can you confirm issue is still presenting?

[kujukuju]

I would, but I'm banned. ;P

[ProgrammerDan]

RIP forgot about that. @Lcth up to you then!

[awoo-civ]

@ProgrammerDan So far snitches have been working. Good job :)

[ProgrammerDan]

Okay great. Let me know if that changes!

[ttk2]

closing for now then.