Posting to Discord multiple times

[rinakondur]

I have the bot setup and running with only a single ESI account posting mostly structure notifications to Discord. It's posting everything pretty much three times on whatever cache value is set in the config. I tried raising it as high as 20 minutes to see if there was just some kind of weird issue with it thinking the message didn't go through, but no joy.

Ever run into this before? The only thing I changed were the settings in order to run it. At the moment I'm running on Ubuntu 17.10 with Python 3.6.

[flakas]

Hey @rinakondur ,

I've seen it in one instance, but not as severe as you describe. In my case it happened only once every so often.

In the current example in run.py the CachingNotifier has a caching duration of 3600 seconds (run.py). Try increasing it to something longer, for example 7200 seconds.

I'm guessing that maybe there's an issue with how "new notifications" are selected in tasks.py, causing cached messages to be cleared, but the already used notification is still being treated as "new".

Let me know if that helps or not. Thanks!

[rinakondur]

Adjusted to 7200 seconds. Still triple posting.

On Thu, Aug 16, 2018 at 2:54 AM Tautvidas Sipavičius < notifications@github.com> wrote:

Hey @rinakondur https://github.com/rinakondur ,

I've seen it in one instance, but not as severe as you describe. In my case it happened only once every so often.

In the current example in run.py the CachingNotifier has a caching duration of 3600 seconds (run.py https://github.com/flakas/reconbot/blob/master/run.py#L190). Try increasing it to something longer, for example 7200 seconds.

I'm guessing that maybe there's an issue with how "new notifications" are selected in tasks.py https://github.com/flakas/reconbot/blob/master/reconbot/tasks.py#L15, causing cached messages to be cleared, but the already used notification is still being treated as "new".

Let me know if that helps or not. Thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/flakas/reconbot/issues/9#issuecomment-413444840, or mute the thread https://github.com/notifications/unsubscribe-auth/APseiSbQDe8OyHs5YHGd7H93P_iSDjN9ks5uRRcigaJpZM4V-wHU .

[flakas]

How do you have it configured? Perhaps messages are being posted without being cached? Or have multiple notifiers posting to the same channels? Nothing obvious pops to my mind in code that could cause specifically triple posting.

The logic of posting notifications is as follows:

• Reconbot determines that new notifications are those created in the last hour.
• Because notifications are being pulled every 10 minutes, the same notification could be posted up to 6 times.
• But the CachingNotifier instance configured with 2 hour caching duration should allow only the first message through. The following 5 times it would see the notification has already been sent and so it would discard further duplicates.

If you could share an anonymized version of your config - that could potentially help identify the issue. Thanks!

[rinakondur]

I think I sanitized it enough, hopefully. I tried tweaking the cache timer all different ways with no change in results. Sometimes it posted more than three times but on average it was three. To my knowledge I haven't change any other settings anywhere.

import schedule
import time
import sqlite3

from reconbot.tasks import esi_notification_task
from reconbot.notifiers.caching import CachingNotifier
from reconbot.notifiers.slack import SlackNotifier
from reconbot.notifiers.discord import DiscordNotifier
from reconbot.notifiers.splitter import SplitterNotifier
from reconbot.notifiers.filter import FilterNotifier
from reconbot.apiqueue import ApiQueue
from reconbot.esi import ESI
from reconbot.sso import SSO

# Configuration

db_file = 'datadump/sqlite-latest.sqlite'

# ESI notification endpoint cache timer in minutes
notification_caching_timer = 10

# Discord bot integration API key and channel
discord = {
    'personal': {
        'token': 'xxx',
        'channel_id': 'xxx'
    }
}

# Eve online SSO application Client ID and Secret Key, used to get access
# tokens for ESI API. Get them on:
# https://developers.eveonline.com/applications
sso_app = {
    'client_id': 'xxx',
    'secret_key': 'xxx'
}

# A dictionary of API key groups.
# Get refresh tokens for your characters by following Fuzzwork's guide:
# https://www.fuzzwork.co.uk/2017/03/14/using-esi-google-sheets/
eve_apis = {
    'fc-team': {
        'notifications': {
            'whitelist': [
                'EntosisCaptureStarted',
                'SovCommandNodeEventStarted',
                'SovStructureDestroyed',
                'SovStructureReinforced',
                'StructureUnderAttack',
                'OwnershipTransferred',
                'StructureOnline',
                'StructureFuelAlert',
                'StructureAnchoring',
                'StructureServicesOffline',
                'StructureLostShields',
                'StructureLostArmor',
                'TowerAlertMsg',
                'StationServiceEnabled',
                'StationServiceDisabled',
                'OrbitalReinforced',
                'OrbitalAttacked',
                'SovAllClaimAquiredMsg',
                'SovStationEnteredFreeport',
                'AllAnchoringMsg',
                'SovAllClaimLostMsg',
                'SovStructureSelfDestructRequested',
                'SovStructureSelfDestructFinished',
                'StationConquerMsg',
                'notificationTypeMoonminingExtractionStarted',
                'MoonminingExtractionFinished',
                'MoonminingLaserFired',
            ],
        },
        'characters': {
            'ccp-example-1': {
                'character_name': 'Rina Kondur',
                'character_id': 92014786,
                'refresh_token': 'xxx'
            },
        },
    }
}

def api_to_sso(api):
    return SSO(
        sso_app['client_id'],
        sso_app['secret_key'],
        api['refresh_token'],
        api['character_id']
    )

api_queue_fc = ApiQueue(list(map(api_to_sso, eve_apis['fc-team']['characters'].values())))

db_connection = sqlite3.connect(db_file)
db_connection.row_factory = sqlite3.Row
db = db_connection.cursor()

def notifications_job_fc():
    esi_notification_task(
        eve_apis['fc-team']['notifications'],
        api_queue_fc,
        'discord',
        CachingNotifier(
            SplitterNotifier([
                DiscordNotifier(
                    discord['personal']['token'],
                    discord['personal']['channel_id']
                )
            ]),
            duration=3600
        )
    )

schedule.every(notification_caching_timer/len(eve_apis['fc-team']['characters'])).minutes.do(notifications_job_fc)

while True:
    schedule.run_pending()
    time.sleep(1)

[flakas]

The example run.py script was wrong, I've prepared a new example version of it on master.

The fix itself is in 974495e46c60d03943ed7f27b4569558601c7c2b. The issue was that a new CachingNotifier instance was being created with every notifications_job_fc call, which essentially wiped the cache clean. CachingNotifier is the component that should prevent duplicate notifications.

Moving creation of the notifier chain to outside of the notifications_job_fc() function should resolve the issue.