wevote / WeVoteServer

We Vote's API application server written in Django/Python. Election data pulled from many sources, used by https://github.com/wevote/WebApp and https://github.com/wevote/WeVoteCordova and https://github.com/wevote/Campaigns.
https://api.wevoteusa.org
MIT License
50 stars 454 forks source link

iOS & Android: Update Activity Number on Phones #1450

Closed DaleMcGrew closed 3 years ago

DaleMcGrew commented 4 years ago

We trigger process_activity_notice_seeds_triggered_by_batch_process (WeVoteServer/activity/controllers.py) every minute. I would recommend adding a new block (at the end of the function, so it is run after all of the other processes finish):

    update_interval = 20
    time_now = now()
    if time_now.minute % update_interval == 0:
        continue_retrieving_notices_to_be_updated = True
        activity_notice_seed_id_already_reviewed_list = []
        safety_valve_count = 0
        while continue_retrieving_notices_to_be_updated and \
                safety_valve_count < 1000 and \
                when_process_must_stop > now():
            safety_valve_count += 1

I think having it run every 20 minutes would be sufficient. Within this block we can call a function that reaches out to all registered phones so they can update their activity number.

We currently don't have a way to calculate the number of ActivityNotice entries which have not been seen by the voter (activity_notice_seen == False), but that function can be added to ActivityManager (I would recommend starting with retrieve_activity_notice_list_for_recipient).

SailingSteve commented 4 years ago

I worry about sending 72 mostly unnecessary messages to the phone of every voter every day. We can just keep track of the previous number we sent, and only send updates if it is different.

DaleMcGrew commented 4 years ago

@SailingSteve I totally agree -- we should only reach out to the phone if there is a change.

SailingSteve commented 4 years ago

@DaleMcGrew process_activity_notice_seeds_triggered_by_batch_process seems fragile, and like it is going to have some big problems in the future. I don't see how it could scale to 10X or 100X the volume.

As a first step, what would you think about me not using the "activity app" for the "What's on your mind?" comment (plus any others that would update the Cordova icon notification badge count) and simply doing the notifications in-line (which would provide almost instant notifications to Cordova apps)?

Someday we might have voters with thousands of followers, and at which point we might want to consider using an AWS Lambda function handler in Python. This would remove all the processing from our API server, and move it into an infinitely scalable cloud process.

Actually, if we replaced the batch oriented "activity app" with some real-time AWS Lambda functions it would become completely scaleable -- all "slow batch processing" issues would go away, as well as all "process collision" issues. The voter_daily_summary processing, could also be in AWS Lambdas that were started at some specific time of day.

Alternatively we could replace the "activity" app's homegrown queuing with an AWS Simple Message Queue, which would also solve the scaling problems, and reduce the complexity.

SailingSteve commented 3 years ago

This has been resolved, with a different solution than what was proposed in this issue.