Sms automation - the big picture

[murtraja]

requirements:

1. MM has to manually write the sms, select records and then send them using /sms webapp
2. he wants it to automate it.
3. assume the due date is is 'x', then sms will be sent a month before x, a week before x and a day before x

challenges:

1. /sms uses ajax call to send sms, which means that there should be some client interface(web browser) that takes care of sending the sms
2. PHP script times out when sending more than 300 records. so automating to send so many sms without timeout needs to be done

confusions/questions:

1. how to select records, based on what parameter?
2. how to know, when to send the sms, are the dates mentioned above fixed?
3. to what degree this automation will work, what 'work' we cannot automate, which essentially becomes the bottleneck for this system (e.g. system does not know the dates when to actually send the sms, we need to tell it for every installment?)
4. how does the admin know that sms has been sent
5. what if some sms are sent and some are not
6. how to secure this 'automated' script, so that only cron job can access it, otherwise accidentally calling it will waste a lot of money and create confusion
7. what sms need to be sent on what date? who will write those sms? where will be they stored?

possible solutions:

1. start afresh, make everything new, bring all the aspects of filtering and fetching and club it with sending
2. to fix the timeout problem: 2a. either use threads or asynchronous php stuff 2b. use the bulk sending api
3. Bulk sending api: i talked with mustansir bhai, he says that there is no problem in sending the bulk sms, you can send as many as you like; if that is the case, then why it didn't work before, it worked for two sms, but failed for 300 or so

[yusuf4u52]

Can we automate in batches, send 30 messages? Or are you looking at bulk sms?

[murtraja]

bulk sms will kill two birds with one stone. i have requested MM to contact me whenever he needs to send bulk sms, i will send the sms and verify if it is working or not

P.S. No birds were harmed during the writing of this comment

[murtaza52]

Challenges -

1) I didnt understand this, please explain the problem here. 2) Increase the timeout here. What is timing out apache ? There should be a config to increase the limit.

Confusions -

1) You will select all active thaalis and who have an mount pending for the next installment. 2) Yes fix the dates. Based on one month before miqaat, 1 week before etc. Get this from MM. 3) Y will system not know. Feed in english dates for all miqaats this year, and then let system calculate the dates accordingly. Create a page where all calculated dates are displayed, so that the admin knows when the SMS will be sent. 4) For every SMS that is sent keep a record in DB on what exactly is sent. Once u select the records, that data should be saved in the DB. Using that data send an SMS. Then create an email out of it and send it on the help@faistudents.com. This will ensure that admin knows exactly what SMS were sent. On the SMS page have a log section which lists in descending order SMS that have been sent out - ex 142 Pending Hub SMS on 12/08/2016, 132 Pending Hub SMS on 12/06/2016. If there is n error, send error email instead. 5) When could that happen ? 6) There should be a way through apache to secure it. so that its not exposed on the net. 7) Have a specific SMS associated for each event - before a month, before a week etc. Store it in DB.

Possible solutions - 1) I dont have context on this, please explain. 2) Need to understand better to comment on this. Cant u have a solution where the cron just triggers a script, the script then runs in the background, and updates the DB with the result. So no timeouts or anything. Then another cron job runs, which sends email based on the result of the first cron job. 3) U know better :)

On Wed, Aug 10, 2016 at 8:02 PM, Murtaza Raja notifications@github.com wrote:

bulk sms will kill two birds with one stone. i have requested MM to contact me whenever he needs to send bulk sms, i will send the sms and verify if it is working or not

P.S. No birds were harmed during the writing of this comment

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/murtaza52/faiz-static/issues/240#issuecomment-238932987, or mute the thread https://github.com/notifications/unsubscribe-auth/AAxu_NHlx8kiLIe1ycF0nNxWZ0SvIXTIks5qegQYgaJpZM4Jawg2 .

[murtraja]

Challenges 1) In order to send one SMS we just have to fire one url containing - SMS body, number and auth token. since doing that just once, causes 'x' seconds, for 300 SMSs it will cause 300x seconds, timing out. therefore, the hostinger DOES NOT call the url, the web browser does this through AJAX now since we are automating, there will be no influence of the web browser and consequently AJAX. so essentially what it means is the hostinger has to call the urls and do that within timeout.

2) yusuf bhai tried that, apparently we can't

Confusions.

1) a) the point of asking the question was, what if there is some student(s) who want to be excused for his hub payment this miqaat and he has already taken the approval of MM, in that case, he shouldn't receive the SMS though he has outstanding hub left. it may sound a bit of overthinking at my part and this is a special case but it is not rare, so how do we go about to take care of this situation? (we can't write in every sms that "ignore if already paid" or "ignore if already talked with MM")

b) you mentioned that all active thalis with outstanding amount should receive SMS, what if the outstanding amount is Rs. 100? Should the system ignore? Can we set some a threshold value, below which SMS will not be sent but above which it will be sent? P.S. there are currently 3 thalis who have next_install value of 100 including me

c) the points (a) and (b) may sound trivial and special cases but they cease to be that once our system automatically converts the transport label from "required" to "pickup". Yes, MM did mention it that once we have all the defaulters, we(the system) should automatically convert their transport status

2) 3) Ok we wil fix the dates (we will hardcode the "pre-duration" like 30 days, 7 days, 1 day, etc. but actual dates will be auto generated from the miqaat dates)

4) i have mixed feelings for this one. sms.myn2p.com does a perfect job of tracking who received SMS for us, so why should we do that? on the other hand, the system is not very "thali" friendly so to speak, since the admin has to map the mobile numbers with thali to find the defaulters. also, we need to invest time in this SMS tracking and also remove the databases once the miqaat is done and people have all paid the hub so as to not clutter the MySQL database

5) we are talking about hostinger here on free account, there is nothing stopping us from imagining ;P

6) Please refer http://www.hostinger.in/knowledge-base/50

7) great!

Possible solutions; 1) ignore that. it just meant that whatever existing code we have for the SMS system, it won't work and we have to code it from the scratch. No big deal!

2) i don't think pipe lining cron jobs will be a viable solution. it is not easy to send 300 or so SMS through cron job because it must be "light" as given here http://www.hostinger.in/knowledge-base/50 so that is why i am stressing on the bulk SMS api (using this api, one can send batch sms by just posting to a single url), the sooner i test it, the more problems will get sorted out.

anyways, here is my unimplemented model so far based on the above discussion

0) dates_creator.php some magic script which takes 8 dates and automatically creates the table dates below

1) table - dates date - the date to send SMS on { (3 pre-durations + 1 transport off) x 8 miqaats } = 32 dates) type - the type of sms to send - 1, 2, 3, 4 corresponding to "you have a month remaining" or "you have a week" or "you have a day" or "Your transport has been changed to pickup" respectively

2) table - sms type - the same type above (namely 1 for month, 2 for week, 3 for a day remaining and 4 for transport stop) body - the sms body like "Salaam , your hub for next miqaat is due on "

3) send_sms.cron a cron job which will run every day and check whether it is time to send sms or not by getting information from dates database, if yes, run the send_sms.php script.

4) script send_sms.php (1) - confusions (1) fetch all the records to which the SMS has to be sent (2) - confusions (7) get the body of the SMS from sms table and replace , etc with actual name (3) - possible solutions (3) call the Bulk SMS API to send SMS (4) - confusions (4) & (5) notify the admin - send email perhaps? (5) - requirement given by MM if type==4 then automatically update the database changing transport from yes to pickup this should be reflected in CR/NR email that goes at 7:30 pm everyday

[yusuf4u52]

Dont do 5, lets discuss this again

[murtraja]

so the final (as of today, atleast) implementation for this is as follows:

1. Table sms_date(id,date,description) containing details of miqat

2. Table sms_template(severity,template,fired_before_days,description) containing when (fired_before_days) to send what message (template)

3. a script runs which does the following:

• retrieves next miqat date
• determines if we want to fire sms today
• fetch the records who will receive the sms
• make an XML tree from those
• send it via XML API

[murtraja]

Fixed by #276