Automate importing of Disqus commenters data into sendy

[sumeetpareek]

@tkuldeep

We need to automate the process of moving data collected over disqus comments to the sendy mailing server.

• Here is our mailing system = http://messenger.aamaadmiparty.org/
• I have shared admin login credentials with you on the mail
• If you goto a particular brand (say Aam Aadmi Party) > then goto lists > then inside a particular list > then import subscribers > that is where you can add in users to a list
• This file has the code that tells you what happens when you take a 'Add subscribers' action on a list.
• This file runs as a cron job every minute to bring in data from csv file to the database.
• I will document in a wiki later on how to import users by directly uploading the CSV files to the uploads/csvs directory of the sendy server, but here is the gist
  • Files should be named N-M[anything], where N and M are userid and listid respectively. Eg = 1-13-xyz --or-- 1-10.csv
  • Files should not have more than 5k to 10k records otherwise sendy hangs up pretty bad. So if you have say 100k data items in csv file, using the split command on it is a good idea
  • Files should be owned by www-data user and group and have the 777 permission
  • The cron script actually deletes the files after processing them
  • The list that currently has the disqus data has the ID = 16, Name = Website Disqus Commenters

I guess now you have all the data you need to write up a bash script or a php script to automate moving data from disqus to sendy.

[tkuldeep]

What I understood:

• I have to create a script, that will create csv file from xml data, exported from disqus comment.
• File should be created in the directory uploads/csvs.

[tkuldeep]

I have create a script https://gist.github.com/tkuldeep/6dc8b25b6c288844151a which will work as follows.

• We have to send request to get disqus comments from disqus dashboard.
• Now we will have a file containing all disqus comment in xml format.
• To parse this xml data, run this script by issuing command _sudo php parse_disqusnew.php [path of xml data file]
• It will generate csv files in directory /uploads/csvs/ with needed permission.
• And also it will create a new file [must required] name past_imported_data.txt, which will keep the track of previously imported data.
  • In this file I have declared two variable list_id to keep track of list id.
  • disqus_last_location to keep track of users which have already created.
  • According to me When we download disqus comment, every time, we will get same file , with some extra
    comment if users have given or new users have registered.
  • So when we first time run this script will parse the xml file, it will save the pointer of EOF of xml file in
    disqus_last_location variable in past_imorted_data.txt file.
  • So next time this script will automatically parse the content form next to this pointer.
  • Thus our script will always try to parse fresh content form disqus exported comment.
• Code can be seen in the given link. [Above]