search

CodeforNepal / nepalmap_app

An application that maps census and other official data for Nepal to make data more easily accessible and understandable to the public. Want to help us? Check out the Wiki.
https://nepalmap.org
MIT License
64 stars 42 forks source link

create FB bot that will answer qs about Nepal data #118

Open ravinepal opened 7 years ago

ravinepal commented 7 years ago

is it possible that we program a fb bot in a way that would respond to user's data related questions if there's info on NepalMap?

if it's useful here's white house fb chatbot: https://github.com/WhiteHouse/fb_messenger_bot

prashishh commented 7 years ago

This is a great idea @ravinepal. I will do some research on this.

Love it.

ravinepal commented 7 years ago

@prashishh -- i wanted to see if you got chance to look into this or ask other c4n members to do so

ravinepal commented 7 years ago

@prashishh it'd be great have a team working on this? We can give them access to our FB page and they can start working on this :)

ravinepal commented 7 years ago

I think the MVP of FB bot could just simply direct people to the link of certain district, VDC or municipality.

for example, when someone asks how many people have education in Kavre. Bot could simply respond by saying "Glad you are interested in Kavre. Here's the info: http://www.nepalmap.org/profiles/district-24-kavrepalanchok/"

If someone asks about a neighborhood and we don't have that location in our database, FB bot could respond by saying:

"Hmm can you tell me which VDC, Municipality or district, your location is in, and I can find related data for you."

I believe this FB bot could be very useful especially to students, and reporters.

Looping @studenton and @cliftonmcintosh in the conversation for their thoughts, and see if they know someone who would be interested in doing this!

mesaugat commented 7 years ago

I have a suggestion. If you don't want to code all of this there's a better solution. https://chatfuel.com - Build a Facebook bot without coding.

Get your bot up and running in a few minutes, anyone can do it.

ksodari commented 7 years ago

ofcourse its possible way to do... we can train our chatbot ourselves on how it will respond to our users when they input certain keywords.

prashishh commented 7 years ago

@mesaugat @ksodari - Thanks for the heads up.

I went through ChatFuel and it was fairly easy to set up a basic chatbot. I think it is great for a tree-based conversation for Code for Nepal, but I'm curious if you have worked on integrating this with a datastore? There is an integration with Google Sheets or a JSON API. They can act like a data store if someone searches for the term - tell me about "a district" / what is the population of "a district" (advanced).

The census data are stored in the form of csv in this repo (https://github.com/Code4Nepal/data/tree/master/datasets/census). Do you think we can save them in an easy-to-deploy-and-test datastore to fetch these information for the chatbot?

mesaugat commented 7 years ago

Not worked with the JSON API in Chatfuel but just had a quick look. Only having the dataset won't do us any good. We'll need to create an API of our our own for the dataset so that we can query our API when someone asks about the census for a district.

For Example:

If someone asks: What is the census of Siraha? We could setup Chatfuel to query our API: http://api.ourserver/census/siraha or /census?district='siraha' which could return a JSON response like:

{
  "district": "Siraha",
  "population": 637328,
  "year": 2011
}

and return a text to the user.

Bot: The census of Siraha as of 2011 is 637328

We could change this JSON response as Chatfuel wants it to be, mentioned here. Even if we were creating our own bot we would be doing something similar.

Hopefully, this helps.

ravinepal commented 7 years ago

thanks, guys! let me bring @cliftonmcintosh to the convo as he is the expert on NepalMap and can perhaps guide us on how to efficiently create an API for NepalMap.

cliftonmcintosh commented 7 years ago

I'm not sure what it would take to expose an API. I glanced through the Wazimap code and don't see anything, but Census Report, on which Wazimap is based, has an api: https://github.com/censusreporter/census-api

I'm not sure how it is used since both Census Reporter and Wazimap seem to deliver templated HTML views for each request. A JSON API is likely not something that comes with our framework.

vksbhandary commented 7 years ago

Hello everyone. I have been following this issue. And i guess chatfuel isn't the best option available given it has free usage limit. I would like to suggest another option API.ai. I don't have much working experience with this, but i have created very simple bot using this with in few minutes. (Its name is Open data nepal I have hard coded to give population of Kathmandu only, without any coding! )

The main advantage of api.ai here are

One thing to note here. For any chatbot framework to show dynamic response, we will need api (or data access code to feed to bot). As mentioned by @mesaugat Here is a sample webhook implementation in python. Something similar can be used in our situation for dynamic replies.

I don't know anything about Wazimap or Census report. But I guess, we can add REST APIs to django using REST framework. Technically its Django right? @cliftonmcintosh

mesaugat commented 7 years ago

The problem is not Chatfuel. It's free as well unless you hit too many users. 

Chatfuel’s bot-building platform is free. 

If you plan on hitting more than 500,000 monthly active users, please talk to us at premium@chatfuel.com.

Yes, https://api.ai/ is great. It has a lot of integrations. But the ease of Chatfuel is big, anyone can configure anything. No wonder it's getting popular. You don't need to deploy a bot unlike api.ai and other services. If we are planning for custom use cases then we can definitely go for api.ai. That being said we can choose any service.

The pain point here is we don't have an API to query the census data. From what I see we already have various census data's in CSV format. We just need to feed that data into a database and create a fairly simple API which should not take time.

The census API doesn't not even need to be in this repository. It could be a simple app in any language deployed somewhere and we just query from it.

UPDATE:

As @ravinepal mentioned above: 

For example, when someone asks how many people have education in Kavre.
Bot could simply respond by saying "Glad you are interested in Kavre. Here's the info: http://www.nepalmap.org/profiles/district-24-kavrepalanchok/"

If someone asks about a neighborhood and we don't have that location in our database, FB bot could respond by saying:

"Hmm can you tell me which VDC, Municipality or district, your location is in, and I can find related data for you."

I think we don't even need an API for doing something like this. Just map a location name with their corresponding Nepal Map URL in Chatfuel. It should be fairly simple? @prashishh

That means: 'Kathmandu' will be mapped as http://www.nepalmap.org/profiles/district-27-kathmandu/ in Chatfuel. I think this is possible.

If someone asks: Census of kathmandu? The bot can simply reply with a text card: Here you go: http://www.nepalmap.org/profiles/district-27-kathmandu/

That's seems to be what we need, no?

amitness commented 7 years ago

I explored how Wazimap library is developed internally in venv/lib/python2.7/site-packages/wazimap/views.py and found that Wazimap already implements the census-reporter API and gives a endpoint. So I can access the projected population of Kathmandu for 2031 as

http://nepalmap.org/api/1.0/data/show/latest?table_ids=POPULATION_PROJECTION_2031&geo_ids=district-27,country-NP

It gives back a JSON response. 

{
   "release":{
      "name":"National Population and Housing Census 2011",
      "years":"2011"
   },
   "tables":{
      "POPULATION_PROJECTION_2031":{
         "title":"Projected population in 2031",
         "universe":"Population",
         "denominator_column_id":"total",
         "table_id":"POPULATION_PROJECTION_2031",
         "stat_type":"number",
         "columns":{
            "total":{
               "indent":0,
               "name":"Total"
            },
            "female":{
               "indent":1,
               "name":"Female"
            },
            "male":{
               "indent":1,
               "name":"Male"
            }
         }
      }
   },
   "data":{
      "district-27":{
         "POPULATION_PROJECTION_2031":{
            "estimate":{
               "total":2729056,
               "male":1469787,
               "female":1259269
            },
            "error":{
               "total":0,
               "male":0,
               "female":0
            }
         }
      },
      "country-NP":{
         "POPULATION_PROJECTION_2031":{
            "estimate":{
               "total":33597033,
               "male":16313189,
               "female":17283844
            },
            "error":{
               "total":0,
               "male":0,
               "female":0
            }
         }
      }
   },
   "geography":{
      "district-27":{
         "geo_code":"27",
         "child_level":"vdc",
         "short_name":"Kathmandu",
         "name":"Kathmandu",
         "full_name":"Kathmandu",
         "parent_geoid":"country-NP",
         "full_geoid":"district-27",
         "square_kms":null,
         "geo_level":"district"
      },
      "country-NP":{
         "geo_code":"NP",
         "child_level":"district",
         "short_name":"Nepal",
         "name":"Nepal",
         "full_name":"Nepal",
         "parent_geoid":null,
         "full_geoid":"country-NP",
         "square_kms":null,
         "geo_level":"country"
      }
   }
}
cliftonmcintosh commented 7 years ago

That's an interesting find, @studenton

ravinepal commented 7 years ago

Thank you, @mesaugat, @vksbhandary, @cliftonmcintosh. and special thanks to @studenton for that find :) We have a prototype of a bot that @ksodari and I created using Chatfuel - let me know if you guys would be interested in accessing it or contributing to it. If you just want to test it, type "Code for Nepal" in your FB messenger and let us know what you think

@ksodari - can we use text card to map URLs as @mesaugat is suggesting?

mesaugat commented 7 years ago

Great!

Maybe not. Text cards only seem to support texts. There's a gallery card that definitely supports a URL.

ksodari commented 7 years ago

@ravinepal yes Sir! Gallery card can be used very efficiently on this as mentioned by @mesaugat . Thank you all for the findings and suggestions. It should be now more smooth in upgrading the chat bot "Shanti" :) I am trying onto this. @cliftonmcintosh I would like to know about how data are maintained on our website Nepalmap??

ravinepal commented 7 years ago

hi @cliftonmcintosh - grateful if you could kindly advise @ksodari. pls see his comment https://github.com/Code4Nepal/nepalmap_app/issues/118#issuecomment-296749520

cliftonmcintosh commented 7 years ago

@ksodari

Apologies for missing your question. Data in NepalMap is kept in a postgres database on the same server as the webapp. The database is not exposed externally.

When new data points are added, tables are created and data is updated via SQL statements.

The SQL statements are here.

Does that provide you with the information you need?

ravinepal commented 7 years ago

thanks, @cliftonmcintosh. @ksodari does that help? @amitness - would you have time to help @ksodari build API?

amitness commented 7 years ago

Sure @ravinepal @ksodari . How are you guys coordinating on this and what's the current status?

ravinepal commented 7 years ago

thanks, @amitness! So we have a prototype "Code for Nepal" bot - named as "Shanti". You can check it out on your fb messenger. i think we need to create API or figure out a way to use gallery in chatfuel card that definitely supports a URL. @ksodari - can you pls provide more updated to Amit

ravinepal commented 6 years ago

hi everyone, it'd be great to restart this project. @ksodari - happy to chat if you are still interested.

wizofe commented 6 years ago

hi @ravinepal. would you provide me on more info on what enhancements you need for your bot? i can work with AI and Markov models, if that's on your scope.

ravinepal commented 6 years ago

thanks a lot, @wizofe! We have a prototype of a bot that we created using Chatfuel. If you just want to test it, type "Code for Nepal" in your FB messenger and let us know what you think. Or test via our page by sending us a message: https://www.facebook.com/codefornepal/

Basically, we are trying to make a bot respond to users' queries about data on Nepal. We have tons of Nepal data here: https://nepalmap.org

I think the MVP of FB bot could just simply direct people to the link of certain district, VDC or municipality on https://nepalmap.org

for example, when someone asks how many people have education in Kavre. Bot could simply respond by saying "Glad you are interested in Kavre. Here's the info: http://www.nepalmap.org/profiles/district-24-kavrepalanchok/"

If someone asks about a neighborhood and we don't have that location in our database, FB bot could respond by saying:

"Hmm can you tell me which VDC, Municipality or district, your location is in, and I can find related data for you."

I believe this FB bot could be very useful especially to students, and reporters.

Does this help?

I'm also open to other ideas!

rajsanjib commented 6 years ago

You can easily create AIML structures to best match user queries and then respond with the data you have instead of providing a link. Bot would repond to query like "What is the population of Rasuwa?" with "The population of Rasuwa district is 43,300". PandoraBot uses AIML to create chatbots.

I built a similar bot for Kathmandu University. https://github.com/rajsanjib/KU-Bot It would mainly provide answers to dates of admission, entrance, course syllabus etc.

ravinepal commented 6 years ago

@aishraj and @spaudel -- flagging this to you. grateful if you someone from Bay area could help with this if possible!

aishraj commented 6 years ago

@ravinepal Sure! Currently most of us here in the Bay Area have been working around the Akshar digital literature portal. But I remember a few folks expired desires work on this in the past. I'll double check if anyone is still interested

harsh2ai commented 5 years ago

well i think creating a chatbot is a nice idea but n facebook, they do have a platform called chatfuel which is intended for creation of chatbots and linking the to a particular page on facebook. fb does charge on that but trust me its worth it because facebook would itself want to improve in every possible day, it is the biggest step for creating chatbots without any coding