Developer Documentation and Migration to New Server

[allella]

@Jonessg1Wofford @bikeoid @meenasrik @jonahmoore17 @ (I don't know Killian's handle). If the conversation is of interest then please subscribe to this issue for future updates.

I'll share a cleaned up version of notes from our hangout with @Nunie123 and we can take it from there.

[allella]

@Jonessg1Wofford @bikeoid @meenasrik @jonahmoore17 @Nunie123 @pax-man Here are notes from last Tuesday. Please post any additional notes or corrections.

If you're interested in helping co-maintain this project then please connect on this thread. If you're not interested then you should be able to unsubscribe using the button on GitHub.

Hosting

• Currently hosted on Heroku because it's free. It was on AWS Ubuntu using the deploy_notes.md
• Heroku flow - login to Heroku and exectue a git pull, which sources from GitHub
• See heroku-deploy.md for deploy notes

General Application Notes

• Uses Flask API endpoint for the API routing
• app.py - The primary application file. This gets a JSON of all the meetup tech organizations that used Meetup.com or Eventbrite's API and lookups up the event service and API key for that service a syndicates the info as a JSON endpoint
• By default it displays only future meetings, but it could be changed to allow past meetups
• There's some basic filtering of the events by dates and organizations tags.
• The tags are defined in the data.openupstate.org Drupal CMS. For instance, tag #1 is for OpenWorks and is used to filter only events hosted by OpenWorks, ex https://greenville-cal-service.herokuapp.com/api/gtc?tags=1
• get_group_lists() gets the org API data from data.openupstate.org. It uses Python "List Comprehension" to do some magic on all_sources to filter out to only the values of field_event_service and then groups things into a dictionary of related "event services"
• There's no logging

Meetup.com and Eventbrite APIs

• One of the event services (Meetup.com or Eventbrite) doesn't, by default, do past events
• You can give the Meetup.com API all the group IDs and it will return all the info in bulk
• The Eventbrite API will only do one response per group ID so it's necessary to make requests for each group.
• With Eventbrite you also have to lookup the name of the group
• Both have a throttle / ban limit so you can't make too many calls, which would only be likely to happen in testing or by mistake
• update_cal_data.py actually does the writing to a file
• There's a basic set of unit tests. Run python tests.py

Environment / Configuration

• config.ini is not committed and should not be because it contains sensitive information
• config.ini.example is a template of the environment variables which should be set in each environments config.ini

Deployment

• Right now update_cal_data.py runs via a cron job that runs on Ed's computer. It passes in Linux environment variables to make Heroku happy since Heroku you can't write files to Heroku
• This process creates an all_meetings.json as a local file and Ed's cron job does some magic to get this to Heroku.

Setting Up a Development Environment

• Clone repo the Git repository
• Install Mini Conda 3 globally (a more popular virutal environment tool is Virtualenv, or something similar)
• The virtual environment will handle installing the correct Python version and all dependencies
• Once the virtual environment is functioning just start an interactive shell, like the following -- Go to a directory with a environment.yml -- conda env create -- source activate cal_service (which reads the environment.yml) -- from app import get_group_lists -- app.get_group_lists()
• To generate a new all_meetings.json run
• source activate cal_service && python update_cal_data.py
• conda env update (can be run to update the virtual environment with any new or changed dependencies)

Next Steps (Jim will create Issues as Needed)

• Get credentials from Ed (Heroku, Meetup.com auth key, Eventbrite.com auth key)
• Setup a cron job if we're going to continue using the current all_meetings.json method
• Setup or get API keys for Eventbrite and Meetup.com
• Get dev environments setup for the developers who are going to help maintain the project
• delete the print statement on like 229
• POST requests in the route could be deleted
• If we switched to using another hosting service it be smart to write the all_meetings.json to an S3 bucket (or similar) or else start using a database.
• remove python.crontab from the environments.yaml

[allella]

With initial pointers from Derek Wright on the HG Slack channel, parts of the documentation, notes from this thread, and figuring some things on my own, it's working and now migrated. @Nunie123 you can terminate the Heroku instance.

This is hosted on the same server as data.openupstate.org and the new API endpoint is https://events.openupstate.org/api/gtc

Initial setup notes are now a bit more complete regarding running a gunicorn localhost WSGI server.

The API consuming sites I'm aware of are working again.

• https://hackgreenville.com/events
• https://hackgreenville.com/orgs
• https://joinopenworks.com/dashboard/meetups.php

I ended up setting up an env with Miniconda, setup gunicorn as a systemd using a UNIX domain socket (UDS) instead of TCP, then used Apache's ProxyPass to pipe the /api traffic to the gunicorn daemon socket file, and then had to do so damn SELinux magic to get all the context and policies right to allow Apache to talk to the socket.

UDS is always a pain in the but and triggers SELinux permissions and policy eye-stabbing, but it's theoretically faster than a TCP proxy and perhaps more secure by limiting access to a Linux user/group, which would otherwise be open to any user on the localhost port.

I'll possibly write up Apache config stuff since there's a lot less Apache + Python docs on the web.

[allella]

@MarkMcDaniels @bcsnipes @meenasrik and anybody else, there are more specific notes for setting up Python development environment, via Miniconda, in the deploy README file https://github.com/codeforgreenville/upstate_tech_cal_service/blob/master/deploy_notes_initial.md