Panoptes-to-CartoDB CSV Data Sync

[shaunanoordin]

Required functionality: Panoptes-to-CartoDB CSV Data Sync

• A system must be built that (1) allows the export of the following data into a CSV file format...
  • Wildcam Gorongosa Subjects
  • WG Classifications
  • WG Aggregated Data (if possible)
• (2) gets said data pushed into/pulled from a CartoDB SQL database, and...
• (3) does so on a regular (say, once per 24 hours) basis.
• Related to: #1 Map Explorer.

Notes:

• This has been moved from the Beta development to Alpha.
• Previously, the SQL data in the CartoDB was, for the Alpha, supposed to be static.
• However, this meant that any new classifications made by the testers during the Alpha would not be reflected in the Map Explorer. We've re-prioritised this 'I Want To See My/My Group's Classifications' feature into a major feature for the Alpha.

Technical:

• The data stored in CartoDB must provide sufficient information for the Map Explorer to:
  • Show camera data and the camera snapshot/photo of a subject/classification.
  • Filter by species, habitat, season and date-range. (Specified by @aliburchard for Alpha)
  • Filter by Zooniverse User ID and WGE Group ID
  • Assumption: When a user submits a classification in Wildcam Gorongosa (Vanilla Version), if she/he belongs to a WGE group, her/his Group ID will be stored in the classification. To my understanding, @marten is working on this. (Edit: derp, it's issue #27 )
• General breakdown of this feature is as follows:
  • Task 1: Create the CSV data exports for Subjects/Classifications/Aggregations. This is a standard Panoptes feature, (go into PFE and jump into the admin of a project you own to see this in action,) although generating the files may take a while.
  • Task 2: The CSV must be re-formatted into something SQL-friendly; currently, the format taken raw from Panoptes stores the important metadata as a string containing a JSON object. I understand that @marten and/or @aliburchard already have a script for this task. (In Python?)
  • Task 3: The reformatted CSV needs to be pulled from/pushed to the CartoDB SQL database. Investigations need to be made into the necessary end points, but talking to the CartoDB team may be a good place to start.
• Note: Marten helpfully pointed out that the automation of this task will likely be inserted into the Panoptes (Ruby) Client - https://github.com/zooniverse/panoptes-client.rb . I'll need to find time to poke my nose into this.

Business:

• We may need to make the most of our partnership with the CartoDB team to get the data sync functionality working. @aliburchard will be in the best position to communicate with the CartoDB team, provided she gets the technical specs from @shaunanoordin.

ISSUES:

• Unknown Time Estimate: currently, the main concern isn't so much 'what' we need to do or 'how' we're going to do this, but by 'when' we can get everything done. I'm afraid I can't give an estimate for this feature right now.

[aliburchard]

@shaunanoordin this is very thorough - thanks!

Quick comment: As much as I would love to have the data exports/imports wired appropriately for the alpha, the more I think about it, the more I'm okay with just using a static dataset that includes fake/generated "classroom" IDs for display. Even if we can't actually see the classifications we've really just made, we can still test out the "me/my class/the world" functionality with this generated dataset.

I'm envisioning just using the classification dump in CSV format and adding a column for some fake classroom IDs. That buys us some time to figure out the best format for exporting the data from Panoptes and wiring up the Carto import process. I can send that csv with fake column to you if you don't have it on hand.

Does that sound okay?

[marten]

That sounds pretty good actually. I guess it's a matter of defining what the goal for Alpha is: is it to test whether this is the right set of tools (can users do what they want with this, even though half of it might not be wired up yet), or is it supposed to be a very buggy version of the final product. Sounds like you're leaning towards the first, which is seems sensible.

[aliburchard]

I like the way you separate those two goals. And yes, I agree that the goal for alpha is to test whether this is the right set of tools, even if all the underlying pieces aren't wired up.

[shaunanoordin]

Thanks for this feedback! Marten, you communicated it really clearly - Alpha is for testing the concept & ideas with the users, not create a "final product but with bugs". Ali, noted on your views on the 'fixed' data. I'll strive to get at least a basic data sync going on by the Alpha deadline - because it'd be cool to have it - but we'll have the static data as a fallback should the other features (i.e. the interface) require more time.