proper zotero oauth

[rlskoeser]

I still think django-social-auth is the right way to move forward with this, but after hacking at the backend a little more I don't think I'm much closer.

Assuming we can get django-social-auth to work, it will give us nice django integration with login/logout functionality, storing tokens in the database, etc. I think we might have to rethink how the zotero part of the site works a bit, but in the end it should provide us with nicer functionality.

@erose do you think you can take another crack at this? I'm perfectly willing to help investigate and troubleshoot. I've been looking at the social-auth backend & views code some today, but I don't think I quite understand how the zotero oauth flow is supposed to work so I'm not certain where it's failing.

I think we might want to ask for help on the django-social-auth forum, here (I see other people have asked for help with setting up new backends and have gotten replies): https://groups.google.com/forum/#!forum/django-social-auth

[erose]

Yeah, I'd like to try doing this again. I'll think I'll have the chance later this week. Thanks for the forum link! I'll let you know when I inevitably will need help investigating & troubleshooting.

[rlskoeser]

I went ahead and submitted a question to the forum. Maybe we'll have some feedback by the time we get back to looking at this.

[rlskoeser]

Here's the thread where I asked the question and got an initial response asking I put up the code as a gist--

https://groups.google.com/forum/#!topic/django-social-auth/wv6Akky4yus

[mialondon]

Quick question re the roadmap for the immediate next phase - do you think you'll have time to include this or should we push it back to the wishlist?

[rlskoeser]

I feel like I'm making progress but it's hard to tell how close I am. Just posted an update & question to the django-social-auth group.

I think this is kind of important, because the way we're using zotero is a little bit broken now - if people use serendipomatic with zotero multiple times, we're generating a brand new key every time, which is really not ideal.

Are the google analytics set up in a way that lets us tell how much people are using the zotero part? That might be helpful to get a sense of priority...

It's also worth noting that if we can get proper zotero oauth to work, we'll have to rethink some of the UI and workflow aspects around zotero interaction. We should be able to make things nicer (e.g., allow users to select a particular collection of items), but it will take some thought and perhaps some experimenting. Although, now that I think about it, we might not have to tackle that right away. Maybe we could just start with something close to the existing zotero functionality backed by the new oauth, and then refine later on.

Also, users will effectively be logging in to our site using their zotero credentials - so we'll also need to update the site design to be able to indicate that they're logged in and give them the option of logging out.

[mialondon]

re Analytics - not specifically, though we could tag the Zotero go button https://support.google.com/analytics/answer/1033068?hl=en

[scottkleinman]

I haven't been able to follow a lot of this Zotero oauth discussion, but I'm starting to suspect that the "Save to Zotero" functionality issue should be merged with it. I think the reason the bookmarklet code tries to use Zotero standalone has to do with account access. I tried digging into the Javascript, but it's layers and layers of minimised code, which would take me ages to puzzle out. Using oauth would be much easier. Plus, it appears that the COinS translator does not work for all data types, so we might want the greater flexibility of the API.

On 13 August 2013 07:45, Rebecca Sutton Koeser notifications@github.comwrote:

I feel like I'm making progress but it's hard to tell how close I am. Just posted an update & question to the django-social-auth group.

I think this is kind of important, because the way we're using zotero is a little bit broken now - if people use serendipomatic with zotero multiple times, we're generating a brand new key every time, which is really not ideal.

Are the google analytics set up in a way that lets us tell how much people are using the zotero part? That might be helpful to get a sense of priority...

It's also worth noting that if we can get proper zotero oauth to work, we'll have to rethink some of the UI and workflow aspects around zotero interaction. We should be able to make things nicer (e.g., allow users to select a particular collection of items), but it will take some thought and perhaps some experimenting. Although, now that I think about it, we might not have to tackle that right away. Maybe we could just start with something close to the existing zotero functionality backed by the new oauth, and then refine later on.

Also, users will effectively be logging in to our site using their zotero credentials - so we'll also need to update the site design to be able to indicate that they're logged in and give them the option of logging out.

— Reply to this email directly or view it on GitHubhttps://github.com/chnm/serendipomatic/issues/111#issuecomment-22569919 .

Scott Kleinman Professor of English Director, Center for the Digital Humanities California State University, Northridge

[mialondon]

I've started an wiki page to discuss our overall architecture to save us trying to keep track of it across different Issue threads - pop over to https://github.com/chnm/serendipomatic/wiki/Serendipomatic-architecture where we can work on these issues...

[scottkleinman]

@mia_out: Like.

On 13 August 2013 08:44, Mia notifications@github.com wrote:

I've started an wiki page to discuss our overall architecture to save us trying to keep track of it across different Issue threads - pop over to https://github.com/chnm/serendipomatic/wiki/Serendipomatic-architecturewhere we can work on these issues...

— Reply to this email directly or view it on GitHubhttps://github.com/chnm/serendipomatic/issues/111#issuecomment-22574429 .

Scott Kleinman Professor of English Director, Center for the Digital Humanities California State University, Northridge

[rlskoeser]

Progress! We have a working zotero backend for django-social-auth, and I've even set it up so we can easily configure which api permissions we want when requesting access - https://groups.google.com/forum/#!searchin/django-social-auth/zotero/django-social-auth/wv6Akky4yus/SclxYFwi5RAJ for anyone who wants the details.

If we do want to add items to zotero via the api rather than using COinS we'll need write access, but I don't think we should ask for it if we're not going to use it.

However it seems that if we use zotero oauth for login and people logout (e.g. access on a public machine) and then log in to zotero again, we're still going to be minting a new api key every time, and the django-social-auth developer who's been helping me doesn't see a way around this. It seems to be something to do with the zotero oauth implementation, so we might ask the zotero devs if they're willing to consider updating their implementation, but that seems likely to be a longer-term question. @erose didn't you have some conversations with the Zotero guys about their oauth implementation during OWOT? I vaguely recall they were willing to look into implement something we thought was missing, but I don't remember what.

What are people's thoughts? It seems kind of troubling to me, but I'm not sure what the best solution is. My only current thought is that we'd have to let people log in with a different account (twitter, google, etc) and then let them associate their zotero credentials with that account, but that seems kind of clunky...

[mialondon]

Nice one! I don't think we should build API methods around saving results, I'm sure we can find a way to make the methods familiar to most people from other sites work.

I wonder how many people would repeatedly be coming back to the site and running their Zotero library through it? Is there a technical or security reason we need to worry that a few people might have two or three old keys floating around?

[rlskoeser]

Good point, @mialondon. Maybe it's not such a big deal since most users wouldn't have too many (I had tons that I had to clean out manually, but that was because of all my testing and investigation). And they ought to see the date the keys were last used if they do notice and want to clean up, so I guess that's not horrible. Not sure if there is any technical/security reason to be concerned about the keys, I can't think of any.

I'd also prefer to use COinS (or something similar) rather than using the zotero api directly, because I expect the latter will be a lot more complicated to write and maintain.

[rlskoeser]

When I get a chance, I'll clean up the code and push a branch to github so others can try out the new zotero auth and we can figure out how to update the workflow. I could probably also deploy a copy to my dev heroku instance (not the auto-deploy) so people who don't have their own development instance running can still try it out - please comment if you think that would be helpful.

[rlskoeser]

Finally had some time to work on zotero integration and get it to the point where other people can try it out.

Code is on proper-zotero-oauth branch if anyone wants to check it out and run it themselves; note you'll have to install new requirements, run syncdb (for the new social-auth db models), and adjust your zotero api key configs in localsettings (see the example for adjusted format). FYI, I switched from libZotero to pyzotero because it has a more pythonic interface and because it actually has documentation - http://pyzotero.readthedocs.org/

I've renamed my personal dev heroku instance and pushed this branch so non-developers can try out the new functionality: http://rsk-serendip.herokuapp.com/

Once you login with zotero, you should be able to choose from your collections, groups, or tags as the source. I think you should stay logged in until you logout (placeholder login info & logout link at top-right), although I don't know how long that lasts.

@briancroxall would you please test? I know you have a large zotero library and presumably have different collections, groups, and tags. I'm hoping you'll have a sense if the selection options that come back are useful, and if the results are doing roughly what you'd hope. (The data I'm pulling from zotero to generate search terms is still provisional; it seems to be different than what we were getting before, but I'm not certain how the fields I'm getting compare to the other fields.) FYI, I updated the permissions for group and note access. (Not sure how people treat notes and whether or not we need to treat them differently.)

@mialondon could you also take a look, or maybe who else might look if you're too busy; I'm sure I haven't thought thru the way the user moves through the site. Probably lots of error handling I haven't done yet either.

Once we get the functionality closer to what we want, there will obviously be design/layout issues to address as well.

[mialondon]

I'll have more time after tomorrow - I wanted to try setting things up on my own Heroku anyway so it's a good excuse to bring it all together. Are the instructions at https://github.com/chnm/serendipomatic/wiki/Heroku-setup still valid?

[rlskoeser]

The heroku instructions should still be accurate, but I think you'll probably be the first person to actually try following them. Please let me know if you find things I left out or that don't make sense. :-)

[mialondon]

The heroku instructions worked! It just took me a bit of time to stop typing unix commands on a dos prompt... Anyway, I'd missed your bit about running it at http://rsk-serendip.herokuapp.com/ so I didn't need to do that anyway (but I'm glad I did).

First things first - it worked! I tried it on a collection, a group and a tag, logged out and logged back in again.

I'm attaching a screenshot of the front page after logging in because I think the UX needs some work - some technical and some is design and copy.

First, if you've logged into Zotero you're probably intending to explore that, so once you're logged in and have returned to the home page it should reflect that. I'm also wondering how we should visually tie the 'you're logged into Zotero' bit on the upper RHS with the 'you're now working in Zotero' mode.

When you're in 'Zotero mode' I think the home page should de-emphasis the 'copy random text' bit on the LHS. We also need to style the Zotero box so it looks like it's in a different ('active') state, and write some copy to let people know what's now going on on the RHS - what are you choosing between and what happens when you choose? (i.e. it may not be obvious that you can run it again and make different choices each time). Also I assume it limits the number of tags it returns? I have a whole bunch of random tags inherited from a shared group somewhere (I think) so I didn't get to see most of the tags I apply. Maybe they could be ordered by descending usage?

As a design concept, do 'general' and 'logged into Zotero' mode have a spatial relationship to either side of the screen, or can Zotero move over to the LHS in the logged in state?

On the results screen, it'd be lovely if the 'Serendip-o-matic ran with these words from your text:' copy could tell me which tag, collection or group was used to generate those terms.

[mialondon]

Can someone check that the original functionality is working, and if so we'll close this call after go live.

Then the potential 'text changes depending on Zotero-logged-in-status' should go in a separate call; as should showing the user which Zotero tag, collection or group was used and any tag selection/sorting fixes.

I've only skimmed the rest of the discussions, so have I missed anything?

[rlskoeser]

This code is still in a separate branch and I think has UI/UX questions that need to be resolved before I can finish it and merge it in to the main development branch. @briancroxall suggested maybe he and I can talk it through (after he has a chance to test the new functionality) and then write up how we think it should work for others to review/comment.

This won't be ready for inclusion in the immediate release we're hoping to push out next week, but it would be great if we could target the early November release.

[mialondon]

Sounds good!

[briancroxall]

The functionality of the Zotero authentication is working fine. @rlskoeser and I spent some time, however, talking about how to best onboard users and help them understand the different interactions. We've got some ideas and will be sharing them more soon.

[rlskoeser]

Diagram from @briancroxall for what we think the basic flow through the site should be, incorporating the new zotero functionality and splitting it out from the current text input. This diagram & these notes don't necessarily have to make complete sense to everyone else - our thought is that we will work on implementing this in the zotero dev branch, and when we get it farther along we will ask other members of the team for help refining it (particularly some of the design/UI questions).

Some other notes from our conversation about these changes:

• add a zotero login / zotero logout button at top right in line with the current home, about, connect links (may require we lose the green to accommodate zotero logo with red - wondering if we can pull the gray we're currently using for the zotero input box?) - this will be the shortest path to zotero login for users who are familiar with the site
• add a line under nav buttons to indicate user is logged in, e.g. "welcome briancroxall"
• remove the current zotero input on the home page and make the text input full width (might allow us to shorten it also)
• under the text input box, we will have a secondary zotero entry point (if not logged in) -> click through takes user to a page explaining how the zotero functionality works
• when user is logged in to zotero, results page will need parallel nav elements to return to text or zotero input forms; we'll need to come up with some language for zotero similar to the "feed the machine more text" (@briancroxall also noted these might need to be larger/more noticeable than the 'go home and feed...' currently is)

I suspect there are a number of other enhancements that we'll want to make for how we're dealing with the zotero library (groups, subgroups, tags, display on results page, etc), but I'd like to get the revised workflow / site organization figured out first.

[mialondon]

Thanks for taking this one, guys! It might be worth putting some mocked up screens on a user-testing site for feedback e.g. do we want an indicator of Zotero functionality visible above the text entry box? People might not see it once they're focused on the 'copy text into the box' path. (Assuming I've interpreted the above correctly)

I'm also wondering if putting the Zotero stuff up in the main nav makes it seem like it's tied into the functionality of the whole site and not just the 'input into the magic box' part? e.g. that you're logged into Serendipomatic rather than Zotero?

[rlskoeser]

Comments from @mialondon:

• suggests keeping the two-thirds column layout for the text box and adapting the current zotero page space to provide additional text / cues / instruction.
• we should look into adding login/source for other systems, like Mendeley - we should make sure login/logout ui/ux allows for multiple systems
• should keep login out of top-level nav because putting it there indicates it is core functionality whereas it's really optional
• wording important also: logout of zotero vs just logout (since user's mental model is very likely not that they've logged into serendip-o-matic)

[rlskoeser]

Idea: might be useful to write a script (could maybe even be a gist snippet or something like that?) that anyone can run to see what information we can pull from zotero to help us refine which parts of the content feed into zotero as input.