search

The-Yak-Collective / yakcollective

The Yak Collective website & related automation.
17 stars 9 forks source link

Sync data (channel memberships, display name, etc.) from Discord to Knack #32

Open necopinus opened 4 years ago

necopinus commented 4 years ago

Use the roles from Jordan’s #subscribe-projects to badge Yak Collective members.

We can probably do all of this with Jekyll, but seem like a more flexible framework like Gatsby might make this easier.

tilnow commented 4 years ago

consider zapier. though i agree some bot work would be needed. this example does not have badges or joining a channel as a trigger. but i assume the bot can convert that into a "message" or even a form submission for netlify (for which we do not need zapier and can just use "functions")

tilnow commented 4 years ago

doing a bit of work on bots earlier today:

  1. if we do it in jekyll or gatsby, it means that it is done at build time, no? so each such change is used to drive a rebuilding of site? or do you mean we build dynamic objects using gatsby?
  2. we can do it in JS running on the page (as a bot)
  3. if we have any type of backend, or a web service (e.g. here), we can run such a robot there and it will collect information from discord and push it to netlify functions, which will update website display.
  4. we need a mapping of user names to discord names. for now it can be a field in member data (until we have some sort of database...). i suggest that moving forward, we create the names automatically at the website and then create them as members of the discord channel (using a bot) and then the invite is to a server they are already registered on. i understand most or all discord users are not using the same handle on other discord servers. if that is an issue, we can maybe flip the registration order: first you join discord and then a bot command can allow you to become a member of the yak website.
necopinus commented 4 years ago

(1) I was thinking that we can do this at build-time. For Jekyll, this can potentially be done using a function that is hit during build and makes a commit to the backend (which then triggers a rebuild). That's inefficient (we'll often have two back-to-back rebuilds). Since Gatsby is a NodeJS app though, we should be able to incorporate this code into the actual build process later on (though I still like the idea of committing to GitHub, as that creates a record of past changes).

(4) Why not just add Discord user names to the appropriate member markdown file in _members? We're not trying to grab everyone on Discord, just those people who we already consider "members" for the purpose of the website.

tilnow commented 4 years ago

re (4) the discord users do not need to know that. when created on discord, they do not have a "member" badge, so they do not appear as members on webpage. what you seem to be suggesting is manual updating everytime a member is added.

for the initial setup, being to lazy to enter 34 handles for names, i will need a robot added to server so i can query the list of members and try to automatically match them up with members. i can write the robot, i just need the invite url. this bot would also be used to grab the roles each member has .

re gatsby and nodejs, it seems like you can add steps to the process in netlify (that is what fauna does), so i am assuming we can add any step we want to netlify build process. we may need to write a plugin for that: https://docs.netlify.com/configure-builds/build-plugins/

vgururao commented 4 years ago

FYI people who’ve earned website level membership are awarded madeyak role on discord. But automated population will be limited because you’ll need additional details.

On Monday, July 20, 2020, tilnow notifications@github.com wrote:

re (4) the discord users do not need to know that. when created on discord, they do not have a "member" badge, so they do not appear as members on webpage. what you seem to be suggesting is manual updating everytime a member is added.

for the initial setup, being to lazy to enter 34 handles for names, i will need a robot added to server so i can query the list of members and try to automatically match them up with members. i can write the robot, i just need the invite url. this bot would also be used to grab the roles each member has .

re gatsby and nodejs, it seems like you can add steps to the process in netlify (that is what fauna does), so i am assuming we can add any step we want to netlify build process. we may need to write a plugin for that: https://docs.netlify.com/configure-builds/build-plugins/

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/The-Yak-Collective/yakcollective/issues/32#issuecomment-661672596, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGMFWZLLXJUNTQGQRHMT4TR4U32BANCNFSM4OXNCVUQ .

tilnow commented 4 years ago

the madeyak role is helpful.

re new onboarding. in my vision, when somebody joins the yak collective the google form (or whatever) collects whatever information we need including a name for discord and generates the discord login for them. i am assuming a discord bot can create a user.

vgururao commented 4 years ago

I don’t think we want to interrupt the initial intake that strongly even if we could. Keep that low friction for people to check us out. There’s like 400+ people on there and we regularly prune people who didn’t post an intro and so aren’t in basic yak role. Otherwise it would already be 1000+. We want to interrupt the first-time project signup workflow for this. Getting on the site should remain the earned outcome for a minority, not the default expectation or experience. So only project and infrastructure contributors, not general scenesters. Think of basic yak role as spectator/NPC. Game starts with madeyak.

On Tuesday, July 21, 2020, tilnow notifications@github.com wrote:

the madeyak role is helpful.

re new onboarding. in my vision, when somebody joins the yak collective the google form (or whatever) collects whatever information we need including a name for discord and generates the discord login for them. i am assuming a discord bot can create a user.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/The-Yak-Collective/yakcollective/issues/32#issuecomment-661682017, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGMFW75X4QZ5X2SXCLF3OLR4U6PXANCNFSM4OXNCVUQ .

tilnow commented 4 years ago

ok. and also discord does not seem to support automatic account creation. so flipping it. when a person is madeyak (or whatever) they get a link (to website) to fill in whatever information we need for website. does that work? i assume the pruning is automatic, right?

necopinus commented 4 years ago

I wonder if we could add a hook in Discord that hits a Netlify function endpoint (say, /.netlify/functions/create-update-member) with basic member data, and that creates some kind of skeleton markdown file in _members (or updates an existing file). Depending on how much is available, we get name, a photo, join date (the date the function is triggered) and roles that we can use for flair this way. We obviously want more, but that would be enough to generate a (very) basic member card.

The admin interface will let people add more information. Maybe we send them a message on Discord?

(For updates, we'd need to pull in the file, rewrite the content, and commit it back. That's more complicated than what we're doing now, but not at all out of the scope of what OctoKit can do.)

tilnow commented 4 years ago

i think right way to do this (also thinking about a possible change to gatsby) is a robot that receives the event, generates a "package" for github (maybe first reading from github) and then pushes it to github. for this i need a robot key. (read privs are enough)

anyway, the subject of the issue seems to have migrated away from title. it now includes:

  1. enter user handles from discord into members and (yes or no?) show them
  2. copy some sort of information from discord either at build time or at periodically to github or to github and trigger a rebuild
  3. information includes: badges? discord name? roles? other?
  4. mirror roles in discord to website, including icons

maybe issue should be renamed

oh. and now is the time to consider if we want to store all this information at, for example, faunaDB (if you do not like central servers and databases) and decide who has priority member.md files or this database or maybe you want live updates back and forth

necopinus commented 4 years ago

I think we should keep using the YAML/markdown files in _members. Alternative would be to move all member data into faunaDB or something. I don't want to split data across two systems though.

For now, let's just start by adding Discord usernames to the _members files, and then pulling in the channels.

I've made a limited access GitHub token available as an environmental variable; it should have enough access to commit. Check out lines 111 -- 121 of functions/create-post.js to see how this (and other GitHub-related environment variables) can be used. (The WEBHOOK_TOKEN used for authentication at the top of that file is just a dummy variable for the dev sites, so for the moment I'd add the check, but keep it commented out during testing).

After a quick search, it doesn't look like Discord can be configured to hit a webhook on a given event, so we'll need to poll it. I think we should still just use a Netlify function, but hitting it should just trigger a poll of Discord to refresh relevant user information. If we have it always commit a file with a last-run timestamp (or potentially even more useful, just a random number), then hitting that hook would guarantee a site rebuild even if there's no updates from Discord. That might be useful for #27.

tilnow commented 4 years ago

the way to do this in discord is to have a "bot" which can get an event anytime anything happens. i do not have the privilege to create a bot on this server.

necopinus commented 4 years ago

Paging @vgururao

vgururao commented 4 years ago

Wait what exactly do you want to do with the discord bot? Pull madeyak role info? I don’t think much else can be inferred automatically. Channel memberships are only a noisy indicator of project memberships.

necopinus commented 4 years ago

I think the idea is that we can either poll Discord for member information (channel memberships, avatars, names, maybe more in the future), or we can use a bot that pushes updates to us whenever there's a relevant change. The latter is more efficient, but only works if we have a something Discord-side that hits a URL whenever X changes about member Y.

But maybe Discord bots can't do this?

tilnow commented 4 years ago

they can (i have been playing a bit with them). discord generates a lot of different events and the robot can listen in on any of them. think of the robot as a program running on your computer, that is logged onto discord and is attentive to channels, servers, members, etc. robots can also execute commands on discord, like sending messages, adding roles, etc.

tilnow commented 4 years ago

we need to decide where project membership is stored. wherever it is, we propagate it to the other tools. so if a project is a "role" on discord, i can read it from there. if it is stored in the website/github, i can get it from there. and if we have a 3rd party DB, then from there.

the madeyak role is just to make it easy to know which members to synchronize between discord and website/database/etc.

necopinus commented 4 years ago

Currently, project membership is the union of the lead_editors and contributors array in the corresponding project file in _projects.

But really, this is just "project contributor", whereas for the purpose of "project membership" flair we want to use "madeyak + project channel membership". So I think we want to store channel information in the _members data files.

There's an open question here about whether we want to collapse both "project member" and "project contributor" into "madeyak + project channel membership". But I don't think we need to answer that here/now.

vgururao commented 4 years ago

Can we move this discussion to #infrastructure on discord, and/or add Jordan Peacock to this git project? He set up the channel bot. I think he’s @hewhocutsdown on github too

On Fri, Jul 24, 2020 at 8:05 AM necopinus notifications@github.com wrote:

Currently, project membership is the union of the lead_editors and contributors array in the corresponding project file in _projects.

But really, this is just "project contributor", whereas for the purpose of "project membership" flair we want to use "madeyak + project channel membership". So I think we want to store channel information in the _members data files.

There's an open question here about whether we want to collapse both "project member" and "project contributor" into "madeyak + project channel membership". But I don't think we need to answer that here/now.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/The-Yak-Collective/yakcollective/issues/32#issuecomment-663584145, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGMFW4LX5KQ6K275BYGXN3R5GPJ3ANCNFSM4OXNCVUQ .

tilnow commented 4 years ago

i think the discussion will be more on the detail level, so here, i think. your mention pages him, no?

hewhocutsdown commented 4 years ago

Yes, I was paged, just catching up on emails now. What's the question to me?

hewhocutsdown commented 4 years ago

Sounds like the question is "where is canonical project membership stored" and at the moment I don't know that it is; certainly not in any robust, automatically-extractable way.

tilnow commented 4 years ago

@hewhocutsdown perhaps the question is more like "where do you want it stored". three choices:

  1. some central or distributed database (like FaunaDB)
  2. in discord (e.g., as roles)
  3. in github data of website (e.g., in yml front data of member cards)

whichever one it is, i woudl want to propagate the data automatically to the others.

tilnow commented 4 years ago

another thought. if we decide on option #3, i suggest moving the member data into a single file, so that the data is more easily available (Jekyll reads files from "_data" directory into its variable namespace automatically)

hewhocutsdown commented 4 years ago

How does this relate, if at all, to Vinay's yak-network-map project? I feel like there are some strong potential overlaps.

tilnow commented 4 years ago

i do not think there is overlap, but rather this is infrastructure for Vinay's project. we do not want the yak information entered and stored in two places. and his map s/w will need to read data from somewhere. (or alternatively, if he collects the data, send it to the website for storage for later use) at the development stage, he will need to be able to access the member data without his map being part of the website. at later stage, we want the map to be part of the website, i assume

hewhocutsdown commented 4 years ago

Given that, options 1 or 3 seem ideal, with other systems being derivative. I'm not familiar with Fauna, is that something you've worked with in the past?

djinna commented 4 years ago

hello, I've been lurking here for about a week -- fwiw: for ~4 years I've run a small membership database with Knack » here's a link to their membership directory template » https://www.knack.com/templates/member-directory » folks are using Integromat, IFFFT, and Zapier to do all kinds of scraping for useful cross platform stuff. Our members get a login to manage their own profiles which seems pretty important for YC too. I searched high and low for something off the shelf, stable, secure, before settling on Knack. Will be really interested to hear if YC infrastructure team find a good way to do this /and/ would be happy to collab on firing a Knack test up for YC. /jenna dixon

tilnow commented 4 years ago

@hewhocutsdown no. i have not really worked with fauna in the past, but i got the feeling that YC did not want to use a backend server, nor do they like using javascript. i personally prefer a centralized sql-type database, hosted on some server. depending on volume, etc. i use pythonanywhere. i assume it could be ported to azure, etc. fauna has an integration with netlify, though.

tilnow commented 4 years ago

@djinna i think that knack's pricing (and overall balance) does not seem to quite match our needs. starter ($39) has up to 20k records - i do not think we need so many, but api only gives you 1k calls a day and only 10 per second. we would want to use the api to build a page and also to allow browsing, etc. (think yak-map). more difficult, matching up discord data and knack data is sure to use up some of these requests. maybe i am being too harsh though - they allow multiple records (up to 1000) in one call. still, i am worried using knack will require a more complex system.

are you aware of solutions that have a different balance - maybe less user interface for browsing and more api support?

let me take a step back. i think we need member data in an accessible place for the software to coordinate web site, various tools (like yak-map). my personal belief is that if browsing members is an important function, i want it to be mediated through the website, in order to provide the website with a critical volume of daily attention by yak-members.

these opinions and thoughts are not strongly held... and i am happy to try out a test-run. the "easy" part is porting the member data. the slightly less easy part is retrieving that data for use while building the web site. the interface with discord does not exist yet in any case. maybe we can ignore it. so, say "when".

tilnow commented 4 years ago

@hewhocutsdown speaking of discord. i want to build a robot that synchronizes role data from discord to {wherever we store the data}. i need permission for such a robot - read only, for now. or maybe, since you prefer #1 or #3, i actually need the ability to add roles too.

tilnow commented 4 years ago

@djinna thinking a bit more about knack. a critical (?) feature is that we have member data which the members cannot change. (maybe we also need data the members cannot see, but i am not sure yet). does knack support this?

djinna commented 4 years ago

@tilnow thanks for taking a look, here's a couple of replies/thoughts. sorry if tmi :}

~ The Yak Map project says they started by scraping info from #introductions which is fine to start but generally problematic, I know I didn't try to include anything close to a skills+interests map. One way or the other it seems we'll want/need to create a way for folks to create and maintain their own information? Also perhaps with some ability to dial up and down how it's used within YC since our default is working in public?

In our current approach, we are collecting basic data from #introductions channels to extract your name, discord username, and domains/skills that you mentioned in the intro you posted https://docs.google.com/document/d/13ihQ2VdjZbfjWjxeOmCvFl_gt9Y1xuFkolyHdW6yNGY/edit

Again, I'm no shill for Knack other than I do wish them well as the only no-code database tool I found after searching pretty hard about four years ago. Their continued existence would be nice. :) I also don't know too much about the tricky Discord hookups being talked about here and which sound interesting.

Just thinking about how to have usable frontend for Yaks that doesn't take rare-ish skills to maintain. I'll be totally psyched to see another route to this member frontend that you guys have the chops to build out!

In the YC calls I've listened in on, reducing the need for critical skills in our infrastructure was one value that's come up. Any somewhat digital native could help out in the Knack backend, as opposed to the higher bar of helping out on Jekyll Gatsby etc. The API stuff would of course alway need the higher level skillset.

_ fwiw: Has this group looked at Eleventy? https://www.11ty.dev/ Chris Coyier used it for a similar sounding site last year: https://css-tricks.com/a-site-for-front-end-development-conferences-built-with-11ty-on-netlify/

tilnow commented 4 years ago

while studies show otherwise, i do not believe in TMI.

tell me more about how you think we can do "project submittal+voting stuff could be routed that way "

i think once somebody is logged into discord, we can send him a link to fill in (optional + stays available under his DM so he can fill in later) instead of posting an introduction and we can generate that introduction for him. but see the above comment by @vgururao:

I don’t think we want to interrupt the initial intake that strongly even if we could. Keep that low friction for people to check us out. There’s like 400+ people on there and we regularly prune people who didn’t post an intro and so aren’t in basic yak role. Otherwise it would already be 1000+. We want to interrupt the first-time project signup workflow for this. Getting on the site should remain the earned outcome for a minority, not the default expectation or experience. So only project and infrastructure contributors, not general scenesters. Think of basic yak role as spectator/NPC. Game starts with madeyak.

djinna commented 4 years ago

ok then 😬

New Yaks // maybe we keep the current welcome setup per @vgururao note; and add a second welcome step of being invited to be included on the Yak Map (opt-in) after some threshold of activity that Discord could automatically detect? And that's when the profile form link would drop into a new Yak's DM? Transparency: we'd add that info to the New Yak Roam page of course

Projects // as soon as the info needed to kick off a project idea is sufficiently standard to define our standard data fields, Knack forms can do all the usual stuff. Could also be that we leave the initial voting in Discord with the reaction emoji. Would be interesting to compare. Project managers could use an admin UI to pull participants onto a list that was then scrapable by the Yak Map folks. One source of truth kind of thinking.

UI // did I miss an earlier conversation around how we'd give Yaks control of generating and updating their YC profile info?

Nathan @necopinus had mentioned he'd be tied up this week with a conference, so happy to wait until he's around again. Knack also has a free trial so feel free to fire up a test account. I could jump on a screenshare call sometime to for a first walkaround.

The cost does seem prohibitive for YC (?), so it could just be a fast prototyping spot for the team for a month, that then you 🤓 smarter-than-me folks build out with a non-no-code tool. Creating the member profile and project info UI can only be done by getting granular with it, at the right time, and does seem separate from figuring out access to the .md files underlying the website

tilnow commented 4 years ago

re ui. the current webpage allows you to propose an edit to your membership information re projects. there are several contending proposals: via gatsby, via, discord, via github and now via knack. too many choices... i think we need to hone in on the requirements a bit more. i think that is waiting for @vgururao, since the backend side seems to have several viable alternatives. re newyaks - i would like to avoid the roam step. roam (right now) needs to be manually updated. i like manual updating even less than i like fidgety s/w setups. now, the last part of your rant seemed to imply some sort of criticism. hmmm. i would like to suggest that copying is usually smarter than building. but building is sometimes more

FUN

djinna commented 4 years ago

re ui: got it, that handles the public information side of a profile. If I understand the current data setup, it's at root .md files? re projects: yes, firm requirements, then iterating btw what the tools can be made do and back again re Roam: sorry I just meant updating the static New Yak welcome page, The action would be in Discord. re rant? criticism? no rant-ish tone intended. agree that building is fun, I like working in Knack quite a bit when I'm not hitting a wall lol

tilnow commented 4 years ago

current data setup, is at the front matter (yaml-style) portion of md files. if you go to yakcollective.org/admin you will see the interface, such as it is. it only allows you to edit some fields, the others are invisible to users.

do you know if knack works well as an iframe or can be used as an embedded widget? one the one hand i do not want to multiply the infrastructure portions too much. on the other, i also do not want to multiple the front end interfaces too much either -> we already have 3 go-to places: roam, discord and website.

necopinus commented 4 years ago

I want to bring this ticket back around to the original purpose: It is not about proposing a new website back-end, but rather the idea that it would be useful to port some information from Discord into the site. In the original ticket, this was limited to Discord username + (some) channel memberships (which could be used as "badges"), and during subsequent conversation it was expanded to include name, avatar, and (maybe) timezone.

Right now, data and code for the website are stored in GitHub, the site is built using Jekyll, and optimized/served using Netlify. There is an experimental UI that can be used to edit data files in GitHub.

There are two proposals to change this:

If we want to add additional data sources to the website, I'm warming up to the idea... But I think that needs to be proposed as a separate issue (fine for this one to depend on it).

Finally I think I agree with @djinna's reluctance to gather too much information. Right now we provide a member directory + aggregate public blog posts. We're looking at pulling in additional Yak Collective related information in this ticket. But we really need to also be thinking about data minimization here. We should be gathering no more information than is needed (so, for example, I don't think we should pull in anything from Discord from someone who isn't already a "madeyak"); to do otherwise introduces risks to our members that they have not (implicitly) agreed to.

tilnow commented 4 years ago

can i suggest rewording it as follows and propose a scope:

  1. build robot that reads data you mentioned from discord
  2. push such data to knack (if we do knack - @djinna)
  3. push such data to github in the _members folder
  4. when website rebuilt, you see the information
  5. the "robot" can be run at build time, or, better, is run once a day and when it is done, it triggers a netlify rebuild.

if and when we have a backend, this can all be changed

necopinus commented 4 years ago

I agree with this, with the caveat that (3) might take a lot of different forms.

I've created a new issue, #71, to track the creation of some kind of member database. Bumping this issue to the same milestone (v3.0+). We should consider #71 a dependency of this issue; if we're clever, I think we can probably resolve this issue in the process of resolving #71.

tilnow commented 4 years ago

@hewhocutsdown - reminder about that robot. also, do you have a preference about where i host it? i am thinking maybe even in github actions - we get 2000 minutes a montha nd i guess we need no mor ethan 5 a day.

necopinus commented 3 years ago

Updating title to reflect that we want to do this in Knack. Thus depends on #71. Also, maybe this is something that should be done with ActionsFlow (#19)?