GitHub web hook based script upload

[jesus2099]

Frankly, a great feature like on open userjs, it would allow those growing number of people using git for their scripts to avoid having to manually update greasy fork each time.

[JasonBarnabe]

It's not a web hook, but you can provide a URL pointing to the raw version of your file on GitHub, and it will be synced once a day.

[jerone]

Duplicate: https://github.com/JasonBarnabe/greasyfork/issues/4

Still +1 on OAuth with Github, for easy import and instant sync.

[jesus2099]

Oh cool, i didn’t notice. I will try but maybe it will force update if we go in our admin.

[JasonBarnabe]

I know next to nothing about OAuth so I can't offer an opinion right now. Will look into it.

[jesus2099]

Jason, it is not said in https://greasyfork.org/import that it will auto-update everyday, is it the correct place ? Maybe you should add a notice there.

[jerone]

I know next to nothing about OAuth so I can't offer an opinion right now. Will look into it.

This looks interesting... http://www.codecademy.com/tracks/oauth & http://peter-murach.github.io/github/

[jesus2099]

I see a people going away when knowing it was only updated once a day… I don’t feel strong necessity myself for more but it’s true that when users know that kind of stuff they might, you know, try to find another way…

[JasonBarnabe]

You're right that webhooks provide a better experience than a sync-once-a-day. It's also easier on the server to sync on demand rather than have to check every day. A problem not yet mentioned is that the sync times for each script is different (it's once-a-day, but not everything at once). If a script requires a library and both were updated on GitHub, then it's possible that one and not the other is updated, which could cause problems.

After doing some investigation and thinking, I think I've figured out how this will work. Let me know if this sounds right:

• Sync by URL will still exist for non-GitHub URLs (and for GitHub if people are lazy/don't care)
• Users will have a link from their user page providing instructions for setting up a GitHub webhook.
• The instruction page will let users generate a (one, single) secret key. This secret key will only be displayed once; if the users forgets, they will not be able to recover it and will have to generate a new one. For informational purposes, the instruction page will also list GitHub-synced scripts the user has already created along with the sync URL.
• Users will set up a GitHub webhook using the secret key provided, pointing to a payload URL like https://greasyfork.org/users/1-jasonbarnabe/webhook, content type application/json (the default), and "Just the push event." (default). Users could potentially hook up multiple repositories using the same secret key.
• All webhook requests from GitHub will be validated using the secret key. If the secret key is wrong, Greasy Fork will return a 403 and not do anything.
• When a valid push event comes in, GreasyFork will get the repository URL and find all scripts in that repository. It will then download the sync URL for each of those scripts and update them based on that. The sync type of these scripts will be set to "Webhook" and will cease to sync every 24 hours, if that's how they're configured. The user will be able to change the sync type back to another value, but it will keep getting set back to "Webhook" on every push.

So this solution will not allow these cases (which I don't see a reason for anyway)

• A user setting up a webhook for scripts they don't own (by pointing to another user's webhook URL).
• A user creating a webhook for a repository but excluding some scripts that sync from that respository from the webhook.

Still not sure what OAuth has to do with anything.

[JasonBarnabe]

Actually, I can't think of any reason to not display the secret key whenever the user wants to see it, as long as only they can see and they can regenerate if they want. If someone gains access to another person's account, there's nothing the key will give them that being able to update the scripts won't.

[jesus2099]

It sounds great! Thanks for taking the time to create a great service.

[jerone]

Still not sure what OAuth has to do with anything.

OAuth will make these steps redundant:

• The instruction page will let users generate a (one, single) secret key. This secret key will only be displayed once; if the users forgets, they will not be able to recover it and will have to generate a new one. For informational purposes, the instruction page will also list GitHub-synced scripts the user has already created along with the sync URL.
• Users will set up a GitHub webhook using the secret key provided, pointing to a payload URL like https://greasyfork.org/users/1-jasonbarnabe/webhook, content type application/json (the default), and "Just the push event." (default). Users could potentially hook up multiple repositories using the same secret key.
• All webhook requests from GitHub will be validated using the secret key. If the secret key is wrong, Greasy Fork will return a 403 and not do anything.

[JasonBarnabe]

In other words, it would allow using the GitHub API to create the webhook?

[JasonBarnabe]

Couple changes to my long comment above:

• Greasy Fork will only attempt to sync the scripts that were modified (instead of all scripts from that repo)
• Greasy Fork will also grab the commit comments and show them in the history tab.

[JasonBarnabe]

Another change. GitHub's "raw" links can be cached for up to 5 minutes depending on the last access, so I'm going to have Greasy Fork only do the sync 5 minutes after the webhook request.

[JasonBarnabe]

OK, try it out. Instructions are linked to from your user page.

[jesus2099]

I’m testing the stuff… That COLLECTION HIGHLIGHTER is now WebHooked in the sync tab but has yet to be auto-updated. I updated it today but it still says 6 days ago ATM.

[JasonBarnabe]

In GitHub's webhook setup screen, it'll list recent requests. What do you see there?

[JasonBarnabe]

Nevermind, I found the problem. I had to restart delayed_job so it could see the changes I made to the sync lib. I reprocessed your hook requests.

[jesus2099]

Excellent, I even can see the commit messages in history tab !

[jesus2099]

I must have done something wrong but I don’t understand why that other script has not jumped to webhook status (still auto updating daily).

[JasonBarnabe]

There were carriage returns (\r) in your sync URLs, probably from when you imported them. The site split them on \n and you probably had \r\n from being on Windows. I've made the site strip whitespace from the sync URLs and I've updated the existing data.

Likely the carriage returns were fine when making the sync requests, but when doing a search from the webhook, they prevented the scripts from being found.

[jesus2099]

I just pasted URLs in that textarea with new lines (i’m windows indeed). If you mean it’s repaired and won’t happen again, great, thank you. :)

[jesus2099]

Another small minor note. You are showing the commit messages separated by “ - ”. Maybe it would be nice to put each one in a html paragraph (p) and to preserve newlines (br) inside each one ?

[JasonBarnabe]

The changelog right now is just inline text content. I don't think I want to change that.

[jesus2099]

Ok, no problem, it's already very handy! I think everything works now, doesn't it?

[JasonBarnabe]

Just waiting for someone else to confirm.

[jerone]

Just waiting for someone else to confirm.

Confirmed ( https://github.com/jerone/UserScripts/commit/caf7525c86b2ef8db632833ae8e0340bd4622212 -> https://greasyfork.org/scripts/54-github-gist-share )

[JasonBarnabe]

Thanks!