[Feature] Integrate GitHub with TakeNote

[taniarascia]

GitHub authentication is complete. Now you can log in with GitHub, but the functionality doesn't integrate with GitHub yet. Instead of saving to local storage, all notes in the app will live in a single special Gist, and all notes will be a file within that gist.

Will need to think of some methods of saving settings and categories - such as a special Gist containing JSON for all that data. Will need to think of methods to revert to default if someone chooses to edit that file.

[harrysullivan235345]

Can you add localhost:3000 to the approved URIs on auth0 for development

[taniarascia]

It seemed like a bad idea to me to leave localhost:3000 open for development on the same Auth0 app used for production. It's easy to create an Auth0 account and make an app with all URLs pointing to localhost:3000 and use that clientId instead. I can put up instructions for that if necessary.

[harrysullivan235345]

Ok. Just wanted to ask to find out if you had a special way you wanted to do it or if we would all use a common account but now I know. Thanks!

[ctrlplusb]

@taniarascia - I'm interested in the motivation for using a Gist rather than a repository?

[taniarascia]

It just seemed to make more sense for notes, but it's not set in stone.

[ctrlplusb]

Ok no worries. 😄

Would you be open to reconsidering the sync against a repository (possibly configurable?). I feel like a repository would allow for a great browsing experience within GitHub. Perhaps categories could map to folder structures? It may even be easier to perform partial synchronisation (i.e. at a file level), but I am just guessing at the limitations of the Gist API.

[taniarascia]

One thing I like about only using Gist is having access to the minimal possible amount of user's data.

[papigers]

While I agree that gist permissions are far more appealing than repo permissions, there is no API for filtering and searching through user's gists, you'd have to paginate the user's gists and find the takenote gist manually. this can be a long process if the user has many gists.

[yogan]

It seemed like a bad idea to me to leave localhost:3000 open for development on the same Auth0 app used for production. It's easy to create an Auth0 account and make an app with all URLs pointing to localhost:3000 and use that clientId instead. I can put up instructions for that if necessary.

@taniarascia: I would actually appreciate some instructions. I tried to get the app running with my own freshly created Auth0 account, but got stuck (I have no experience with Auth0). What I did so far:

• registered on Auth0, created a dedicated Application of type SPA
• set Allowed Callback URLs, Allowed Web Origins, and Allowed Logout URLs to http://localhost:3000
• put "domain": "<myuser>.auth0.com" and "clientId": "<my-client-id>" into config.json (actually I used a config-local.json and adapted the import path in src/index.tsx, should be fine, was the only usage)
• npm run start → Sign in or Sign up → get redirected to myuser.auth0.com

However, I only get a pretty generic error page "There could be a misconfiguration in the system or a service outage. […]" there. The data sent in the request to https://*myuser*.auth0.com/authorize looks good as far as I can tell (:authority header is right, query params contain client_id=<my-client-id> and redirect_uri=http://localhost:3000), but again, I don't have much knowledge here.

Did I miss something obvious?

[taniarascia]

Everything sounds good as far as I can tell... I haven't seen that error. That's the way I go about it as well.

1. Create Auth0 account
2. Create application:
   • Domain: .auth0.com
   • Application Type: Single Page Application
   • Allowed Callback URLs: http://localhost:3000
   • Allowed Web Origins: http://localhost:3000
   • Allowed Logout URLs: http://localhost:3000
   • Allowed Origins (CORS): http://localhost:3000
3. Change config.json to:

{
  "domain": "you.auth0.com",
  "clientId": "your-client-id"
}

4. Universal Login - Select "New".
5. Ensure "GitHub" with Gist permissions is the Connections.

[yogan]

Nevermind, it was misconfiguration by me. Somehow I had foo.auth0.com as domain in the JSON, but the correct one for my account is actually foo.eu.auth0.com. 🤦‍♂

[jm1200]

Out of curiosity, why not use Firebase for everything? Hosting, Auth and storing user data?

[EricPKerr]

One thing I like about only using Gist is having access to the minimal possible amount of user's data.

Unsure of the options for GH permissions - is it possible to only allow permissions to a single repo?

While I agree that gist permissions are far more appealing than repo permissions, there is no API for filtering and searching through user's gists, you'd have to paginate the user's gists and find the takenote gist manually. this can be a long process if the user has many gists.

Potential solution to help with this - you could store/cache the gist reference url in localStorage, and then only iterate through the gists once if not cached yet.

[taniarascia]

@EricPKerr Hi! I don't think it's possible to only allow permissions to a single repo, but I'm not entirely certain. For this use case, I'd have to create a private repo for the user (something like takenote-notes) then access that in all subsequent requests. I'm not sure how this could be abused but I'd be interested to know what some of the potential security issues regarding having full repo access might be.

Overall, I believe a repo makes more sense than a gist/multiple gists because:

• Easy to clone/download the entire data set all at once
• Notes can be real files with filenames in a regular directory structure instead of gist uuids
• Not mixed in with a bunch of other gist data the user might have
• The filter/search issue mentioned before

My concerns are:

• What happens if a user modifies the config data (settings, categories, need to be saved in a JSON file somewhere or in a database)
• What happens if a user changes the repo name
• Any security issues around having full repo access

[EricPKerr]

@taniarascia I think the main concern as a potential user would be "this gives access to all my repos, potentially even work related ones" - but that sounds like that's the case with all GitHub OAuth connections.

I like the flexibility of a single repo with all of the formats you're describing. It would even potentially be a distribution/discovery mechanism for new users to find out about the takenote service when they're searching github or google.

[gnzlbg]

Why use a gist?

I'd prefer if my notes would be sync'ed to a git repository, e.g., with one directory per note, and inside each directory one would have the markdown file for the note, attachments, and maybe a human readable toml or json file with metadata like tags.

That way, the directory history can be leveraged by applications, e.g, to show how a note changed over time.

[taniarascia]

For now, I think just one file per note, not a directory per note. That's the way I'd rather have my notes organized. I don't think there will be attachments for TN.

[mikelehen]

Not sure if this will be helpful, but FWIW I've thought about GitHub sync a bit and am bummed about the issue @EricPKerr mentioned that the "repo" OAuth scope gives access to all public and private repos, so it's not ideal from a permissions perspective.

An alternative that might be feasible would be to leverage a GitHub App for the integration since "GitHub Apps can be installed directly on organizations and user accounts and granted access to specific repositories." But I'm not very familiar with the limitations of GitHub Apps and it might be awkward to try to structure it in that way (I think GitHub Apps are meant to monitor and respond to actions in a repo rather than e.g. mirror changes from a web app into GitHub commits 😄).

A final alternative could be to leverage GitHub Deploy Keys and use normal git+ssh to interact with GitHub instead of the GitHub API. Essentially TakeNote would generate an ssh key pair and instruct the user to add the public key to the repository's deploy keys with read/write access, and then TakeNote would use normal git+ssh using the private key to interact with the repo. One advantage of this approach is that it would work with any git server, not just GitHub.

That said, ignoring the issue of being overly permissive, normal OAuth access will by far be the best user experience, not requiring any manual configuration on their part. :-/ So this is mostly just food for thought.

[taniarascia]

Hi @mikeehen,

Thanks for the input!

I looked into GitHub apps as my first option (since I only planned to use one data and authentication source) but it seems like GitHub apps do not apply to my use case. They seem more like bots that work on behalf of a user or organization than a centralized app that many users can log into.

From the docs:

• Don't expect the GitHub App to know and do everything a user can.
• Don't use a GitHub App if you just need a "Login with GitHub" service.
• Don't build a GitHub App if you only want to act as a GitHub user and do everything that user can do.

I also don't want the user to have to do a lot of manual setup and for the app to be accessible from the web, so I'm not so sure about the deploy keys method.

I think there are tons of third party apps out there that integrate with GitHub and grant repo access, and people don't really question it too much. If someone doesn't feel comfortable giving repo access to a third party, that's understandable, but I guess this app just won't be for them. Potentially in the future it could integrate with some other third party, like Dropbox. I just don't really see any other viable option for having a web app that integrates with a GitHub repo.

[mikelehen]

Thanks for the response. I agree with all that. I'm probably a little extra sensitive / paranoid about giving an app access to all repos because I used to work at a company that had private GitHub repos and so it was important not to grant apps "repo" access, since it would have been giving access to internal company code... But many people won't care I'm sure. 😄

[taniarascia]

Makes sense. There just doesn't seem to be any way around it. 😞

[taniarascia]

I'm having a thought about storing notes...ideally, you would be able to save all notes as their titles (what appears in the note list sidebar). However, the app currently allows identical titles, as the uuid is the only unique factor of the note.

The easy way to save all notes into a GH repo would be to save them as <uuid>.md, but ideally it could be note-title.md, or even category/note-title.md with categories used as directories, but that would require not allowing duplicate note titles, or knowing to save them as -1.md or (1).md, and I'm not sure a clean/easy way to do that.

[BrianHung]

The easy way to save all notes into a GH repo would be to save them as .md, but ideally it could be note-title.md, or even category/note-title.md with categories used as directories, but that would require not allowing duplicate note titles, or knowing to save them as -1.md or (1).md, and I'm not sure a clean/easy way to do that.

One way could be to use front matter, like how Jekyll parses markdown files with a publishing title and tags.

---

Edit In addition to the plugin mentioned below, there's also https://github.com/jonschlinkert/gray-matter to help parse the front-matter.

[vhf]

@BrianHung I like this idea: many markdown-based projects support YAML frontmatter ; GitHub renders them nicely ; it lets you add arbitrary metadata to your files. react-markdown doesn't seem to have support for it (yet) but it's based on remark which has a good frontmatter plugin.

[taniarascia]

I was definitely already planning on frontmatter, since when you download a note it has frontmatter. I guess my only concern is people might be annoyed that their repo of notes is all uuid.md files, but it’s the only way to guarantee uniqueness...

[vhf]

@taniarascia You probably already thought about it but I haven't seen it mentioned (sorry if I missed it): What about postfixing the filename with a short uuid? I also think naming files with a uuid only would be a bit annoying but something like ${title-slug}-${short uuid}.md wouldn't be so bad IMO. The uuid could be stored in the frontmatter if need be. Another idea would be to prefix the filename with the creation date?

[taniarascia]

Those are really good ideas! At least then you’ll get an idea of what’s in the file.

[norbertbede]

hi @taniarascia

We are using pouchdb on web and mobile devices as db. https://pouchdb.com/ You should store md userdata as document-store json documents. Pouchdb running in browser database (e.g indexdb). This make it simple in various deployment scenario - A. Local for testing. B. Connect to couchdb server (similar to firebase) and share over team.

With above design approach, you can implement over pochdb/couchdb singluser/shared datamodel (or in first stage stay simple userdb).

The best is couchdb can run offline/online and handle full sync automatically.

norbert

[vHanda]

Hey guys.

I'm the author of GitJournal (GitHub, Website) - a mobile first note taking app with git integration. I just wanted to chime in as this project seems to blend in quite well with exactly what I'm trying to do with GitJournal.

Filenames - By default I use the title of the note, in the case of duplication, I append a _0 or _1. However, different users have different preferences and in the end this is now configurable with a few present options. I'll soon be adding a simple templating language to allow it to be configured exactly as the user wants. My conclusion is that some developers want to be able to control everything.

Authentication: Right now, you can only authenticate with SSH. GitJournal is registered as an OAuth app and if given authorization, it will create the repo / use an existing repo and add the SSH key as a deploy key. However, from the stats of the last 30 days - about 40% of the users preferred to create the repo manually and add the deploy key manually and not give GitJournal access with OAuth. I'm not sure if this is because of them not trusting GitJournal or because in the OAuth mode, it creates/uses a repo called journal and this is not configurable.

Storage - The data is stored in markdown files with a YAML header. Though the YAML header is optional. I don't have data on how many users choose to disable it.

--

For TakeNote, instead of using the GitHub APIs another option would be to use git from the browser. This way you could target all Git providers - GitHub / GitLab / Gitea / etc. Though you will still suffer with the problem of all or nothing access. You could switch to SSH, if takenote is also made into an electron app.

--

I'm really excited to see this project evolve. Please let me know if I can provide any statistics from GitJournal's users to help with any decisions. Cheers!

[alexkreidler]

Here are my thoughts.

They may conflict with the existing internal implementation details, so feel free to point out things that seem difficult.

• All notes should be stored in one Git repository
• notes should be named <note-name>.md
• notes should either
  • contain frontmatter that includes any app metadata, like category, UUID, date_created, etc
  • or be in folders by category (this allows for flexible setups on local machine with lots of notes)
• in general however, app metadata should be a minimum. E.g. I should be able to create a .md file on my computer without having to generate a UUID or any other metadata, commit, push, and see it in the app after refreshing

Security concerns:

• personally, I like to limit access as much as possible. This means
  • for Github OAuth, allow user to choose between access to public vs private repos. However GH OAuth allows access to all repositories. Thus from this perspective, the better option is:
  • Github Apps, which allow restricting access to specific repositories. This should work fine with the model above, b/c I want one @me/notes repository to work from

I really don't see why we would want to have access to every repository from a given GH user. Some users I think will want to create multiple repos to isolate different types of notes, but I think most should be fine with one. Also, it won't be any new code, just the user having to do different configuration to add the app to a different repo.

It would also be cool to have something like GitPod-style URLs such as https://takenote.dev/#<github_repo> that would automatically open public repos. We could then even have a TakeNote badge that says: this repo is editable!

Another idea: lots of repos have docs/ folders and other weird structures where markdown is embedded with code. Allow editing of those repos by just listing all markdown files as notes as well.

This is an awesome app, and I'm excited to hear more! Let me know if there's anything I can do to help.

[ericop]

Yeah, this is a way cool app. But before I would 'use it for real' we need to have the OAuth locked down to a single repo. To creepy to have access to every repo public and private.

[taniarascia]

As mentioned above, there is no way to lock down OAuth to a single repo.

[ericop]

Thanks for clarification. I didn't know. https://stackoverflow.com/a/29450135 So I don't know if you have an FAQ on this yet, but to improve the UX you might have, a note that says,

"Giving full access to all your repos too scary? Github guidelines suggest build a specialize account per OAuth use case (i.e., create a special account to sycn your TakeNote data)".

That might be putting words in their mouth, but it seems to be the only option they leave you with, if you only want access to one repo, build a new user accou t with just one repo, correct? https://developer.github.com/apps/differences-between-apps/#requesting-permission-levels-for-resources

On Sat, May 2, 2020, 9:15 PM Tania Rascia notifications@github.com wrote:

As mentioned above, there is no way to lock down OAuth to a single repo.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/taniarascia/takenote/issues/165#issuecomment-623037801, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHZ7SKDY3KIEQ6D34E6Z3LRPTATJANCNFSM4JIIOKEQ .

[taniarascia]

It might be a good idea to get rid of OAuth entirely and go with the deploy keys strategy that @mikelehen suggested previously. Working with any Git repo and restricting access to all but one repo seems like a much better approach. I'll have to do some research and probably a lot of rewriting. Expecting users to create a new GitHub account just to access this app doesn't seem feasbile.

[Usarneme]

Not sure if this is the right place or I should open a new issue so apologies in advance.

Using GitHub account to sign into your application required both READ and WRITE access to all public and private GH repos. I had to stop and back out when I saw that. Please allow for read-only or an alternative method of signup; concerns about negligence or maliciousness, either way people won't want to grant that level of access to a notetaking app.

Screenshot of auth requirements:

[taniarascia]

@Usarneme would appreciate it if you read the thread or even the last few comments on this one as this topic has been discussed over and over.

I believe my final decision will be to not have any public facing application, and only supply the source code for anyone to have their own self-hosted app if they wish.

[kodyclemens]

@taniarascia Have you decided which path you are taking regarding data persistence?

[taniarascia]

I'm thinking a private repo called takenotes gets created when the user logs in for the first time. It will contain all notes as markdown files as well as a categories.json and settings.json.

• If the user deletes those json files, categories and settings will reset to default.
• If the user deletes the takenotes repo, all their data is lost and on the next log in it will create a new one.
• If the user sets the repo to public, the app will show a warning.
• The app can continue as normal and only save to GitHub when the sync command is run, in which case it should make a new commit for any file that has changed.
• Not certain on filenames - saving them as the uuid would be easiest but ugliest. Maybe note slug + uuid?
• I would like categories to be like subdirectories and notes to be nested in them, but that would add complication

That's what I've thought of so far.

[taniarascia]

Another standstill. There's no way to make multiple files in one API call. So I tried using the Git data API to do it the low level way, but you still have to make an API call to blobs for every single file. So there's no way I won't get rate limited and syncing won't take forever once you have a lot of notes. I'm going to have to abandon the GitHub sync idea because it's just not proving feasable.

// Create blob
// https://docs.github.com/en/free-pro-team@latest/rest/reference/git#create-a-blob
const blob = await SDK(Method.POST, `/repos/${username}/${repo}/git/blobs`, accessToken, {
  content: 'My first blob',
})

// Create tree
// https://docs.github.com/en/free-pro-team@latest/rest/reference/git#create-a-tree
const tree = await SDK(Method.POST, `/repos/${username}/${repo}/git/trees`, accessToken, {
  tree: [{ path: 'blobfile.md', mode: '100644', type: 'blob', sha: blob.data.sha }],
})

// Create commit
// https://docs.github.com/en/free-pro-team@latest/rest/reference/git#create-a-commit
const commit = await SDK(Method.POST, `/repos/${username}/${repo}/git/commits`, accessToken, {
  message: 'This is a test commit',
  tree: tree.data.sha,
})

// Update a reference
// https://docs.github.com/en/free-pro-team@latest/rest/reference/git#update-a-reference
const update = await SDK(
  Method.POST,
  `/repos/${username}/${repo}/git/refs/heads/master`,
  accessToken,
  { sha: commit.data.sha, force: true }
)

[taniarascia]

Or I could save all the data as one json file. It wouldn't be pretty and the data wouldn't be super useful outside of the context of the app but it would work......

[taniarascia]

Got that all working!

[taniarascia]

Todo:

• [ ] Update categoryFetch, settingsFetch, notesFetch to pull from the API (GitHub).
• [ ] Error handling

[alexkreidler]

Just something I saw recently: https://isomorphic-git.org/en/, https://isomorphic-git.org/docs/en/fs, https://github.com/jvilk/BrowserFS

[taniarascia]

That's a neat project but doesn't really work for this use case.

[taniarascia]

I merged in all the GitHub integration code with this note:

TakeNote is a note-taking app for the web. You can use the demo app at takenote.dev. It is a static site without a database and does not sync your notes to the cloud. The notes are persisted temporarily in local storage, but you can download all notes in markdown format as a zip.

Hidden within the code is an alternate version that contain a Node/Express server and integration with GitHub. This version involves creating an OAuth application for GitHub and signing up to it with private repository permissions. Instead of backing up to local storage, your notes will back up to a private repository in your account called takenote-data. Due to the following reasons I'm choosing not to deploy or maintain this portion of the application:

• I do not want to maintain a free app with users alongside my career and other commitments
• I do not want to request private repository permissions from users
• I do not want to maintain an active server
• I do not want to worry about GitHub rate limiting from the server
• There is no way to batch create many files from the GitHub API, leading to a suboptimal GitHub storage solution

However, I'm leaving the code available so you can feel free to host your own TakeNote instance or study the code for learning purposes. I do not provide support or guidance for these purposes.

TakeNote was created with TypeScript, React, Redux, Node, Express, Codemirror, Webpack, Jest, Cypress, Feather Icons, ESLint, and Mousetrap, among other awesome open-source software.