Extract Readwise API into its own sdk project

renehernandez / obsidian-readwise

Sync Readwise highlights into your obsidian vault

MIT License

109 stars 7 forks source link

Extract Readwise API into its own sdk project #7

Open renehernandez opened 3 years ago

renehernandez commented 3 years ago

This would allow to more easily:

Implement retry mechanism when reaching rate limits against Readwise
Auto-paginate, when pulling more documents/highlights than the page_size value
Test the implementation
Mock the API interface in obsidian-readwise for better testing

technicalpickles commented 3 years ago

I'm taking a Building a Second Brain course, and there was a session with the folks from Readwise. I asked:

Do you have any resources you'd recommend for developers wanting to integrate with Readwise? I'm using Obsidian, and there is a community plugin, but it only lets you pull in highlights once you install the plugin. I've read the API docs which look super helpful to get started, but looking for patterns like... pulling down existing highlights initially

And they answered:

If you need some assistance with our API, just email us hello@readwise.io and we’ll be happy to help with your workflow.

technicalpickles commented 3 years ago

Is the thought to make this a separate package on like NPM? Or extract in a different sense?

technicalpickles commented 3 years ago

For testing, I think we could use a combination of:

https://www.npmjs.com/package/node-fetch for being able to run the tests outside of running obsidian
https://www.npmjs.com/package/nock for mocking http requests

I'm going to start a branch to start playing with those ideas.

renehernandez commented 3 years ago

If you need some assistance with our API, just email us hello@readwise.io and we’ll be happy to help with your workflow.

Most likely, this is due to them working on their own Readwise plugin

Is the thought to make this a separate package on like NPM? Or extract in a different sense?

That is something I am considering. The idea of extracting to a separate API would be the best route IMO. There is another Readwise plugin, that it has an API that already supports this features, see here.

technicalpickles commented 3 years ago

There is another Readwise plugin, that it has an API that already supports this features, see here.

Would you be able to use that client? There is not a license in the repository unfortunately. @jsonMartin any chance of adding a license to your repository?

jsonMartin commented 3 years ago

Hey @technicalpickles , sure thing. Just went ahead and added a MIT license to the repo 👍

technicalpickles commented 3 years ago

That's great, thanks! I took a closer look, and the only thing that would need to change in the library is that it uses Notify for updating the status bar. This plugin has its own way to use that, so I think it would just need a more general hook for when there is a status update.

technicalpickles commented 3 years ago

Trying to drop the API in as is within minimal modifcation was probably overly optimistic 🤔 What I'm thinking now is to:

start with the API client in
refactor out a parent class that is more generalized (ie handling things like auth + 429)
import that parent class into obsidian-readwise
refactor obsidian-readwise's API client to use the generalized API client

I don't know how much time I'm going to be put towards it and on what time scale, but I will keep this issue updated.

renehernandez commented 3 years ago

@technicalpickles Sounds good. If you start working on a separate repo for the Readwise API, share it here. I can contribute to it after finishing with the improvements left for v0.1