Provide an API layer - Githubissues

mootari commented 4 years ago

Time to revisit a possible API. Observable's internal API has grown a lot since I created this repository, and their scripts provide a nice index of potential API functions. I'd argue that we shouldn't create our own abstraction, but instead replicate the existing API as close as possible.

The existing API signatures are listed below (extracted and unminified).

Batch 1:

[ ] activityLoadUser({type = "all", days = "7", all})
[ ] collectionLoad(user, collection, {before, page} = {})
[ ] collectionLoadPublic(user, collection, {before, page} = {})
[ ] collectionsLoadUser(user ,{type, page} = {})
[ ] collectionsLoadUserPublic(user, {page} = {})
[ ] documentLoad(id, version)
[ ] documentLoadForks(id, {before, page, type} = {})
[ ] documentLoadHead(id)
[ ] documentLoadHistory(id)
[ ] documentLoadLiked(id)
[ ] documentLoadLikes(id)
[ ] documentLoadMeta(id)
[ ] documentLoadSecrets(id)
[ ] documentMembers()
[ ] documentSearch(query, page) // "query" trimmed and lowercased before request
[ ] documentsLoadByUser(user, {before, page} = {})
[ ] documentsLoadPopular({date, limit} = {})
[ ] documentsLoadPublic({before, page, limit} = {})
[ ] documentsLoadTop({page, limit} = {})
[ ] documentsLoadTrending({page, limit} = {})
[ ] documentsLoadUser({before, page, types} = {})
[ ] documentsLoadUserLikes({before, page} = {})
[ ] documentVersion(id)
[ ] userApiKeys()
[ ] userIdentity(user)
[ ] userLoad(user) // param is optional
[ ] userPublicStats(user) // param is optional
[ ] userSecrets()
[ ] userStats()
[ ] userSuggestions({before, page, type} = {})

Batch 2:

[ ] activityLoadTeam(team, {type = "all", days = "7"})
[ ] documentLoadDataConnector(id, name)
[ ] documentsLoadTeam(team, {before, page, types, share_level} = {})
[ ] teamApiKeys(team)
[ ] teamDataConnectors(team)
[ ] teamInvitations(team)
[ ] teamMembers(team)
[ ] teamSecrets(team)
[ ] teamStats(team)
[ ] teamSuggestions(team, {before, page, type} = {})
[ ] userDataConnectors()

Backlog:

collectionAddDocument({id, document_id})
collectionDelete(collection)
collectionEdit({id, title, description, feature_rank, pinned, type, ordered})
collectionNew({title, description, owner, type, ordered})
collectionRemoveDocument({id, document_id})
collectionReorderDocument({id, document_id, order})
collectionThumbnail(collection, thumbnail) // format for "thumbnail" unclear
continueEditing(unified) // param format and use unclear
continueEditingTeam(team)
documentAllowDataConnector(id, name, allow) // format for "allow" unclear
documentAllowSecrets(id, allow) // format for "allow" unclear
documentDeleteFile(id, name)
documentFork(id, version, {events, team, comments = [], temporary_id = null}) // some params might be flags
documentLike(id, like) // "like" value unclear, possibly status flag
documentMerge(id, version, {events, target, files = []})
documentNew(doc, team)
documentPublish(id, version, title, unlisted = false)
documentPutBack(id)
documentRevert(id, version)
documentsEmptyTrash(team) // optional param, format unclear
documentsLoadCommonAncestry(id, t) // "t" param unclear, use: `/documents/${id}/${t}/ancestry`
documentSubscribe(id, status) // "status" values unclear, also see documentUnsubscribe
documentSuggest(id, {to_id, from_title, version, description})
documentThumbnail(id, thumbnail) // format for "thumbnail" unclear
documentTransfer(id, user)
documentTrash(id)
documentUnpublish(id)
documentUnsubscribe(id)
documentUpdateRoles(id, roles) // format for "roles" unclear
documentUploadFile(id, file) // format for "file" unclear
suggestionClose(e) // "e" might be document id, use: `/suggestion/${e}/close`
teamAcceptInvitation(team, id)
teamDeleteApiKey(team, key) // might be key or id, use: `/team/${team}/api-key/${key}/delete`
teamDeleteDataConnector(team, name)
teamEdit(team, data) // "data" possibly an object with various keys; need to check form
teamEditAvatar(team, avatar) // format for "avatar" unclear
teamEditEmail(team, email)
teamGenerateDataConnectorSecret(team, name)
teamInvite(team, email, role) // "role" values unclear
teamLoadDataConnectorCheckToken(team, name)
teamMemberRemove(team, user)
teamMemberRole(team, user, role)
teamNew({login, name, email, stripe_token, cycle, coupon})
teamNewApiKey(team,{description})
teamRescindInvitation(team, id)
teamSetDataConnector(team ,{name, type, local, credentials}) // some param formats unclear
teamSetSecret(team, {name, value})
unsubscribeAnonymous(id, user, code, status = "none") // "code" unclear, possibly OTP mail token?
uploadAnonymousFile(e, data) // param "e" unclear, data is object with at least key/object "file.size" (blob? no special handling in formdata)
userDeleteApiKey(key) // might be key or id, use: `/user/api-key/${key}/delete`
userDeleteDataConnector(name)
userEdit(data) // "data" possibly an object with various keys; need to check form
userEditAvatar(avatar) // format for "avatar" unclear
userEditEmail(email)
userEmailConfirm(id)
userGenerateDataConnectorSecret(name)
userLoadDataConnectorCheckToken(name)
userNewApiKey({description})
userSetDataConnector({name, type, local, credentials})
userSetLogin(login)
userSetSecret({name, value})

Out of scope:

couponLoad(e) // value unclear (coupon code?), use: (`/coupon/${e}`
recoverLogin(email)
report({anonymous, subject, reason}) // param formats unclear
teamBilling(team)
teamBillingCycleUpdate(team, cycle) // "cycle" values unclear
teamBillingDelete(team))
teamBillingUpdate(team, stripe_token)
teamInvoices(team, starting_after)
teamInvoiceUpcoming(team)
teamSubscriptionNew(team, stripe_token, cycle, coupon)
usersExist(logins) // "login" is array; possibly a check for available user names

bryangingechen commented 4 years ago

If nothing else, it might be useful to include the scripts you used to scrape the API signatures in this repo. At some point down the line we could even set up some GitHub actions jobs to check whether anything has changed (not sure, that might be against TOS).

mootari commented 4 years ago

I discovered the API while looking for hidden parameters for notebook listings (spoiler: apparently there aren't any) in one of the index-*.js scripts.

The extraction process was:

copypaste the whole API code into an editor
derive meaning of minified parameter names from function bodies (or param destructuring), remove/replace minified param names
remove function bodies to avoid copyright/license violation

Currently I don't see how this process could be automated, but you can find the code by using the global search feature in dev tools to search for any of the above function names.

RandomFractals commented 4 years ago

great list. not sure if my pings prompted you to go back to them.

I'll be utilizing some of those end points in the new vscode extension I started working on to free observable js notebooks from its site bounds. See this repo if you are intersted:

https://github.com/RandomFractals/js-notebook-inspector

mootari commented 4 years ago

not sure if my pings prompted you to go back to them.

Nope, as mentioned above I stumbled across the API and saw an opportunity to port it without have to create a concept before hand. Personally I'm planning to use the API (once added) to whip up an alternative notebooks navigator in Svelte (and learn Svelte in the process).

See this repo if you are intersted:

I can absolutely see the value of a VSC integration, although presently not for myself. Btw, I take it you're aware of the other extension by @GordonSmith?

RandomFractals commented 4 years ago

yeah, Gordon published his extension a day after I announced mine. Not sure if that was just a coincidence.

In any case, I plan to make JS Notebook Inspector more generic for not just observable notebooks & integrate it with live share, new native notebooks api in vscode and a few other things to enable collaborative js notebooks in that IDE. Pretty sure that's not what Gordon's extension does, or has in plans.

mootari commented 4 years ago

I've split up the functions based on how I would prioritize their implementation. Batch 1 and 2 should only contain functions that fetch data without modifying it. If there are any objections (or if I missed something crucial) please let me know.

Pretty sure that's not what Gordon's extension does, or has in plans.

TBH, I really don't care. ;) Let's stay on topic, please.

mootari commented 4 years ago

One thing that I'd like to achieve with this API layer is to prevent Observable's dreaded 401 responses with their {errors: []} body.

The API should validate (although not sanitize) data whenever possible. REST responses should however be returned as is. We could add a generic assert for validation that can be overridden if someone wants to skip prevalidation. The assert callback could be a param to the client.

RandomFractals commented 4 years ago

maybe break it into separate tickets to work on.

I don't think I'll be using it, but have fun!

GordonSmith commented 4 years ago

@RandomFractals my release date was pure coincidence (I had looked for an extension before I started and didn't see one and didn't notice yours until a week or so after I published!).

@mootari - will there be a get/put Cells API for a given notebook (not seeing it in the list above)?

In case folks don't know the VSCode insiders has a new "generic" notebook editor in the works, which may be a more "natural" editing experience for folks familiar with Juypter (and lesser extent the ObservableHQ) notebook paradigm.

RandomFractals commented 4 years ago

@GordonSmith np. I think that Jupyter notebooks paradigm is very applicable to Observable js notebooks. It's unfortunate that cell code display got reversed in the initial design of that platform & I am not a huge fan of imports at the bottom practice.

For reference, see this ticket with links to the native notebooks UX and API vscode team is working on. I plan to implement those interfaces for Observable js notebooks when they release it in public vscode v.

https://github.com/RandomFractals/js-notebook-inspector/issues/16

bryangingechen commented 4 years ago

@GordonSmith The edits to notebooks on observablehq.com are communicated via a websocket connection, not API calls. I played around with editing notebooks via a notebook here (it doesn't work since I think ws.observablehq.com was changed to check the referrer after I published, and it appears that the protocol has changed a bit too).

GordonSmith commented 4 years ago

@bryangingechen really looking for an uploadWorkbook(cells) API call, as getting the cells is possible today, just putting them back that is missing...

bryangingechen commented 4 years ago

New notebooks can be uploaded by making a POST request to https://api.observablehq.com/document/new with a JSON object with fields {token, title, version, nodes}. (I guess this is documentNew in the first post, though I don't understand the signature there.)

You can't make edits to notebooks via a direct API call.

mootari commented 4 years ago

will there be a get/put Cells API for a given notebook (not seeing it in the list above)?

@GordonSmith I'd love to add one! But it will likely be a separate class, and we should track its progress in another issue (want to create one? 😉 ). I have an old bare-bones implementation here: https://gist.github.com/mootari/683ac4f22f1e1287e41da63fcf194643

since I think ws.observablehq.com was changed to check the referrer after I published

@bryangingechen referer or origin? Referer would sound like a rather icky and unreliable measure.

I guess this is documentNew in the first post, though I don't understand the signature there.

Both parameters are optional. The first, doc, defaults to:

{
  nodes: [{
    id: 0,
    value: "md`# Untitled`"
  }]
})

Required keys for the JSON payload to /document/new appear to be token and nodes, additional keys are title, version and team.

bryangingechen commented 4 years ago

@bryangingechen referer or origin? Referer would sound like a rather icky and unreliable measure.

Ah yes, I meant origin. 🤦

GordonSmith commented 4 years ago

@mootari which repo for the new issue (document/update - similar to document/new)?

I did add a forum post a while back; https://talk.observablehq.com/t/api-to-upload-notebook/3319

RandomFractals commented 4 years ago

not sure how much Observable platform has changed since 2018 when I was futzing around with their notebooks, but do we have github/gists integration yet? since we can now login with our github accounts & should probably be able to easily sync our notebooks via gists and repos. I've seen some hacks you guys and others have done for this.

However, as I recall @mbostock was not open to opening up observable to 3rd party dev tools and platforms for integration when they were still in beta back then. That actually got me in trouble when I was hacking on it :)

mootari / observable-client

Provide an API layer #4