cloudrac3r
commented
2 years ago Why is a new API needed - what's wrong with /v1/?
How is pagination planned to work in the new API?
Open SamantazFox opened 2 years ago
cloudrac3r
commented
2 years ago Why is a new API needed - what's wrong with /v1/?
How is pagination planned to work in the new API?
SamantazFox
commented
2 years ago Why is a new API needed - what's wrong with /v1/?
I'd like to make breaking changes to the API in order to clean things up. This will give the other devs enough time to migrate, without breaking anything for the end-user (especially given that we don't control how long it takes for changes to propagate across the various instances).
For instance, I'd really like to create data objects that can be used in different contexts, instead of having flat structures everywhere (=> easier to maintain on both sides).
E.g, the author details (from the video endpoint):
{
"author": "<name>",
"authorId": "<ucid>",
"authorUrl": "/channel/<ucid>",
"authorThumbnails": [
"/path/to/thubmnail1.png",
"/path/to/thubmnail2.png",
"/path/to/thubmnail3.png",
"/path/to/thubmnail4.png"
],
"subCountText": "1.2M"
}would become a dedicated object, which can be reused for almost all the other endpoints (search, hashtag, feeds, ...) as well as other objects on the same endpoint (e.g: related videos):
{
"author": {
"name": "<name>",
"id": "<ucid>",
"url": "/channel/<ucid>",
"thumbnails": [
"/path/to/thubmnail1.png",
"/path/to/thubmnail2.png",
"/path/to/thubmnail3.png",
"/path/to/thubmnail4.png"
],
"subCount": 1200000,
"verified": false
}
}How is pagination planned to work in the new API?
On that part, it depend on what youtube allows us to do.
I'd like to use page numbers as much as possible, but given how some continuation tokens became impossible to predict, it seems reasonnable to always provide said token in the response, and use that all the time.
Without a more advanced caching system is available, I hardly see how we can circumvent that problem, without having to do hundred of innertube queries.
EDIT: indeed, for invidious playlists, there will be regular paging, as we fully control the content. page and per_page would control what's returned (with per_page being limited to 200 or something)
SamantazFox
commented
2 years ago I'm also considering following the OpenAPI specification to provide a proper API documentation (using swagger).
cloudrac3r
commented
2 years ago From my thoughts developing NewLeaf, I'd prefer to use continuation tokens as much as possible, because it makes caching easier for everybody and they can be bookmarked on the web-side without losing one's place. I talked about this in great detail in CloudTube on Matrix but that's the summary. Even if there's a way to random access seek into pages by page number, I think continuation token is more stable and easier, and should always be supported.
I still think it would be best to paginate playlists by a token even when we control the playlist content, for consistency.
I tried using OpenAPI/Swagger in the past and it was an absolute pain. If you really want to try it (why?) set yourself a time limit and don't work past that time limit, or you'll frustrate yourself and be out of time.
cloudrac3r
commented
2 years ago Agreed on moving author data to an author object, but what if one fetching method provides more information than another? For example, watching a video would provide author icon url, but related videos wouldn't provide that. Are most of the author fields just going to be optional, and provided on a best-effort basis depending on which API endpoint is being called?
SamantazFox
commented
2 years ago From my thoughts developing NewLeaf, I'd prefer to use continuation tokens as much as possible, because it makes caching easier for everybody and they can be bookmarked on the web-side without losing one's place. I talked about this in great detail in CloudTube on Matrix but that's the summary. Even if there's a way to random access seek into pages by page number, I think continuation token is more stable and easier, and should always be supported.
Ah, thanks for the input :)
Note that in some places (like a channel's playlist or community page) the token represents a fixed point in the content (timestamps, in the case of the community page), but that in most places, the continuation token is relative (e.g in playlists, we provide an offset n, and it returns 100 results starting from there. The returned results will be different if new videos were inserted at the beginning of the playlist).
I still think it would be best to paginate playlists by a token even when we control the playlist content, for consistency.
I'm not sure how to achieve that. What would represent that token, in that case? a specific video ID? How would it change if the playlist order changes?
One sure thing is that we'll have to send a lastModified info (both in headers and body?)
I tried using OpenAPI/Swagger in the past and it was an absolute pain. If you really want to try it (why?) set yourself a time limit and don't work past that time limit, or you'll frustrate yourself and be out of time.
Thanks, noted!
Agreed on moving author data to an author object, but what if one fetching method provides more information than another? For example, watching a video would provide author icon url, but related videos wouldn't provide that. Are most of the author fields just going to be optional, and provided on a best-effort basis depending on which API endpoint is being called?
I'd still send the thumbnail field, but it would be an empty string. Same thing with the "verified" info: it would be false if upstream doesn't provide anything.
In general, I don't like optional fields. I prefer using default values.
cloudrac3r
commented
2 years ago When you're the author of an API, you need to use either optional fields or null values so that API consumers can tell whether the data is real or is a placeholder. views: 0 and verified: false mean no views and not verified. You can't use those to mean "well, they might be or they might not". Otherwise, how can anyone trust the API? Returning incorrect information is always a bad idea, and here it is completely preventable.
SamantazFox
commented
2 years ago @cloudrac3r Ah, yeah, makes sense! I'll probably go with nil/null, then.
panki27
commented
2 years ago What about the /latest_version endpoint?
I can't find any documentation on that, but I can successfully use it to get links to embeddable streams in my project.
There seems to also be an /api/manifest endpoint, both of these aren't under /v1/ - should these be included?
SamantazFox
commented
2 years ago What about the
/latest_versionendpoint? I can't find any documentation on that, but I can successfully use it to get links to embeddable streams in my project.
The /latest_version endpoint is specific to the video proxy. It's used to get a fresh video playback URL, as they expire after 6h (we don't want to serve an outdated URL from the cached entry stored in the DB).
There seems to also be an
/api/manifestendpoint, both of these aren't under/v1/- should these be included?
This is for compatibility with the endpoint of the same name on Youtube. It provides the DASH manifest required to play adaptative videos. Similarly, it's part of the video playback (and proxying) system.
iBicha
commented
1 year ago I recently started looking into Invidious, as a backend of choice for a Roku TV app (https://github.com/iBicha/roku-youtube)
From initial observation, the default front end is way more capable than what the API can do in most cases. Or at least this is my perception (perhaps the information on https://docs.invidious.io/api/ is incomplete)
On non-web frontends, multi-instance support is a consideration. For example, switch to an instance with the least load, etc. In this case I'm talking about the https://api.invidious.io/, but instances that advertise their load can help. Additionally, I see a lot of people asking about proxying and how it is expensive to them. It would be interesting to see if the instance has proxying enabled or not.
89Q12
commented
1 year ago Mention because it affects all projects. @FireMasterK @cloudrac3r @SamantazFox
Working on my fork/rewrite of Invidious and in terms of interoperability I think creating one API spec that fits all is the wrong approach. Since each project works a bit differently, e.g. Piped uses client-side rendering while Invidious uses server-side rendering, the APIs already have different needs and are therefore different anyway.
In my project for example I want to support multiple backends as well as different services, such as SoundCloud using NPE, I came up with a set of interfaces that allow me to implement any service/API I want. Not saying that my are the way to go but its a good starting point I think.
You might think why not define a standardized API specification since we are all working with the same data anyway, but currently the Piped API provides different data than the Invidious API, e.g. licenses are present in the Invidious API but not in the Piped API. So now if Invidious were to implement the Piped API, what about the license data for a particular video? Return an empty string or make a separate request to YouTube? Well, that depends, but using said interfaces, it wouldn't really be a problem since the Piped API would have to supply the license data or the interface implementation would have to return placeholder data. Now you might say, but isn't that exactly the same as before? No, it's not, because if Piped now decides to supply the license data in the API, the change is just a few lines of standardized code for everyone.
I think the best approach for each project would be to use the same approach as NPE, i.e. modify the APIs so that they can be used to implement a number of the standardized interfaces mentioned. This would provide enough flexibility for each project while allowing interoperability at no additional cost, other than the upfront cost of developing the standardized interfaces.
This is also easy to implement in each language of each project and has the advantage that it would allow the use of basically any service as long as the returned data can be represented via the mentioned interfaces.
What do you all think of this idea?
FireMasterK
commented
1 year ago Well, I like the idea of having a unified API, but first we should first define a concrete specification, preferably as an OpenAPI one
89Q12
commented
1 year ago That would be an option too but I think I haven't expressed myself clearly, what I meant is that we define a concrete adapter design composed of interfaces(like the ones I linked) that can be used with every project API. I don't mean that we all have to speak the same API, because with a standardized adapter there is no need for a unified API as I think this would introduce unnecessary overhead. I hope its a bit clearer what I meant, sorry for not being that clear in the first place.
iBicha
commented
6 months ago After a while of consuming the v1 api, here are some of my concrete thoughts on the limitations
/api/v1/auth/subscriptions should return more than the ID and the name. At least the thumbnails, and the sub count. We need a way to display the channels like in https://www.youtube.com/feed/channelsvideos/:id is an all or nothing, which is a problem. We fetch the basic video info, the next endpoint (for related videos and such) and another hit for streaming data. This is often an overkill and slower than it needs to be. Also, when the streaming data has an error (such as the VideoNotAvailableException("The video returned by YouTube isn't the requested one")) we end up getting none of the other info, which usually does not have restrictions.I know some of these are requiring not only an api change but also a db change, but I just wanted to share some of the areas of improvements from the perspective of a 3rd app.
tezlm
commented
6 days ago In my experience, modifying arrays with rest is painful. It's better to model data like playlists as an object with ordered keys, i.e.
{
"a": { ... },
"b": { ... },
"c": { ... }
}
// though it might be nicer to do
[
{ "seq": "a", ... },
{ "seq": "b", ... },
{ "seq": "c", ... }
}By modeling playlists way, editing becomes much easier and more well defined. String keys could be calculated with something like mudderjs, though it shouldn't be too hard to roll your own. I imagine the edit playlist api could look like
PATCH /playlists/:id
{
"upsert": [
{ "seq": "d", ... },
{ ... }, // eliding a seq appends; the server will generate a seq id for you
],
"delete": ["b"],
"rename": [{ "from": "a", "to": "e" }] // may not be necessary, delete/upsert works fine if you only need id and seq properties
}
Here is the v2 API mockup I came with. Comments are appreciated!
APIs I got inspiration from: