API v1 & major refactoring: Hono, file read optimizations, and upload streaming

[linkability]

15

A better & more sensible API which is actually usable for developers

[Jack5079]

A mollermeth of sorts

[nbitzz]

true!

[Jack5079]

Genius! Mr. 2283 and Mr. Impersonator will never find out

[nbitzz]

i think we should try porting the client to api v1 too maybe we should also have search and a page endpoint I just wonder how we're gonna optimize the files section tbh /shrug

[nbitzz]

we should also probably do some proper planning, idk /shrug but i think consistent naming would help, everything should be camelcase, do not use snake case or all lowercase

[nbitzz]

i think we should also make v0 basically just fwd to v1 also admin endpoints can probably be merged into accounts like DELETE /account/@jack etc...

[nbitzz]

Read over a bit of discord.js code and...

We'll need to see if we can do something else to stream files directly to Discord, discord.js resolves streams to buffers so it's not going to help much

[nbitzz]

We'll need to see if we can do something else to stream files directly to Discord, discord.js resolves streams to buffers so it's not going to help much

Great! Oceanic does this too. @Jack5079 we'll need to investigate and build our own library or something to stream files directly to discord :/ if we're kicking out a discord client library for file uploads, we might as well do it for everything: the One Small Issue is ratelimits

[nbitzz]

Trying to implement some basic REST client now; this hopefully means in the future we can remove our dependency on discord.js

Also, since Hono forces v18, might as well embrace it! We should switch all axios calls to the native node fetch API 🙃

[nbitzz]

Fuck

I don't believe there's a way to customize what the HEAD route returns What should we do? Hope that it works OOTB? Try tricking Hono into believing a file from monofile is a real file on the disk or something? Are we switching to Fastify?

[nbitzz]

Reading over Hono's codebase reveals that Hono just internally fires a GET request then returns just its headers

I'll need to read over it some more, but if Hono just cancels the request after it gets the headers or something, we should be fine...right? If not, is there some way we can detect that the request is a HEAD request?

We could always fork Hono to mix in something that allows us to handle the HEAD request without getting the files from Discord, but I'd rather not do that unless required (We are not making Fastifyle, fuck you)

[nbitzz]

I think later today I'll go ahead and attempt to implement the optimizations described in #44 then maybe write the /file/:id route myself; surely it can't be too bad

[Jack5079]

Reading over Hono's codebase reveals that Hono just internally fires a GET request then returns just its headers

I'll need to read over it some more, but if Hono just cancels the request after it gets the headers or something, we should be fine...right? If not, is there some way we can detect that the request is a HEAD request?

We could always fork Hono to mix in something that allows us to handle the HEAD request without getting the files from Discord, but I'd rather not do that unless required (We are not making Fastifyle, fuck you)

@nbitzz

app.get('/route', async (c: Context) => {
  if (c.req.method === 'HEAD') {
    // lol
  }
})

too lazy to test

[Jack5079]

also sending streams if you can't use ctx.stream

const body = new ReadableStream({ // You can convert Readable to ReadableStream using Readable.toWeb
   start(controller) {
         controller.enqueue(m);
   },
   cancel: off, // fires if request aborted i think
 })
 return ctx.body(body)

[nbitzz]

Reading over Hono's codebase reveals that Hono just internally fires a GET request then returns just its headers

I'll need to read over it some more, but if Hono just cancels the request after it gets the headers or something, we should be fine...right? If not, is there some way we can detect that the request is a HEAD request?

We could always fork Hono to mix in something that allows us to handle the HEAD request without getting the files from Discord, but I'd rather not do that unless required (We are not making Fastifyle, fuck you)

@nbitzz

app.get('/route', async (c: Context) => {

  if (c.req.method === 'HEAD') {

  // lol

  }

})

too lazy to test

Oh

[nbitzz]

[nbitzz]

I think later today I'll go ahead and attempt to implement the optimizations described in #44 then maybe write the /file/:id route myself; surely it can't be too bad

It's 11pm and although I can't sleep (for various reasons) I'm not motivated to do it

[nbitzz]

@Jack5079 Maybe we should use the new parameters introduced into the CDN link to find out when a link expires instead of just setting an arbitrary 20 minute value... or not. I don't know, and as long as it works, that's fine.

[nbitzz]

When I get the chance I'll make writeFileStream exist

Also, I guess I might as well separate some things into their own functions, or at least separate the 1.2 -> 1.3 file upgrade thing

[nbitzz]

@Jack5079 later could you take over writing the routes for readFile? (and ofc writeFile once writeFileStream is available)

or don't idk lo

[nbitzz]

For writeFileStream to achieve our goals, it needs to respect backpressure. The issue is that I have no idea how I'd pull it off...

read emits drain on the writable, maybe, where write then pushes and returns false on the writable until read is called again..????

no idea how this would be structured though i'll have to do this at home

[nbitzz]

i think we need to write our.own formdata streaming parser thingyand alot of other stufff 🙃 working on this will be fun!!!! :DDD

uhhi uess il lkidna need to rsemisorta rethink-ish the writefile thing how i had it in mind previously coming back to this... this codebase is a mess osoim gonna have to change a lot abougt it tooooooo

[nbitzz]

Files.readFileStream is unreadable and confusing i'll have to clean it up heavily lmao

[nbitzz]

meowmeowmeowmeowmeowmrowmrowmeowmreowmrowmrowmrowmeowmeowmeowmroowIIii think i know how i'll be pulling off upload streaming now finallymrowmrowmrowmeowmrowmrowmeowmreowi will do it when i canmeowmrowmrowmrowmrowmeowmrowmreowmeow

[nbitzz]

Should probably redo our current lock system where we just throw a "reserved" in there... basically just separate it into some array or object? idk

[soins1]

this is pretty skibidi if i do say so myself

[Jack5079]

skibidi

@beebls Hello Beebles

[nbitzz]

Hello Beebles

???

[retree1]

Hello Beebles

???

:3

[nbitzz]

Hello Beebles

???

:3

meow :3 mrowmrowmrowmrowmrowmreow

[nbitzz]

Fuck it. Personal deadline of early April to wrap up the APIv1 branch

[nbitzz]

i need to check if any currently uploading messages get canceled if the upload is aborted

[nbitzz]

Server now starts on Hono! I do still want to redo readFileStream partially - it's a mess + also may waste requests with ranges Also still need to write our own streaming form-data parser Also just general cleanup, and of course api-v1 itself

[nbitzz]

Finish all of this, though, and after, uh... five months, we'll finally be clear for beta

[nbitzz]

Oh, right, and make UploadStream errors propagate through its child streams

[nbitzz]

@Jack5079 any idea how we could provide support for all routes (/file/:fileId, /:fileId. and /cpt/:fileId/*)? Maybe we should drop /cpt, I don't think anyone uses it like at all

[nbitzz]

Maybe we don't need to write our own formdata parser, let's just try to use formidable first

[nbitzz]

writing this is going to be hellll oh my godd :sob: my current idea is that if there's no full boundary in the chunk but the chunk ends in a part of the boundary then buffer some more until we can detect whether or not it is the boundary

[nbitzz]

I'll probably work on both the parser and api-v1 at the same time

[soins1]

Can i just say that i have no fucking idea what the hell is happening

[nbitzz]

Me neither

[nbitzz]

upload streaming + forms is like . actually done just browsers seem to SUCK why is it aborting the requeeessttt

[nbitzz]

@Jack5079 Shit, you already know I'm obsessed with it...*burp* I can't get enough. *sniffle* Back with another milk-HEEEEELP... HEEEELP.... HEEEEELP MEEE... HEEEEELP

[nbitzz]

Okay looks like i broke the range header i'll fix it tomorrow

[nbitzz]

at least i don't think it does

[nbitzz]

meow! meow! mreoow!! nya!~

[nbitzz]

later ill check if its my fault or honos fault for backpressure notnworking insread of wasting time on that rn tho ill orobbaly api v1 first

[nbitzz]

When are we adding skibidi toilet to monofile

[nbitzz]

@Jack5079 Haiiii can you help me figure out why backpressure doesn't work on /file

[nbitzz]

Uhhh could you test that thanks