API: no huge request

[MrCocoNuat]

I didn't think i was going to get away with it anyway: Vercel limits function runtime to 10 seconds, and on first load, even local RepositoryReaders try to parse hundreds of MB of JSON (let alone Remotes making over 100 graphql calls and many http requests), and there is no chance of them finishing in 10 seconds

Options:

• Break up dataloading
  • but the JSON data is not organized by ids, so it is difficult to know which chunk to load without a human checked mapping from ids to contained files
  • or use a dedicated resumable dedicated data loading api mutation to run for 10 seconds at a time, it goes without saying that this is a real hack
• use a real database instead of relying on everything in memory
  • have to figure out integrations with Vercel to write to it in a way not limited by that 10 second serverless execution

[MrCocoNuat]

Vercel offers postgres, KV (redis), since input data is relational postgres would work but redis is attractive too.....

[MrCocoNuat]

staying in memory is pretty much shot dead by the 1GB limit anyway

[MrCocoNuat]

oh for the love of god... forget the big explorer pages covering every single hero or skill, they are blowing up the server