Improve prerendering logs fetching

[qtgye]

Goal

To be able to to provide a realtime logging during the prerender process. That is, when the script server triggers the prerender on the main server, it should be able to receive logs incrementally, while the prerenderer is still running. This allows the logs to saved onto the script server itself in a realtime manner. This is achievable by leveraging ReadableStreams from the fetch's Response.body.

Tasks

Main Server

• [ ] Prepare the endpoint for the prerenderer, which applies the incremental stream sending:

  // Sample streamable endpoint
  static async test(req, res) {
    let tries = 3;
  
    while (tries--) {
      await new Promise((res) => setTimeout(res, 1000));
      res.write("Some data " + tries);
    }
  
    res.end();
  }

Script Server

• [ ] Prepare the process that will trigger the prerender and incrementally read the stream:

  // Sample only
  (async () => {
  const response = await fetch(MAIN_SERVER_ENDPOINT);
  const reader = response.body.getReader();
  
  while(true) {
    const {done, value} = await reader.read();
  
    const enc = new TextDecoder("utf-8");
  
    console.log(enc.decode(value.buffer));
  
    if (done) {
      break;
    }
  }    
  })();

[qtgye]

Findings

While streaming works fine on using setInterval (Script Server able to receive logs one by one), it doesn't work on the actual Prerender process. MIght have something to do with each log's interval, which in prerenderer its almost always instant.

Dropping this implementation, and having the Scripts Server wait for the success response instead.