Append to file without reading

https://github.com/alexhkurz/Chrome-extension/blob/757ffc9872ca51807c5facd45d54b068bb9a3a0e/server/server.js#L43-L65

This may lead to memory problems, if we end up with history files with thousands of objects (mine is already at >2000), this would require reading and parsing the whole file into a javascript object for every event. Instead, we can just overwrite the '\n]' at the end of the file with with `,\n${stringifiedEvent}\n]}. Haven't tested, but should be something like this:

fs.fstat(fd, (err, stats) => {
      if (err) {
        console.error(err);
        fs.close(fd, () => {}); // Close the file descriptor
        res.sendStatus(500);
        return;
      }

      // Calculate the position where we want to start writing (length - 2)
      const fileSize = stats.size;
      const position = fileSize - 2;

      // Prepare the data to write (overwriting last `\n]` and appending the new event)
      const newData = `,\n${stringifiedEvent}\n]`;

      // Write at the calculated position (overwriting "\n]")
      fs.write(fd, newData, 0, newData.length, position, (err) => {
        if (err) {
          console.error(err);
          res.sendStatus(500);
        } else {
          res.sendStatus(200);
        }

        // Close the file descriptor
        fs.close(fd, () => {});
      });
    });

thanks, I agree that there may be memory problems ... I didnt think yet about what would be the best way to handle that, so I postponed ...

On Sep 7, 2024, at 8:58 PM, Luke Burns @.***> wrote:

https://github.com/alexhkurz/Chrome-extension/blob/757ffc9872ca51807c5facd45d54b068bb9a3a0e/server/server.js#L43-L65 This may lead to memory problems, if we end up with history files with thousands of objects (mine is already at >2000), this would require reading and parsing the whole file into a javascript object. Instead, we can just overwrite the '\n]' at the end of the file with with `,\n${stringifiedEvent}\n]}. Haven't tested, but should be something like this: fs.fstat(fd, (err, stats) => { if (err) { console.error(err); fs.close(fd, () => {}); // Close the file descriptor res.sendStatus(500); return; }

// Calculate the position where we want to start writing (length - 2) const fileSize = stats.size; const position = fileSize - 2;

// Prepare the data to write (overwriting last ]\n and appending the new event) const newData = ,\n${newEvent}\n];

// Write at the calculated position (overwriting "]\n") fs.write(fd, newData, 0, newData.length, position, (err) => { if (err) { console.error(err); res.sendStatus(500); } else { res.sendStatus(200); }

// Close the file descriptor fs.close(fd, () => {}); }); });
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

alexhkurz / Chrome-extension

Append to file without reading #3