chiselstrike / chiselstrike

ChiselStrike abstracts common backends components like databases and message queues, and let you drive them from a convenient TypeScript business logic layer
https://chiselstrike.com
Apache License 2.0
1.08k stars 39 forks source link

INSERT to SQLite taking very long time #872

Closed penberg closed 2 years ago

penberg commented 2 years ago

I am running concurrency.deno test manually against 5445fe5a58c9e3668323644f236565297dabf4c0 (no transaction references held error anymore, btw) and seeing SQLite insertions take a very long time:

WARN - INSERT INTO "ty_Person_FDB1F3D868194BCC9E00D030DF061569" ("id","first_name","last_name","age","human","height") …; rows: 0, elapsed: 5.260s

INSERT INTO
  "ty_Person_FDB1F3D868194BCC9E00D030DF061569" (
    "id",
    "first_name",
    "last_name",
    "age",
    "human",
    "height"
  )
VALUES
  ($1, $2, $3, $4, $5, $6) ON CONFLICT (id) DO
UPDATE
SET
  "id" = $1,
  "first_name" = $2,
  "last_name" = $3,
  "age" = $4,
  "human" = $5,
  "height" = $6
WHERE
  "ty_Person_FDB1F3D868194BCC9E00D030DF061569"."id" = $1 RETURNING *

To reproduce the problem, copy examples/person.ts to models:

import { labels } from "@chiselstrike/api";

export class Person extends Chisel.ChiselEntity {
  first_name: string = "";
  @labels("pii") last_name: string = "";
  age: number = 0;
  human: boolean = false;
  height: number = 1;
}

export class Position extends Chisel.ChiselEntity {
  title: string = "title";
}

and examples/store.js to endpoints:

export default async function chisel(req) {
    if (req.method == 'POST') {
        const payload = await req.json();

        const person = Person.build(payload);
        await person.save();
        return new Response('ok');
    }
    return new Response('ignored');
}

Then run the following load generator script:

#!/bin/bash

export CURL=curl
export CHISELD_HOST=localhost:8080

for t in {1..10}; do
  for i in {1..10}; do
    $CURL --data '{
        "first_name":"Glauber",
        "last_name":"Costa",
        "age": 666,
        "human": true,
        "height": 10.01
    }' -o - $CHISELD_HOST/dev/store &
  done
  sleep 1
done

wait

I suspect this might be related to the issue of exhausting connection pool with YugaByte: https://github.com/chiselstrike/chiselstrike/issues/849

penberg commented 2 years ago

@espindola FYI. I will re-test once we have the request context issue fixed, because perhaps we're using the wrong transaction or something.

glommer commented 2 years ago

@penberg please retest

penberg commented 2 years ago

Fixed by 99b98c6ccf01437f22b8d1ea77972b5c9fce2bfb