issues when scaling up machines on fly

Moorst commented 7 months ago

appreciate this is a fly.io related issue, so I have also posted in the fly community (https://community.fly.io/t/sqlite-error-when-attempting-to-update-records-with-multiple-machines/18837/4) but I thought raise it here in case it's a problem with the fly / litesfs config shipped with the stack.

I followed the deployment & database docs to get my app up and running on fly.io with litefs, but I run into issues as soon as I scale up to two machines. it looks like when I do so, writes are being attempted on the replica machine which is causing unexpected server errors.

I'm seeing fairly similar issues to the ones reported here https://github.com/epicweb-dev/epic-stack/issues/642 but not quite the same:

PrismaClientUnknownRequestError

Invalid `prisma.user.update()` invocation:

Error occurred during query execution:
ConnectorError(ConnectorError { user_facing_error: None, kind: QueryError(SqliteError { extended_code: 778, message: Some("disk I/O error") }), transient: false })

when diving into the app logs I can see some wal error:

2024-03-20 13:14:36.302  [INFO] lhr ...3d708 level=INFO msg="fuse: write(): wal error: read only replica

which is occurring on the replica machine (...3d708).

for context this is occurring when attempting to update a user from within an action:

export async function loader({ request }: LoaderFunctionArgs) {
    ...

    const customer = await createStripeCustomer({ email, name })
    if (!customer) throw new Error('Unable to create Stripe Customer.')

    // issue occurs here
    await updateUserById(user.id, { customerId: customer.id })

        ...
}

is there some configuration missing that would ensure all writes are directed to the primary?

kentcdodds commented 7 months ago

We automatically redirect non-GET requests to the primary. You're doing a write on a GET request which is not recommended in general for security reasons. Is there a reason you're doing this?

I'm any case, if you need to do this, then use ensurePrimary() from litefs-js and that will fix it. But you really should avoid writing to the database on GET requests (ChatGPT can explain why).

kentcdodds commented 7 months ago

To be clear, you said this is happening in an action, but the code you showed is in a loader. This would work fine in an action.

Moorst commented 7 months ago

Ahh ok makes sense - glad I posted here now!

I basically lifted this logic from the stripe stack (https://github.com/dev-xo/stripe-stack/blob/fcfa7432f7c89886cbab53311d023d4d63f58544/app/routes/_layout%2B/account.tsx#L37), where it will redirect to this loader after account creation to assign the stripe customer id.

I'll modify this to handle it in a post action instead.

Appreciate it Kent! Love the work you've done with Remix & this stack 🔥

kentcdodds commented 7 months ago

Sure thing! Glad to have that worked out. It would be cool if we could improve the error message there... I'll bet it's possible with Prisma client extensions.