Scale: allow configuration of read replicas for RO queries to Postgres

[EternalDeiwos]

• [x] Did you check to see if this issue already exists?
• [x] Is this only a single feature request? Do not put multiple feature requests in one issue.
• [ ] Is this a question or discussion? Don't use this, use https://lemmy.ml/c/lemmy_support.
• [ ] Is this a UI / front end issue? Use the lemmy-ui repo.

Describe the feature request below

Disclaimer: I've been using Lemmy for about half a day now — I have very little insight into how the application scales i.t.o. resource usage. This is in response to a post by @Nutomic.

The incoming users from the Reddit exodus are going to put all instances under strain, not just lemmy.ml. r/selfhosted alone is currently around ~250k with 1.2k active users; r/askreddit is >41m users with >78k active. As the global user count grows, we will either need to keep sharding such that large communities have dedicated instances – and even then we will hit a ceiling – or we need to make instances scale better.

My suggestion is to allow read-only queries to be offloaded to read-replicas of the primary database. Not only does this increase query capacity, it also allows the master node to be reserved for database mutation operations, and would support fail-over when deploying Postgres in a clustered configuration.

[NightA]

While this is a good idea, i think it's only half of the solution.

If the main concern here is that individual instances will become overloaded due to a concentrated influx of users, some form of a scalable write-cache will also be needed down the line.

As is, read-only replica instances would only address lurkers for the most part and not necessarily the active userbase. They might ensure any federated instances could still read and display remote content, but it won't factor in for new user registrations or for those who will become "stuck" on overloaded instance as (as far as i am aware) there's currently no option to seemingly migrate users between federated instances.

[EternalDeiwos]

While this is a good idea, i think it's only half of the solution.

Absolutely agreed. This wasn’t intended to be a complete solution to scale, just one thing that could alleviate some pressure that is relatively easy to do in the short term.

To scale any larger you’d need larger write capacity than vertical scale can handle… but that will be quite a large undertaking.

[jbenguira]

What about using clickhouse instead of postgres? It can ingest easily 100x more writes per second

[mikelococo]

For any enterprising lemmy admins who want to try to prove this out in advance of it getting developer attention, it's possible to load-balance among read-replicas using Pgpool-ii even if the application doesn't natively support PG connections to the write leader and to read-replicas... as is currently the case for Lemmy.

• https://www.pgpool.net/docs/42/en/html/runtime-config-load-balancing.html
• https://www.pgpool.net/docs/42/en/html/read-query-load-balancing.html

To do this, one would set up postgres to perform streaming replication to one or more separate read-replicas. Then set up Pgpool-ii between lemmy and postgres as a proxy layer. Configure Pgpool-ii statement-level load-balancing, and it will send SELECT statements that meet certain criteria to the read-replica rather than the primary write leader.

I'll caveat this to say I haven't tested it and don't run a big lemmy instance. There are many things that could go wrong, some usage patterns aren't compatible with statement-level load-balancing, not every SELECT can be routed to the replica. It may fail entirely and it may deliver subpar performance improvements even if it works. But it's interesting as an avenue for exploring database scaling that doesn't require application code changes, and as such can be tested and... if the results are positive... even put into use without waiting for review from core-developers or a release cycle.

[vijaykramesh]

is anybody working on this? i have a branch that is at least creating the read only pool and starting to push the read only queries to that.... but not sure if it's actually something any instance providers would want

[vijaykramesh]

https://github.com/vijaykramesh/lemmy/commit/91a133a2f5f6c42c73cb815161a97b4cc253dd11 just so I don't lose track myself :)

[vijaykramesh]

@Nutomic what's the best way to discuss an approach here? I dunno if this would wait for 0.19, at the very least I'll update my PoC to the 0.18 branch

[tbe]

PostgreSQL DBA here. I second the PGPool approach, suggested by @mikelococo

Besides the fact, that this can be used in the current state of lemmy, it has the huge advantage, that if paired with something like patroni for example, a downtime of the master does not require immediate attention by someone (patroni switches to a new master, pgpool follows, nearly all connections are kept intact as far as the database layer of lemmy is concerned ).

And just for completeness: Ckickhouse is NOT a drop in replacement for a classic RDBMS database. For example it is bad, really bad, at mutating data. While it is true, that the select and append performance is great, updating datasets is terribly slow.

[jbenguira]

PostgreSQL DBA here. I second the PGPool approach, suggested by @mikelococo

Besides the fact, that this can be used in the current state of lemmy, it has the huge advantage, that if paired with something like patroni for example, a downtime of the master does not require immediate attention by someone (patroni switches to a new master, pgpool follows, nearly all connections are kept intact as far as the database layer of lemmy is concerned ).

And just for completeness: Ckickhouse is NOT a drop in replacement for a classic RDBMS database. For example it is bad, really bad, at mutating data. While it is true, that the select and append performance is great, updating datasets is terribly slow.

Do we need so many mutations in Lemmy? What is the biggest part of the queries? SELECT? INSERTS? UPDATES? I suggested ClickHouse because on another project I worked for we moved from Postgres to ClickHouse and performance was amazing, we where limited to 8K inserts per second with Postgres and we reached 155K inserts per second with ClickHouse ... on the same VM of course, I'm not talking about a cluster but a single VM. Also SELECT queries where easily 100x faster than postgres for the same dataset and same WHERE conditions.

I don't want to start a DB war or anything, Postgres is really great for a lot of use cases. Just wanted to suggest something and share my experience.

[tbe]

I don't want to start a DB war or anything, Postgres is really great for a lot of use cases. Just wanted to suggest something and share my experience.

Discussions around this are important. I'm with you.

From my perspective, updates are important. What about updated, federated comments. Many people only see a type after they send der post/comment and then update it.

Or SPAM removal, deletion of community and user data.

There are things that can be done with postgresql to improve the write performance by a lot. That requires postgresql knowledge and tuning. But for Lemmy, a general set of configuration paramwters can be provided. (checkpoint and wal settings, random page costs, vacuum tuning, there is a bunch of them)

[cloventt]

For what it's worth, I second the pgpool solution already mentioned. For admins that are running on AWS and hoping to use read-replicas to improve query performance and availability, there is already a guide on the AWS blog for how to achieve this.

I would oppose adding this as a "native" solution in lemmy code, as it would require maintaining multiple pools of PG clients and passing them around all over the app, and ensuring that the maintainers always use the correct client pool for read vs write queries. That seems to me like a quick way to make the code extremely messy for a feature that only a very small number of admins are likely to use.

[alexandruchircu]

For what it's worth, I have a Bitnami PostgreSQL HA deployed in a cluster, and am testing Lemmy (connected to the postgresql-ha-pgpool service).

`Record not found` when `UPDATE "post" SET "ap_id" = $1 WHERE ("post"."id" = $2)`

``` 2023-07-04T12:27:06+03:00 2023-07-04T09:27:06.785726Z INFO actix_web::middleware::logger: 2a02:2f0d:4403:3e00:802a:3238:e7a5:ca16 'GET /api/v3/user/report_count?auth=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOjMsImlzcyI6InBsLmNoaXJjdS5tZSIsImlhdCI6MTY4ODQ1NTU5OH0.28WLP1PBcHSFjGKZGovEJQTnQpgjjy63bJLj85xzFpc HTTP/1.1' 200 38 'https://my.domain.name/create_post?communityId=4' 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36' 0.014271 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.828475Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}:perform{self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }}:local_user_view_from_jwt: tokio_postgres::query: executing statement s120 with parameters: [] 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.830997Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}:perform{self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }}:local_user_view_from_jwt: tokio_postgres::query: executing statement s117 with parameters: [ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 23, array_oid: 1007 })), Some([0, 0, 0, 3])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 20, array_oid: 1016 })), Some([0, 0, 0, 0, 0, 0, 0, 1]))] 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.832744Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}:perform{self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }}: tokio_postgres::query: executing statement s104 with parameters: [] 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.835328Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}:perform{self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }}: tokio_postgres::query: executing statement s110 with parameters: [ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 20, array_oid: 1016 })), Some([0, 0, 0, 0, 0, 0, 0, 1]))] 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.837078Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}:perform{self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }}:check_community_ban: tokio_postgres::query: executing statement s120 with parameters: [] 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.839485Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}:perform{self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }}:check_community_ban: tokio_postgres::prepare: preparing query s130 with types [Int4, Int4, Int8]: SELECT "community"."id", "community"."name", "community"."title", "community"."description", "community"."removed", "community"."published", "community"."updated", "community"."deleted", "community"."nsfw", "community"."actor_id", "community"."local", "community"."private_key", "community"."public_key", "community"."last_refreshed_at", "community"."icon", "community"."banner", "community"."followers_url", "community"."inbox_url", "community"."shared_inbox_url", "community"."hidden", "community"."posting_restricted_to_mods", "community"."instance_id", "community"."moderators_url", "community"."featured_url", "person"."id", "person"."name", "person"."display_name", "person"."avatar", "person"."banned", "person"."published", "person"."updated", "person"."actor_id", "person"."bio", "person"."local", "person"."private_key", "person"."public_key", "person"."last_refreshed_at", "person"."banner", "person"."deleted", "person"."inbox_url", "person"."shared_inbox_url", "person"."matrix_user_id", "person"."admin", "person"."bot_account", "person"."ban_expires", "person"."instance_id" FROM (("community_person_ban" INNER JOIN "community" ON ("community_person_ban"."community_id" = "community"."id")) INNER JOIN "person" ON ("community_person_ban"."person_id" = "person"."id")) WHERE (("community_person_ban"."community_id" = $1) AND ("community_person_ban"."person_id" = $2)) ORDER BY "community_person_ban"."published" LIMIT $3 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.842499Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}:perform{self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }}:check_community_ban: tokio_postgres::query: executing statement s130 with parameters: [ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 23, array_oid: 1007 })), Some([0, 0, 0, 4])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 23, array_oid: 1007 })), Some([0, 0, 0, 4])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 20, array_oid: 1016 })), Some([0, 0, 0, 0, 0, 0, 0, 1]))] 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.844731Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}:perform{self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }}:check_community_deleted_or_removed: tokio_postgres::query: executing statement s104 with parameters: [] 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.847003Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}:perform{self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }}:check_community_deleted_or_removed: tokio_postgres::prepare: preparing query s131 with types [Int4, Int8]: SELECT "community"."id", "community"."name", "community"."title", "community"."description", "community"."removed", "community"."published", "community"."updated", "community"."deleted", "community"."nsfw", "community"."actor_id", "community"."local", "community"."private_key", "community"."public_key", "community"."last_refreshed_at", "community"."icon", "community"."banner", "community"."followers_url", "community"."inbox_url", "community"."shared_inbox_url", "community"."hidden", "community"."posting_restricted_to_mods", "community"."instance_id", "community"."moderators_url", "community"."featured_url" FROM "community" WHERE ("community"."id" = $1) LIMIT $2 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.849208Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}:perform{self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }}:check_community_deleted_or_removed: tokio_postgres::query: executing statement s131 with parameters: [ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 23, array_oid: 1007 })), Some([0, 0, 0, 4])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 20, array_oid: 1016 })), Some([0, 0, 0, 0, 0, 0, 0, 1]))] 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.851486Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}:perform{self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }}: tokio_postgres::query: executing statement s120 with parameters: [] 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.854126Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}:perform{self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }}: tokio_postgres::prepare: preparing query s132 with types [Int4, Int8]: SELECT "community"."id", "community"."name", "community"."title", "community"."description", "community"."removed", "community"."published", "community"."updated", "community"."deleted", "community"."nsfw", "community"."actor_id", "community"."local", "community"."private_key", "community"."public_key", "community"."last_refreshed_at", "community"."icon", "community"."banner", "community"."followers_url", "community"."inbox_url", "community"."shared_inbox_url", "community"."hidden", "community"."posting_restricted_to_mods", "community"."instance_id", "community"."moderators_url", "community"."featured_url" FROM "community" WHERE ("community"."id" = $1) LIMIT $2 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.856983Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}:perform{self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }}: tokio_postgres::query: executing statement s132 with parameters: [ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 23, array_oid: 1007 })), Some([0, 0, 0, 4])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 20, array_oid: 1016 })), Some([0, 0, 0, 0, 0, 0, 0, 1]))] 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.859576Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}:perform{self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }}: tokio_postgres::query: executing statement s104 with parameters: [] 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.861630Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}:perform{self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }}: tokio_postgres::prepare: preparing query s133 with types [Int4, Int4]: SELECT EXISTS (SELECT "community_language"."id", "community_language"."community_id", "community_language"."language_id" FROM "community_language" WHERE (("community_language"."language_id" = $1) AND ("community_language"."community_id" = $2))) 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.863410Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}:perform{self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }}: tokio_postgres::query: executing statement s133 with parameters: [ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 23, array_oid: 1007 })), Some([0, 0, 0, 37])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 23, array_oid: 1007 })), Some([0, 0, 0, 4]))] 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.865031Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}:perform{self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }}: tokio_postgres::query: executing statement s120 with parameters: [] 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.867653Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}:perform{self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }}: tokio_postgres::prepare: preparing query s134 with types [Text, Int4, Int4, Bool, Text, Int4, Text, Int4, Int4, Bool, Text, Int4]: INSERT INTO "post" ("name", "creator_id", "community_id", "nsfw", "url", "body", "removed", "locked", "updated", "published", "deleted", "embed_title", "embed_description", "embed_video_url", "thumbnail_url", "ap_id", "local", "language_id", "featured_community", "featured_local") VALUES ($1, $2, $3, $4, DEFAULT, $5, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT, $6, DEFAULT, DEFAULT) ON CONFLICT ("ap_id") DO UPDATE SET "name" = $7, "creator_id" = $8, "community_id" = $9, "nsfw" = $10, "body" = $11, "language_id" = $12 RETURNING "post"."id", "post"."name", "post"."url", "post"."body", "post"."creator_id", "post"."community_id", "post"."removed", "post"."locked", "post"."published", "post"."updated", "post"."deleted", "post"."nsfw", "post"."embed_title", "post"."embed_description", "post"."thumbnail_url", "post"."ap_id", "post"."local", "post"."embed_video_url", "post"."language_id", "post"."featured_community", "post"."featured_local" 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.870573Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}:perform{self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }}: tokio_postgres::query: executing statement s134 with parameters: [ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 25, array_oid: 1009 })), Some([84, 101, 115, 116, 55])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 23, array_oid: 1007 })), Some([0, 0, 0, 4])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 23, array_oid: 1007 })), Some([0, 0, 0, 4])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 16, array_oid: 1000 })), Some([0])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 25, array_oid: 1009 })), Some([104, 114, 101, 107, 106, 104, 102, 103, 100, 115])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 23, array_oid: 1007 })), Some([0, 0, 0, 37])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 25, array_oid: 1009 })), Some([84, 101, 115, 116, 55])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 23, array_oid: 1007 })), Some([0, 0, 0, 4])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 23, array_oid: 1007 })), Some([0, 0, 0, 4])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 16, array_oid: 1000 })), Some([0])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 25, array_oid: 1009 })), Some([104, 114, 101, 107, 106, 104, 102, 103, 100, 115])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 23, array_oid: 1007 })), Some([0, 0, 0, 37]))] 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.895158Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}:perform{self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }}: tokio_postgres::query: executing statement s104 with parameters: [] 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.898438Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}:perform{self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }}: tokio_postgres::prepare: preparing query s135 with types [Text, Int4]: UPDATE "post" SET "ap_id" = $1 WHERE ("post"."id" = $2) RETURNING "post"."id", "post"."name", "post"."url", "post"."body", "post"."creator_id", "post"."community_id", "post"."removed", "post"."locked", "post"."published", "post"."updated", "post"."deleted", "post"."nsfw", "post"."embed_title", "post"."embed_description", "post"."thumbnail_url", "post"."ap_id", "post"."local", "post"."embed_video_url", "post"."language_id", "post"."featured_community", "post"."featured_local" 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.900243Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}:perform{self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }}: tokio_postgres::query: executing statement s135 with parameters: [ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 25, array_oid: 1009 })), Some([104, 116, 116, 112, 115, 58, 47, 47, 112, 108, 46, 99, 104, 105, 114, 99, 117, 46, 109, 101, 47, 112, 111, 115, 116, 47, 49, 56])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 23, array_oid: 1007 })), Some([0, 0, 0, 18]))] 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.902702Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635}: actix_web::middleware::logger: Error in response: LemmyError { message: Some("couldnt_create_post"), inner: Record not found, context: SpanTrace [{ target: "lemmy_api_crud::post::create", name: "perform", fields: "\u{1b}[3mself\u{1b}[0m\u{1b}[2m=\u{1b}[0mCreatePost { name: \"Test7\", community_id: CommunityId(4), url: None, body: Some(\"hrekjhfgds\"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }", file: "crates/api_crud/src/post/create.rs", line: 45 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "\u{1b}[3mhttp.method\u{1b}[0m\u{1b}[2m=\u{1b}[0mPOST \u{1b}[3mhttp.scheme\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"https\" \u{1b}[3mhttp.host\u{1b}[0m\u{1b}[2m=\u{1b}[0mmy.domain.name \u{1b}[3mhttp.target\u{1b}[0m\u{1b}[2m=\u{1b}[0m/api/v3/post \u{1b}[3motel.kind\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"server\" \u{1b}[3mrequest_id\u{1b}[0m\u{1b}[2m=\u{1b}[0mbbe11304-d5cc-49b0-a5ae-672f8cfa9635", file: "src/root_span_builder.rs", line: 16 }] } 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.903827Z WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: couldnt_create_post: Record not found 2023-07-04T12:27:08+03:00 0: lemmy_api_crud::post::create::perform 2023-07-04T12:27:08+03:00 with self=CreatePost { name: "Test7", community_id: CommunityId(4), url: None, body: Some("hrekjhfgds"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive } 2023-07-04T12:27:08+03:00 at crates/api_crud/src/post/create.rs:45 2023-07-04T12:27:08+03:00 1: lemmy_server::root_span_builder::HTTP request 2023-07-04T12:27:08+03:00 with http.method=POST http.scheme="https" http.host=my.domain.name http.target=/api/v3/post otel.kind="server" request_id=bbe11304-d5cc-49b0-a5ae-672f8cfa9635 http.status_code=404 otel.status_code="OK" 2023-07-04T12:27:08+03:00 at src/root_span_builder.rs:16 2023-07-04T12:27:08+03:00 LemmyError { message: Some("couldnt_create_post"), inner: Record not found, context: SpanTrace [{ target: "lemmy_api_crud::post::create", name: "perform", fields: "\u{1b}[3mself\u{1b}[0m\u{1b}[2m=\u{1b}[0mCreatePost { name: \"Test7\", community_id: CommunityId(4), url: None, body: Some(\"hrekjhfgds\"), honeypot: None, nsfw: Some(false), language_id: Some(LanguageId(37)), auth: Sensitive }", file: "crates/api_crud/src/post/create.rs", line: 45 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "\u{1b}[3mhttp.method\u{1b}[0m\u{1b}[2m=\u{1b}[0mPOST \u{1b}[3mhttp.scheme\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"https\" \u{1b}[3mhttp.host\u{1b}[0m\u{1b}[2m=\u{1b}[0mmy.domain.name \u{1b}[3mhttp.target\u{1b}[0m\u{1b}[2m=\u{1b}[0m/api/v3/post \u{1b}[3motel.kind\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"server\" \u{1b}[3mrequest_id\u{1b}[0m\u{1b}[2m=\u{1b}[0mbbe11304-d5cc-49b0-a5ae-672f8cfa9635 \u{1b}[3mhttp.status_code\u{1b}[0m\u{1b}[2m=\u{1b}[0m404 \u{1b}[3motel.status_code\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"OK\"", file: "src/root_span_builder.rs", line: 16 }] } 2023-07-04T12:27:08+03:00 2023-07-04T09:27:08.904271Z INFO actix_web::middleware::logger: 2a02:2f0d:4403:3e00:802a:3238:e7a5:ca16 'POST /api/v3/post HTTP/1.1' 404 31 'https://my.domain.name/create_post?communityId=4' 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36' 0.076115 2023-07-04T12:27:36+03:00 2023-07-04T09:27:36.930032Z DEBUG HTTP request{http.method=GET http.scheme="https" http.host=my.domain.name http.target=/api/v3/user/report_count otel.kind="server" request_id=4dfa2bb1-75c5-46a0-8892-e8fbf90b033b}:perform{self=GetReportCount { community_id: None, auth: Sensitive }}:local_user_view_from_jwt: tokio_postgres::query: executing statement s120 with parameters: [] 2023-07-04T12:27:36+03:00 2023-07-04T09:27:36.931245Z DEBUG HTTP request{http.method=GET http.scheme="https" http.host=my.domain.name http.target=/api/v3/user/report_count otel.kind="server" request_id=4dfa2bb1-75c5-46a0-8892-e8fbf90b033b}:perform{self=GetReportCount { community_id: None, auth: Sensitive }}:local_user_view_from_jwt: tokio_postgres::query: executing statement s117 with parameters: [ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 23, array_oid: 1007 })), Some([0, 0, 0, 3])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 20, array_oid: 1016 })), Some([0, 0, 0, 0, 0, 0, 0, 1]))] 2023-07-04T12:27:36+03:00 2023-07-04T09:27:36.932064Z DEBUG HTTP request{http.method=GET http.scheme="https" http.host=my.domain.name http.target=/api/v3/user/report_count otel.kind="server" request_id=4dfa2bb1-75c5-46a0-8892-e8fbf90b033b}:perform{self=GetReportCount { community_id: None, auth: Sensitive }}: tokio_postgres::query: executing statement s104 with parameters: [] 2023-07-04T12:27:36+03:00 2023-07-04T09:27:36.933398Z DEBUG HTTP request{http.method=GET http.scheme="https" http.host=my.domain.name http.target=/api/v3/user/report_count otel.kind="server" request_id=4dfa2bb1-75c5-46a0-8892-e8fbf90b033b}:perform{self=GetReportCount { community_id: None, auth: Sensitive }}: tokio_postgres::query: executing statement s108 with parameters: [ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 23, array_oid: 1007 })), Some([0, 0, 0, 4])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 16, array_oid: 1000 })), Some([0])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 20, array_oid: 1016 })), Some([0, 0, 0, 0, 0, 0, 0, 1]))] 2023-07-04T12:27:36+03:00 2023-07-04T09:27:36.934221Z DEBUG HTTP request{http.method=GET http.scheme="https" http.host=my.domain.name http.target=/api/v3/user/report_count otel.kind="server" request_id=4dfa2bb1-75c5-46a0-8892-e8fbf90b033b}:perform{self=GetReportCount { community_id: None, auth: Sensitive }}: tokio_postgres::query: executing statement s120 with parameters: [] 2023-07-04T12:27:36+03:00 2023-07-04T09:27:36.936095Z DEBUG HTTP request{http.method=GET http.scheme="https" http.host=my.domain.name http.target=/api/v3/user/report_count otel.kind="server" request_id=4dfa2bb1-75c5-46a0-8892-e8fbf90b033b}:perform{self=GetReportCount { community_id: None, auth: Sensitive }}: tokio_postgres::query: executing statement s126 with parameters: [ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 23, array_oid: 1007 })), Some([0, 0, 0, 4])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 16, array_oid: 1000 })), Some([0])), ToSqlHelper(PgTypeMetadata(Ok(InnerPgTypeMetadata { oid: 20, array_oid: 1016 })), Some([0, 0, 0, 0, 0, 0, 0, 1]))] 2023-07-04T12:27:36+03:00 2023-07-04T09:27:36.937397Z INFO actix_web::middleware::logger: 2a02:2f0d:4403:3e00:802a:3238:e7a5:ca16 'GET /api/v3/user/report_count?auth=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOjMsImlzcyI6InBsLmNoaXJjdS5tZSIsImlhdCI6MTY4ODQ1NTU5OH0.28WLP1PBcHSFjGKZGovEJQTnQpgjjy63bJLj85xzFpc HTTP/1.1' 200 38 'https://my.domain.name/create_post?communityId=4' 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36' 0.007589 ```

[dessalines]

This seems to mostly be deployment related, and should probably live in lemmy-docs for how to do something like this.