Open lmmendes opened 4 months ago
hm, I'm unable to reproduce this locally. In production though, please use the tagged release instead of master
which is bleeding edge.
Did the following:
docker pull listmonk/listmonk
> docker images listmonk/listmonk:latest
REPOSITORY TAG IMAGE ID CREATED SIZE
listmonk/listmonk latest b4602d730116 4 months ago 25.3MB
Run the docker-compose.yml using:
docker compose up demo-app
Still got the error in a fresh installation:
Screenshot from the logs tab:
Will try to run from released tag in a couple of hours.
Checkout of the v3.0.0 release
git checkout tags/v3.0.0 -b v3.0.0
Building of the application
make dist
CGO_ENABLED=0 go build -o listmonk -ldflags="-s -w -X 'main.buildString=v3.0.0 (#f9120d9 2024-06-18T13:36:57+0000)' -X 'main.versionString=v3.0.0'" cmd/*.go
Setting up the database:
listmonk-knadh (v3.0.0) ✔ ./listmonk --install
2024/06/18 14:37:30 main.go:102: v3.0.0 (#f9120d9 2024-06-18T13:36:57+0000)
2024/06/18 14:37:30 init.go:150: reading config: config.toml
2024/06/18 14:37:30 init.go:289: connecting to db: 0.0.0.0:9432/listmonk
** first time installation **
** IMPORTANT: This will wipe existing listmonk tables and types in the DB 'listmonk' **
continue (y/N)? y
2024/06/18 14:37:32 install.go:179: setup complete
2024/06/18 14:37:32 install.go:180: run the program and access the dashboard at localhost:9000
Running listmonk
hand having the same issue with the error fetching lists: pq: column "subscriber_count" does not exist
see the log:
➜ listmonk-knadh (v3.0.0) ✔ ./listmonk
2024/06/18 14:37:35 main.go:102: v3.0.0 (#f9120d9 2024-06-18T13:36:57+0000)
2024/06/18 14:37:35 init.go:150: reading config: config.toml
2024/06/18 14:37:35 init.go:289: connecting to db: 0.0.0.0:9432/listmonk
2024/06/18 14:37:35 init.go:618: media upload provider: filesystem
2024/06/18 14:37:35 init.go:541: loaded email (SMTP) messenger: username@smtp.yoursite.com
⇨ http server started on 127.0.0.1:9000
2024/06/18 14:37:42 lists.go:51: error fetching lists: pq: column "subscriber_count" does not exist
2024/06/18 14:37:43 lists.go:51: error fetching lists: pq: column "subscriber_count" does not exist
2024/06/18 14:37:44 lists.go:51: error fetching lists: pq: column "subscriber_count" does not exist
@knadh were you able to reproduce the issue?
The following query seemed to fix the issue on my side, what do you think about it?
-- name: query-lists
WITH ls AS (
SELECT COUNT(*) OVER () AS total, lists.* FROM lists WHERE
CASE
WHEN $1 > 0 THEN id = $1
WHEN $2 != '' THEN uuid = $2::UUID
WHEN $3 != '' THEN to_tsvector(name) @@ to_tsquery ($3)
ELSE TRUE
END
AND ($4 = '' OR type = $4::list_type)
AND ($5 = '' OR optin = $5::list_optin)
AND (CARDINALITY($6::VARCHAR(100)[]) = 0 OR $6 <@ tags)
OFFSET $7 LIMIT (CASE WHEN $8 < 1 THEN NULL ELSE $8 END)
),
statuses AS (
SELECT
list_id,
COALESCE(JSONB_OBJECT_AGG(status, subscriber_count) FILTER (WHERE status IS NOT NULL), '{}') AS subscriber_statuses
FROM mat_list_subscriber_stats
GROUP BY list_id
),
status_sums AS (
SELECT
list_id,
SUM((value)::INTEGER) AS subscriber_count
FROM statuses, jsonb_each_text(statuses.subscriber_statuses)
GROUP BY list_id
)
SELECT ls.*, COALESCE(ss.subscriber_statuses, '{}') AS subscriber_statuses
FROM ls
LEFT JOIN statuses ss ON (ls.id = ss.list_id)
LEFT JOIN status_sums sts ON (ls.id = sts.list_id)
ORDER BY %order%;
This section is what I added to the original query-lists
query:
status_sums AS (
SELECT
list_id,
SUM((value)::INTEGER) AS subscriber_count
FROM statuses, jsonb_each_text(statuses.subscriber_statuses)
GROUP BY list_id
)
Edit: Clarified what was the change introduced in the original query-lists
query.
@knadh were you able to reproduce the issue?
Strange. Unable to reproduce this on master
or v3.0.0
. I'm running Postgres v14.10.
@ (~listmonk) ● (HEAD detached at v3.0.0)
$ make build
CGO_ENABLED=0 go build -o listmonk -ldflags="-s -w -X 'main.buildString=v3.0.0 (#f9120d9 2024-06-19T05:09:51+0000)' -X 'main.versionString=v3.0.0'" cmd/*.go
@ (~listmonk) ● (HEAD detached at v3.0.0)
$ ./listmonk --install --yes
2024/06/19 10:39:57 main.go:102: v3.0.0 (#f9120d9 2024-06-19T05:09:51+0000)
2024/06/19 10:39:57 init.go:150: reading config: config.toml
2024/06/19 10:39:57 init.go:289: connecting to db: localhost:9432/listmonk
2024/06/19 10:39:57 init.go:206: unable to initialize embedded filesystem (no ID found in the file). Using local filesystem
2024/06/19 10:39:57 init.go:228: loading i18n files from: i18n
2024/06/19 10:39:57 init.go:248: loading static files from: static
** first time installation **
** IMPORTANT: This will wipe existing listmonk tables and types in the DB 'listmonk' **
2024/06/19 10:39:57 install.go:179: setup complete
2024/06/19 10:39:57 install.go:180: run the program and access the dashboard at 0.0.0.0:9000
@ (~listmonk) ● (HEAD detached at v3.0.0)
$ ./listmonk
2024/06/19 10:39:59 main.go:102: v3.0.0 (#f9120d9 2024-06-19T05:09:51+0000)
2024/06/19 10:39:59 init.go:150: reading config: config.toml
2024/06/19 10:39:59 init.go:289: connecting to db: localhost:9432/listmonk
2024/06/19 10:39:59 init.go:206: unable to initialize embedded filesystem (no ID found in the file). Using local filesystem
2024/06/19 10:39:59 init.go:228: loading i18n files from: i18n
2024/06/19 10:39:59 init.go:248: loading static files from: static
2024/06/19 10:39:59 init.go:618: media upload provider: filesystem
2024/06/19 10:39:59 init.go:541: loaded email (SMTP) messenger: username@smtp.yoursite.com
⇨ http server started on [::]:9000
PS: Tried the same with make dist
as well.
@knadh Here are the tests that I done:
Version of the local repository:
commit f9120d9b80b1d0d86f27e3f23edfb2ffe6f5b091 (HEAD -> v3.0.0, tag: v3.0.0)
Built version of monklist
:
monklist version=v3.0.0 (#f9120d9 2024-06-18T13:36:57+0000)
PostgreSQL Docker Image | PostgreSQL Version | Status |
---|---|---|
postgres:13-alpine | PostgreSQL 13.13 on aarch64-unknown-linux-musl, compiled by gcc (Alpine 13.2.1_git20231014) 13.2.1 20231014, 64-bit | Issue found |
postgres:14.10-alpine | PostgreSQL 14.10 on aarch64-unknown-linux-musl, compiled by gcc (Alpine 13.2.1_git20231014) 13.2.1 20231014, 64-bit | Issue found |
postgres:latest | PostgreSQL 16.1 (Debian 16.1-1.pgdg120+1) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit | issue found |
To run the database I updated the docker-compose.yml
and simply changed the database version:
x-db-defaults: &db-defaults
image: postgres:14.10-alpine <--- changed this line with the image from the table above
ports:
- "9432:5432"
networks:
- listmonk
environment:
- POSTGRES_PASSWORD=listmonk
- POSTGRES_USER=listmonk
- POSTGRES_DB=listmonk
restart: unless-stopped
healthcheck:
test: ["CMD-SHELL", "pg_isready -U listmonk"]
interval: 10s
timeout: 5s
retries: 6
Before each test destroyed the Docker image including data and started a new instance using:
docker compose rm demo-db -f
docker compose up demo-db
Installed the new database models and run the same build using:
./listmonk --install
./listmonk
All runs gave the same issue when trying to sort the "Subscriptions" in the List view:
> listmonk-knadh (v3.0.0) ✗ ./listmonk
2024/06/19 09:29:37 main.go:102: v3.0.0 (#f9120d9 2024-06-18T13:36:57+0000)
2024/06/19 09:29:37 init.go:150: reading config: config.toml
2024/06/19 09:29:37 init.go:289: connecting to db: 0.0.0.0:9432/listmonk
2024/06/19 09:29:37 init.go:618: media upload provider: filesystem
2024/06/19 09:29:37 init.go:541: loaded email (SMTP) messenger: username@smtp.yoursite.com
⇨ http server started on 127.0.0.1:9000
2024/06/19 09:29:40 lists.go:51: error fetching lists: pq: column "subscriber_count" does not exist
2024/06/19 09:29:40 lists.go:51: error fetching lists: pq: column "subscriber_count" does not exist
This is really strange. I compiled the exact same commit you've listed above with make dist
, ran it against Postgres 14.10-alpine, installed, ran, and visited the subscribers page. It works fine.
$ ./listmonk --version
v3.0.0 (#f9120d9 2024-06-19T11:09:07+0000)
Also, v3.0.0 is the last prod version that's been running in tens of thousands of installations since its release.
Stumped.
Hey there,
We're also encountering the same issue with the "subscriber_count" field not existing when trying to sort by "Subscribers" in the Lists view.
Description of the bug and steps to reproduce:
On the "Lists" page, when clicking to sort the "Subscribers" column, we receive the following error:
error fetching lists: column "subscriber_count" does not exist
Steps we have tried to resolve the issue:
Initial Setup:
git clone https://github.com/knadh/listmonk.git
docker compose up
Despite following these steps, we encountered the issue.
Running Binary Directly:
Architecture Considerations:
Branch and Version:
Environment Details:
Given that the issue is reproducible across different environments and architectures on our end, we're wondering if there's a specific configuration or step we might be missing.
@knadh I'm also flabbergasted with the issue and I feel your pain! For me it's super strange this being happening on my end and you not able to reproduce it. The thing that concerns me and that "enhances the feeling" that I'm doing something wrong is that it seems that no one else reported the issue other than @Jalmeida1994 that works with me.
As @Jalmeida1994 said, could this be a "build" configuration that could be affecting our builds, I mostly dismiss that angle since we are having the issue even the the official docker images.
Meanwhile I did some testing but was not able to get any conclusion since found what feels to me another issue in the sorting of version v2.5.1.
I did a regression test with v2.5.1
and the sorting of Subscribers
in the List view doesn't break but also doesn't seem to be sorting correctly.
Meaning that I don't get the error, but it doesn't seem to be counting all users that are subscribed to the list but most probably just looking to the confirmed ones.
git checkout tags/v2.5.1 -b v2.5.1
➜ listmonk-knadh (v2.5.1) ✔ make dist
CGO_ENABLED=0 go build -o listmonk -ldflags="-s -w -X 'main.buildString=v2.5.1 (#a6a2b69 2024-06-20T07:59:28+0000)' -X 'main.versionString=v2.5.1'" cmd/*.go
go: downloading github.com/gofrs/uuid v4.0.0+incompatible
go: downloading github.com/labstack/echo/v4 v4.10.2
docker compose up demo-db
listmonk-knadh (v2.5.1) ✔ ./listmonk --install --yes
2024/06/20 09:01:39 main.go:102: v2.5.1 (#a6a2b69 2024-06-20T07:59:28+0000)
2024/06/20 09:01:39 init.go:145: reading config: config.toml
2024/06/20 09:01:39 init.go:273: connecting to db: 0.0.0.0:9432/listmonk
** first time installation **
** IMPORTANT: This will wipe existing listmonk tables and types in the DB 'listmonk' **
2024/06/20 09:01:39 install.go:179: setup complete
2024/06/20 09:01:39 install.go:180: run the program and access the dashboard at localhost:9000
listmonk-knadh (v2.5.1) ✔ ./listmonk
2024/06/20 09:11:39 main.go:102: v2.5.1 (#a6a2b69 2024-06-20T07:59:28+0000)
2024/06/20 09:11:39 init.go:145: reading config: config.toml
2024/06/20 09:11:39 init.go:273: connecting to db: 0.0.0.0:9432/listmonk
2024/06/20 09:11:40 init.go:593: media upload provider: filesystem
2024/06/20 09:11:40 init.go:517: loaded email (SMTP) messenger: username@smtp.yoursite.com
⇨ http server started on 127.0.0.1:9000
Sorting from the lower to higher subscriber number (the "Opt-in list" with 4 subscribers stays in the middle, with 4 subscribers).
Sorting from the higher to lower subscriber number (the "Opt-in list" with 4 subscribers stays in the middle, with 4 subscribers).
Sorting from the lower to higher subscriber number (the "Opt-in list" with 4 subscribers stays in the middle, with 4 subscribers).
Sorting from the higher to lower subscriber number (the "Opt-in list" with 4 subscribers stays in the middle, with 4 subscribers).
Yep, the sorting bug was fixed in v3.0.0 - https://github.com/knadh/listmonk/commit/c59825f3a5f4c4ccaac25e2d57b19a4e3084d1df
I have the same issue "Error fetching Lists: pq: column "subscriber_count" does not exist" in v3.0.0 on Repocloud
@benstaf are you still experiencing the issue, can you give a bit more context about the issue:
@knadh any thoughts on how to proceed on this?
Sorry, missed this. I've a setup locally that installs the very first version of listmonk (pre-alpha) and runs every subsequent version with --upgrade until the latest one. If there are any issues in the schema or queries, this helps catch.
I've been unable to reproduce OP's issue in our prod setup or in this test setup. The subscribe_count
field comes from a materialized view which is created during install: https://github.com/knadh/listmonk/blob/821b43d74ffc892fc20c25ef32c0ea0adc02236f/schema.sql#L375
The error can only happen if that materialized view isn't created (which --install surely does), which is only possible if that's somehow been deleted from the DB. Puzzling.
The issue is very bizarre.
Looking at you shared, regarding the materialized view, I have it and everything seems fine.
From the issue that I get from the browser:
{"message":"Error fetching Lists: pq: column \"subscriber_count\" does not exist"}
Error from the Database (docker compose output):
listmonk_demo_db | 2024-07-22 23:42:03.652 UTC [37891] ERROR: column "subscriber_count" does not exist at character 737
listmonk_demo_app | 2024/07/22 23:42:03 lists.go:51: error fetching lists: pq: column "subscriber_count" does not exist
listmonk_demo_db | 2024-07-22 23:42:03.652 UTC [37891] STATEMENT: WITH ls AS ( SELECT COUNT(*) OVER () AS total, lists.* FROM lists WHERE CASE WHEN $1 > 0 THEN id = $1 WHEN $2 != '' THEN uuid = $2::UUID WHEN $3 != '' THEN to_tsvector(name) @@ to_tsquery ($3) ELSE TRUE END AND ($4 = '' OR type = $4::list_type) AND ($5 = '' OR optin = $5::list_optin) AND (CARDINALITY($6::VARCHAR(100)[]) = 0 OR $6 <@ tags) OFFSET $7 LIMIT (CASE WHEN $8 < 1 THEN NULL ELSE $8 END) ), statuses AS ( SELECT list_id, COALESCE(JSONB_OBJECT_AGG(status, subscriber_count) FILTER (WHERE status IS NOT NULL), '{}') AS subscriber_statuses FROM mat_list_subscriber_stats GROUP BY list_id ) SELECT ls.*, COALESCE(ss.subscriber_statuses, '{}') AS subscriber_statuses FROM ls LEFT JOIN statuses ss ON (ls.id = ss.list_id) ORDER BY subscriber_count desc;
I imagine that the query that is been run doesn't expose the subscriber_count
in order to have it searchable, and that you are serializing in a future step to provide the subscriber_count
to the view.
This is your original query, with some minor modifications so that I could run just for this case:
WITH ls AS (
SELECT COUNT(*) OVER () AS total, lists.* FROM lists WHERE TRUE
),
statuses AS (
SELECT
list_id,
COALESCE(JSONB_OBJECT_AGG(status, subscriber_count) FILTER (WHERE status IS NOT NULL), '{}') AS subscriber_statuses
FROM mat_list_subscriber_stats
GROUP BY list_id
),
status_sums AS (
SELECT
list_id,
SUM((value)::INTEGER) AS subscriber_count
FROM statuses, jsonb_each_text(statuses.subscriber_statuses)
GROUP BY list_id
)
SELECT ls.*, COALESCE(ss.subscriber_statuses, '{}') AS subscriber_statuses
FROM ls
LEFT JOIN statuses ss ON (ls.id = ss.list_id)
LEFT JOIN status_sums sts ON (ls.id = sts.list_id)
ORDER BY created_at DESC;
This is what I get:
total | id | uuid | name | type | optin | tags | description | created_at | updated_at | subscriber_statuses |
---|---|---|---|---|---|---|---|---|---|---|
2 | 2 | 4909585a-c472-49b8-afff-3f4197dd651c | Opt-in list | public | double | {test} | 2024-07-16 21:26:03.282474 +00:00 | 2024-07-16 21:26:03.282474 +00:00 | {"unconfirmed": 1} | |
2 | 1 | 632b0f4c-4db1-44cf-9153-3a5ff4d1e7a1 | Default list | private | single | {test} | 2024-07-16 21:26:03.280067 +00:00 | 2024-07-16 21:26:03.280067 +00:00 | {"unconfirmed": 1} |
But once I add sts.*
to the SELECT ls.*, COALESCE(ss.subscriber_statuses, '{}') AS subscriber_statuses
part of the query I'm now able to filter for the field, meaning that having this:
SELECT ls.*, sts.*, COALESCE(ss.subscriber_statuses, '{}') AS subscriber_statuses
Now returns this:
total | id | uuid | name | type | optin | tags | description | created_at | updated_at | list_id | subscriber_count | subscriber_statuses |
---|---|---|---|---|---|---|---|---|---|---|---|---|
2 | 2 | 4909585a-c472-49b8-afff-3f4197dd651c | Opt-in list | public | double | {test} | 2024-07-16 21:26:03.282474 +00:00 | 2024-07-16 21:26:03.282474 +00:00 | 2 | 1 | {"unconfirmed": 1} | |
2 | 1 | 632b0f4c-4db1-44cf-9153-3a5ff4d1e7a1 | Default list | private | single | {test} | 2024-07-16 21:26:03.280067 +00:00 | 2024-07-16 21:26:03.280067 +00:00 | 1 | 1 | {"unconfirmed": 1} |
I'm dropping this here just as a quick and dirty reference to see if it helps nudge us on what could be the issue, but can't still explain why this happens to us, and not to you.
More towards the end of the week I will try to do a more "scientific" approach, understanding how can I do step-by-step debuging of the code and then report here any findings.
Hi @knadh,
Unfortunately, the past couple of weeks have been quite busy, so I haven't been able to make any progress until today.
I’m still unable to run the service without encountering the same issue, which has left me just as puzzled as you are. If you’d like, we can schedule a call to align our efforts.
To restate my issue, I’m encountering the following error message when I try to "sort" the list by "Subscriber":
error fetching lists: pq: column "subscriber_count" does not exist
This seems to be happening because the query being executed doesn't expose the subscriber_count
attribute in the SELECT statement.
-- name: query-lists
WITH ls AS (
SELECT COUNT(*) OVER () AS total, lists.* FROM lists WHERE
CASE
WHEN $1 > 0 THEN id = $1
WHEN $2 != '' THEN uuid = $2::UUID
WHEN $3 != '' THEN to_tsvector(name) @@ to_tsquery ($3)
ELSE TRUE
END
AND ($4 = '' OR type = $4::list_type)
AND ($5 = '' OR optin = $5::list_optin)
AND (CARDINALITY($6::VARCHAR(100)[]) = 0 OR $6 <@ tags)
OFFSET $7 LIMIT (CASE WHEN $8 < 1 THEN NULL ELSE $8 END)
),
statuses AS (
SELECT
list_id,
COALESCE(JSONB_OBJECT_AGG(status, subscriber_count) FILTER (WHERE status IS NOT NULL), '{}') AS subscriber_statuses
FROM mat_list_subscriber_stats
GROUP BY list_id
)
SELECT ls.*, COALESCE(ss.subscriber_statuses, '{}') AS subscriber_statuses
FROM ls LEFT JOIN statuses ss ON (ls.id = ss.list_id) ORDER BY %order%;
Interestingly when I change the query and add SUM(subscriber_count) as subscriber_count
to the statuses
sub-query everything works just fine. With me able to sort the Lists.
-- name: query-lists
WITH ls AS (
SELECT COUNT(*) OVER () AS total, lists.* FROM lists WHERE
CASE
WHEN $1 > 0 THEN id = $1
WHEN $2 != '' THEN uuid = $2::UUID
WHEN $3 != '' THEN to_tsvector(name) @@ to_tsquery ($3)
ELSE TRUE
END
AND ($4 = '' OR type = $4::list_type)
AND ($5 = '' OR optin = $5::list_optin)
AND (CARDINALITY($6::VARCHAR(100)[]) = 0 OR $6 <@ tags)
OFFSET $7 LIMIT (CASE WHEN $8 < 1 THEN NULL ELSE $8 END)
),
statuses AS (
SELECT
list_id,
SUM(subscriber_count) as subscriber_count,
COALESCE(JSONB_OBJECT_AGG(status, subscriber_count) FILTER (WHERE status IS NOT NULL), '{}') AS subscriber_statuses
FROM mat_list_subscriber_stats
GROUP BY list_id
)
SELECT ls.*, COALESCE(ss.subscriber_statuses, '{}') AS subscriber_statuses
FROM ls LEFT JOIN statuses ss ON (ls.id = ss.list_id) ORDER BY %order%;
Looking at the original query, and knowing that you on your side are able to sort by Subscribers (subscriber_count
) where is the expose subscriber_count
coming from?
subscriber_count
is in the materialized view that's setup during installation. https://github.com/knadh/listmonk/blob/fb090f238366cd37f91825005c6256ea08d3d8bf/schema.sql#L376
Can you try describing the schema of the materialized view?
listmonk=# \d mat_list_subscriber_stats;
Materialized view "public.mat_list_subscriber_stats"
Column | Type | Collation | Nullable | Default
------------------+--------------------------+-----------+----------+---------
updated_at | timestamp with time zone | | |
list_id | integer | | |
status | subscription_status | | |
subscriber_count | bigint | | |
Indexes:
"mat_list_subscriber_stats_idx" UNIQUE, btree (list_id, status)
Version:
Description of the bug and steps to reproduce:
On the "Lists" page when clicking on sort "Subscribers" column I get the following error
error fetching lists: column "subscriber_count" does not exist
.Screenshots:
Notes:
I'm not a Go developer and I will try to get a better understanding of the query bellow and how this code connects with the model, once I have that understanding I will try to open a pull-request for the issue.
https://github.com/knadh/listmonk/blob/0940e81109444a372d0bd7fd7ff2531a0527e3fd/queries.sql#L417-L439