Closed l4rm4nd closed 6 months ago
Same here
You might need to restart your PostgreSQL server. I don't actually know why but mine needed two restarts after the upgrade step for vector.rs to start working again.
I had to upgrade immich-database
redis version to 0.2.0
and it worked
I had to upgrade
immich-database
redis version to0.2.0
and it worked
Has likely nothing to do with the bug. Furthermore, is likely not recommended to downgrade the redis image version.
See official docker compose example here
What postgresql14 container should I be using? I've been using tensorchord/pgvecto-rs:pg14-v0.1.11
.
Edit: had to use tensorchord/pgvecto-rs:pg14-v0.2.0
it works now!
invalid type: map, expected a sequences/file_atomic.rs
I don't think I've seen this error before. This sounds like an internal pgvecto.rs bug that you should make an issue for here.
Since you're able to get up to the SELECT pgvectors_upgrade()
line, try committing before that and restarting Postgres before running that command, then restart again.
immich-database | 2024-02-22 01:06:57.346 UTC [40] ERROR: pgvecto.rs: IPC connection is closed unexpected.
Yeah, I am aware but could not yet validate whether the bug lies at pgvecto.rs or immich.
This may be some interesting issues available already:
This may be a potential fix:
I used your @LucaDev method to solve a similar problem. Thank you very much. For a program that is not very familiar, restarting sometimes solves a lot of problems. The following error occurred when I executed the database upgrade command:
NOTICE: schema "vectors" already exists, skipping
CREATE SCHEMA
ALTER DATABASE
SET
UPDATE 1
UPDATE 1
ALTER EXTENSION
UPDATE 1
ERROR: type "vecf16" already exists
ERROR: current transaction is aborted, commands ignored until end of transaction block
ROLLBACK
Before restarting, the immich-server ,immich-microservices log shows the following error,
[Nest] 7 - 02/23/2024, 8:19:28 PM WARN [DatabaseService] Could not run vector reindexing checks. If the extension was updated, please restart the Postgres instance.
/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219
throw new QueryFailedError_1.QueryFailedError(query, parameters, err);
^
QueryFailedError: pgvecto.rs: The extension is upgraded so all index files are outdated.
ADVICE: Delete all index files. Please read `https://docs.pgvecto.rs/admin/upgrading.html`.
at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async DataSource.query (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:350:20)
at async DatabaseRepository.shouldReindex (/usr/src/app/dist/infra/repositories/database.repository.js:127:25)
at async /usr/src/app/dist/domain/database/database.service.js:41:21
at async /usr/src/app/dist/infra/repositories/database.repository.js:185:23 {
query: '\n' +
' SELECT idx_status\n' +
' FROM pg_vector_index_stat\n' +
' WHERE indexname = $1',
parameters: [ 'clip_index' ],
driverError: error: pgvecto.rs: The extension is upgraded so all index files are outdated.
ADVICE: Delete all index files. Please read `https://docs.pgvecto.rs/admin/upgrading.html`.
at /usr/src/app/node_modules/pg/lib/client.js:526:17
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:184:25)
at async DataSource.query (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:350:20)
at async DatabaseRepository.shouldReindex (/usr/src/app/dist/infra/repositories/database.repository.js:127:25)
at async /usr/src/app/dist/domain/database/database.service.js:41:21
at async /usr/src/app/dist/infra/repositories/database.repository.js:185:23 {
length: 203,
severity: 'ERROR',
code: 'XX000',
detail: undefined,
hint: undefined,
position: undefined,
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: undefined,
table: undefined,
column: undefined,
dataType: undefined,
constraint: undefined,
file: 'error.rs',
line: '7',
routine: undefined
},
length: 203,
severity: 'ERROR',
code: 'XX000',
detail: undefined,
hint: undefined,
position: undefined,
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: undefined,
table: undefined,
column: undefined,
dataType: undefined,
constraint: undefined,
file: 'error.rs',
line: '7',
routine: undefined
}
Node.js v20.11.0
Please forgive me for posting the error message in the form of code, but there was an error in my GitHub picture upload.
What postgresql14 container should I be using? I've been using
tensorchord/pgvecto-rs:pg14-v0.1.11
.Edit: had to use
tensorchord/pgvecto-rs:pg14-v0.2.0
it works now!
Worked for me! Just make sure you recreate the container with the new version, then restart it, then run immich again and let it migrate
What postgresql14 container should I be using? I've been using
tensorchord/pgvecto-rs:pg14-v0.1.11
.Edit: had to use
tensorchord/pgvecto-rs:pg14-v0.2.0
it works now!Worked for me! Just make sure you recreate the container with the new version, then restart it, then run immich again and let it migrate
You are basically retyping the release upgrade instructions. We all already know that we have to use a new postgres version.
Such comments do not contribute to fix or troubleshoot the problem.
You are basically retyping the release upgrade instructions. We all already know that we have to use a new postgres version.
Such comments do not contribute to fix or troubleshoot the problem.
Having a bad day huh?
Worked for me! Just make sure you recreate the container with the new version, then restart it, then run immich again and let it migrate
Glad it worked fine for you. 👍
Sorry that above steps didn't work out for you. Even though I think disliking every post (reacting with a thumbs down) is quite rude - here are some more troubleshooting steps for you @l4rm4nd
Please activate the logging functionality (see https://docs.pgvecto.rs/admin/configuration.html) and see if there are additional error messages.
Please double check if you're running the correct images (feel free to share your docker-compose file) - are you running the same image the immich upgrade instructions used? (sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0)
Please post the full output and all commands you ran during the (database) upgrade process. That's the only way we're able to help you out. Which db user did you use to run the commands? Does the user have full DB administration permissions?
Sorry that above steps didn't work out for you. Even though I think disliking every post (reacting with a thumbs down) is quite rude - here are some more troubleshooting steps for you @l4rm4nd
Please activate the logging functionality (see https://docs.pgvecto.rs/admin/configuration.html) and see if there are additional error messages.
Please double check if you're running the correct images (feel free to share your docker-compose file) - are you running the same image the immich upgrade instructions used? (sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0)
Please post the full output and all commands you ran during the (database) upgrade process. That's the only way we're able to help you out. Which db user did you use to run the commands? Does the user have full DB administration permissions?
Hey @LucaDev,
it's all good. I don't want to be rude.
I've followed the instructions by Immich's release comments and upgraded the docker image tags accordingly. Also restarted the stack or postgresql container alone as referenced.
Therefore, comments by others stating that upgrading just worked is great for them and is likely the general outcome for most people. However, such comments do not contribute to fixing my issue.
The issue is not falsely following the upgrade instructions. More a bug on the side of pgvecto-rs or using a specific setup consisting of bind volume mounts or running immich over multiple version upgrades. I don't know yet what the problem is exactly.
I downvote the comments, as they do not bring us towards a potential fix or narrowing the issue down.
unraid specific
Just updating the docker image did not work for me
You may need to run
SELECT pgvectors_upgrade();
inside your v0.2.0 pgvector database for the upgrade to take hold.
immich docs recommended install docker compose page could use a rewrite to emphasize reading the release notes should problems arise from a happy path upgrade
For reference, SELECT pgvectors_upgrade();
just deletes the pg_vectors
folder in the Postgres data, so this is an alternative if you're unable to get that command to work.
For reference,
SELECT pgvectors_upgrade();
just deletes thepg_vectors
folder in the Postgres data, so this is an alternative if you're unable to get that command to work.
This was a brilliant hint!
I could fix the issue by manually removing the pg_vectors
directory from the Docker bind volume instead of running SELECT pgvectors_upgrade();
.
Following procedure worked:
# stop the stack
docker compose down
# replace v1.94.1 version tags with v1.95.1
# replace 0.1.11 version tag with 0.2.0
# pull new images
docker compose pull
# start the stack
docker compose up
# notice errors that immich could not upgrade the pgvector.rs
immich-microservices | The pgvecto.rs extension version is 0.1.11, but 0.2.0 is available.
immich-microservices | Immich attempted to update the extension, but failed to do so.
immich-microservices | This may be because Immich does not have the necessary permissions to update the extension.
# stop the stack
docker compose down
# only start the database container
docker compose up -d immich-database
# exec into database container
docker exec -it immich-database bash
# inside database container, connect to psql database
psql -U postgres -d immich-psgdb
# inside the psql db cli shell
CREATE SCHEMA IF NOT EXISTS vectors;
ALTER DATABASE "immich-psgdb" SET search_path TO "$user", public, vectors;
SET search_path TO "$user", public, vectors;
UPDATE pg_catalog.pg_extension SET extversion = '0.1.11' WHERE extname = 'vectors';
UPDATE pg_catalog.pg_extension SET extrelocatable = true WHERE extname = 'vectors';
ALTER EXTENSION vectors SET SCHEMA vectors;
UPDATE pg_catalog.pg_extension SET extrelocatable = false WHERE extname = 'vectors';
ALTER EXTENSION vectors UPDATE TO '0.2.0';
# exit from the psql database cli and database container
# stop the stack
docker compose down
# remove pg_vector directory from bind volume mounts manually
sudo rm -rf /mnt/docker-volumes/immich/database/pg_vectors
# restart the whole stack
docker compose up
# notice the mentioned warning from the immich release in logs
immich-microservices | [Nest] 6 - 02/26/2024, 8:06:45 AM WARN [DatabaseRepository] Could not reindex index face_index. Attempting to auto-fix.
After that, Immich did it things and the stack bootet properly up.
Now running Immich v1.95.1 successfully.
For reference,
SELECT pgvectors_upgrade();
just deletes thepg_vectors
folder in the Postgres data, so this is an alternative if you're unable to get that command to work.This was a brilliant hint!
I could fix the issue by manually removing the
pg_vectors
directory from the Docker bind volume instead of runningSELECT pgvectors_upgrade();
.Following procedure worked:
# stop the stack docker compose down # replace v1.94.1 version tags with v1.95.1 # replace 0.1.11 version tag with 0.2.0 # pull new images docker compose pull # start the stack docker compose up # notice errors that immich could not upgrade the pgvector.rs immich-microservices | The pgvecto.rs extension version is 0.1.11, but 0.2.0 is available. immich-microservices | Immich attempted to update the extension, but failed to do so. immich-microservices | This may be because Immich does not have the necessary permissions to update the extension. # stop the stack docker compose down # only start the database container docker compose up -d immich-database # exec into database container docker exec -it immich-database bash # inside database container, connect to psql database psql -U postgres -d immich-psgdb # inside the psql db cli shell CREATE SCHEMA IF NOT EXISTS vectors; ALTER DATABASE "immich-psgdb" SET search_path TO "$user", public, vectors; SET search_path TO "$user", public, vectors; UPDATE pg_catalog.pg_extension SET extversion = '0.1.11' WHERE extname = 'vectors'; UPDATE pg_catalog.pg_extension SET extrelocatable = true WHERE extname = 'vectors'; ALTER EXTENSION vectors SET SCHEMA vectors; UPDATE pg_catalog.pg_extension SET extrelocatable = false WHERE extname = 'vectors'; ALTER EXTENSION vectors UPDATE TO '0.2.0'; # exit from the psql database cli and database container # stop the stack docker compose down # remove pg_vector directory from bind volume mounts manually sudo rm -rf /mnt/docker-volumes/immich/database/pg_vectors # restart the whole stack docker compose up # notice the mentioned warning from the immich release in logs immich-microservices | [Nest] 6 - 02/26/2024, 8:06:45 AM WARN [DatabaseRepository] Could not reindex index face_index. Attempting to auto-fix.
After that, Immich did it things and the stack bootet properly up.
Now running Immich v1.95.1 successfully.
Following erro:
root@ec8b6a611c40:/# psql -U postgres -d immich
psql (14.10 (Debian 14.10-1.pgdg120+1))
Type "help" for help.
immich=# CREATE SCHEMA IF NOT EXISTS vectors;
NOTICE: schema "vectors" already exists, skipping
CREATE SCHEMA
immich=# ALTER DATABASE "immich-psgdb" SET search_path TO "$user", public, vectors;
ERROR: database "immich-psgdb" does not exist
immich=# ALTER DATABASE "immich" SET search_path TO "$user", public, vectors;
ALTER DATABASE
immich=# SET search_path TO "$user", public, vectors;
SET
immich=# UPDATE pg_catalog.pg_extension SET extversion = '0.1.11' WHERE extname = 'vectors';
UPDATE 1
immich=# UPDATE pg_catalog.pg_extension SET extrelocatable = true WHERE extname = 'vectors';
UPDATE 1
immich=# ALTER EXTENSION vectors SET SCHEMA vectors;
ALTER EXTENSION
immich=# UPDATE pg_catalog.pg_extension SET extrelocatable = false WHERE extname = 'vectors';
UPDATE 1
immich=# ALTER EXTENSION vectors UPDATE TO '0.2.0';
ERROR: type "vecf16" already exists
immich=#
immich=# ALTER EXTENSION vectors UPDATE TO '0.2.0'; ERROR: type "vecf16" already exists
I'm getting these errors on v1.105.1
. What do I do? I don't know if the discussion linked by @mertalev applies to me.
The bug
I've followed the release instructions to upgrade immich from v1.94.1 to 1.95.1.
What have I done:
This did not work though and the stack did not come up healthy.
I've then proceeded with the manual psql command instructions from the immich's release section. As I am using a different database name for my immich stack, I had to slightly adjust one command to hold my custom database name.
What I did:
The commands went mostly through. The last command
SELECT pgvectors_upgrade();
fails though:I've recovered from backup and running v1.94.1 again flawlessly.
Any hints how to fix the issue?
Might be a bug in pgvecto.rs due to null values (see https://github.com/tensorchord/pgvecto.rs/issues/376#issuecomment-1960701731)
The OS that Immich Server is running on
Ubuntu 22.04 LTS x86
Version of Immich Server
v1.95.1
Version of Immich Mobile App
v1.95.0
Platform with the issue
Your docker-compose.yml content
Your .env content
Reproduction steps