immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
52.37k stars 2.77k forks source link

feat(server): faster metadata extraction #14277

Open mertalev opened 1 day ago

mertalev commented 1 day ago

Description

Besides being significantly faster, it also makes job progress more intuitive as the number of waiting jobs goes down as one would expect without other jobs being queued in their place.

Testing

metadata-extraction-rate

The y-axis is jobs/s. The first part is main and the second is this PR. The jump toward the end is when the queueing job finished queueing assets (the job is somewhat intensive with a large table). There is an increase on main as well if you look closely, but it's more subtle. I canceled both when their progress rate stabilized after the queueing job finished.

jrasm91 commented 1 day ago

It might be sufficient to save the CID value to the database prior to running the link logic part of the job though.