search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
48.15k stars 2.52k forks source link

[BUG] Javascript heap out of memory after extract metadata #3844

Closed eygraber closed 10 months ago

eygraber commented 1 year ago

The bug

I noticed that there were a few duplicate images in my library that were exactly the same except that one looked like it had reverse geocoding information added to it. I though that running the extract metadata job would fix this, so I ran it on All.

At first there were a bunch of lines in the logs like:

immich_microservices     | [Nest] 7  - 08/23/2023, 12:20:49 AM     LOG [MetadataExtractionProcessor] Already extracted asset upload/library/admin/2020/2020-10-07/PXL_20201007_161535958.mp4.

Then there were a bunch of errors that looked like:

immich_microservices     | [Nest] 7  - 08/23/2023, 12:23:02 AM   ERROR [MetadataExtractionProcessor] Failed to extract live photo upload/library/admin/2018/2018-08-12/MVIMG_20180812_191539.jpg: QueryFailedError: duplicate key value violates unique constraint "UQ_userid_checksum"
immich_postgres          | 2023-08-23 00:23:02.547 EDT [214] ERROR:  duplicate key value violates unique constraint "UQ_userid_checksum"
immich_postgres          | 2023-08-23 00:23:02.547 EDT [214] DETAIL:  Key ("ownerId", checksum)=(d15dd236-5058-43b6-99da-b0c02a2db9d9, \x0d994499ab2f0ddd6906983bd5381f6a2d5680cd) already exists.
immich_postgres          | 2023-08-23 00:23:02.547 EDT [214] STATEMENT:  INSERT INTO "assets"("id", "deviceAssetId", "ownerId", "deviceId", "type", "originalPath", "resizePath", "webpPath", "thumbhash", "encodedVideoPath", "createdAt", "updatedAt", "fileCreatedAt", "fileModifiedAt", "isFavorite", "isArchived", "isReadOnly", "checksum", "duration", "isVisible", "livePhotoVideoId", "originalFileName", "sidecarPath") VALUES (DEFAULT, $1, $2, $3, $4, $5, $6, $7, $8, $9, DEFAULT, DEFAULT, $10, $11, $12, $13, $14, $15, $16, $17, DEFAULT, $18, $19) RETURNING "id", "webpPath", "encodedVideoPath", "createdAt", "updatedAt", "isFavorite", "isArchived", "isReadOnly", "isVisible"

and

immich_microservices     | [Nest] 7  - 08/23/2023, 12:23:19 AM    WARN [MetadataExtractionProcessor] The exifData parsing failed due to Error: File not found - /usr/src/app/upload/upload/d15dd236-5058-43b6-99da-b0c02a2db9d9/31dbbd7e-2cdc-4bb8-9160-956006354034.JPG for asset 45496791-74e5-4978-857b-6463fd3974fe at upload/upload/d15dd236-5058-43b6-99da-b0c02a2db9d9/31dbbd7e-2cdc-4bb8-9160-956006354034.JPG
immich_microservices     | [Nest] 7  - 08/23/2023, 12:23:19 AM    WARN [MetadataExtractionProcessor] Error: File not found - /usr/src/app/upload/upload/d15dd236-5058-43b6-99da-b0c02a2db9d9/31dbbd7e-2cdc-4bb8-9160-956006354034.JPG
immich_microservices     |     at ReadTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29)
immich_microservices     |     at ReadTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40)
immich_microservices     | [Nest] 7  - 08/23/2023, 12:23:19 AM   ERROR [JobService] Unable to run job handler: Error: ENOENT: no such file or directory, stat 'upload/upload/d15dd236-5058-43b6-99da-b0c02a2db9d9/31dbbd7e-2cdc-4bb8-9160-956006354034.JPG'
immich_microservices     | [Nest] 7  - 08/23/2023, 12:23:19 AM   ERROR [JobService] Error: ENOENT: no such file or directory, stat 'upload/upload/d15dd236-5058-43b6-99da-b0c02a2db9d9/31dbbd7e-2cdc-4bb8-9160-956006354034.JPG'
immich_microservices     |     at Object.statSync (node:fs:1690:3)
immich_microservices     |     at MetadataExtractionProcessor.handlePhotoMetadataExtraction (/usr/src/app/dist/microservices/processors/metadata-extraction.processor.js:272:45)
immich_microservices     |     at async /usr/src/app/dist/domain/job/job.service.js:104:37
immich_microservices     |     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:342:28)
immich_microservices     |     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:525:24)
immich_microservices     | [Nest] 7  - 08/23/2023, 12:23:19 AM   ERROR [JobService] Object:
immich_microservices     | {
immich_microservices     |   "id": "45496791-74e5-4978-857b-6463fd3974fe"
immich_microservices     | }
immich_microservices     | 
immich_microservices     |

After running the job for a while, this gets printed:

microservices heap out of memory ``` immich_microservices | Request #1692760955977: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded" immich_microservices | Request #1692760955977: Sleeping for 4s and then retrying request... immich_typesense | I20230823 03:23:01.688388 730 log.cpp:523] close a full segment. Current first_index: 425757 last_index: 425765 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000425757_00000000000000425765 immich_typesense | I20230823 03:23:01.924839 730 log.cpp:537] Renamed `/data/state/log/log_inprogress_00000000000000425757' to `/data/state/log/log_00000000000000425757_00000000000000425765' immich_typesense | I20230823 03:23:01.924924 730 log.cpp:108] Created new segment `/data/state/log/log_inprogress_00000000000000425766' with fd=174 immich_typesense | I20230823 03:23:02.298883 645 raft_server.cpp:546] Term: 21, last_index index: 425767, committed_index: 425766, known_applied_index: 425766, applying_index: 0, queued_writes: 0, pending_queue_size: 205, local_sequence: 33327184 immich_typesense | I20230823 03:23:02.298957 736 raft_server.h:60] Peer refresh succeeded! immich_microservices | immich_microservices | immich_microservices | <--- Last few GCs ---> immich_microservices | immich_microservices | immich_microservices | [7:0x595e0840000] 126213 ms: Scavenge 4073.0 (4129.7) -> 4066.3 (4130.2) MB, 3.9 / 0.0 ms (average mu = 0.301, current mu = 0.292) allocation failure; immich_microservices | [7:0x595e0840000] 126222 ms: Scavenge 4073.5 (4130.2) -> 4067.0 (4130.7) MB, 3.1 / 0.0 ms (average mu = 0.301, current mu = 0.292) allocation failure; immich_microservices | [7:0x595e0840000] 126590 ms: Scavenge 4074.1 (4130.7) -> 4067.4 (4147.2) MB, 362.2 / 0.0 ms (average mu = 0.301, current mu = 0.292) allocation failure; immich_microservices | immich_microservices | immich_microservices | immich_microservices | immich_microservices | <--- JS stacktrace ---> immich_microservices | immich_microservices | immich_microservices | FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory immich_microservices | 1: 0xb83f50 node::Abort() [immich_microservices] immich_microservices | 2: 0xa94834 [immich_microservices] immich_microservices | 3: 0xd647c0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [immich_microservices] immich_microservices | 4: 0xd64b67 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [immich_microservices] immich_microservices | 5: 0xf42265 [immich_microservices] immich_microservices | 6: 0xf5474d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices] immich_microservices | 7: 0xf2ee4e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] immich_microservices | 8: 0xf30217 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] immich_microservices | 9: 0xf113ea v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices] immich_microservices | 10: 0x12d674f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices] immich_microservices | 11: 0x17035b9 [immich_microservices] ```
postgres giant SELECT ``` immich_postgres | 2023-08-22 23:23:02.565 EDT [1083] LOG: could not send data to client: Connection reset by peer immich_postgres | 2023-08-22 23:23:02.565 EDT [1083] STATEMENT: SELECT "AssetEntity"."id" AS "AssetEntity_id", "AssetEntity"."deviceAssetId" AS "AssetEntity_deviceAssetId", "AssetEntity"."ownerId" AS "AssetEntity_ownerId", "AssetEntity"."deviceId" AS "AssetEntity_deviceId", "AssetEntity"."type" AS "AssetEntity_type", "AssetEntity"."originalPath" AS "AssetEntity_originalPath", "AssetEntity"."resizePath" AS "AssetEntity_resizePath", "AssetEntity"."webpPath" AS "AssetEntity_webpPath", "AssetEntity"."thumbhash" AS "AssetEntity_thumbhash", "AssetEntity"."encodedVideoPath" AS "AssetEntity_encodedVideoPath", "AssetEntity"."createdAt" AS "AssetEntity_createdAt", "AssetEntity"."updatedAt" AS "AssetEntity_updatedAt", "AssetEntity"."fileCreatedAt" AS "AssetEntity_fileCreatedAt", "AssetEntity"."fileModifiedAt" AS "AssetEntity_fileModifiedAt", "AssetEntity"."isFavorite" AS "AssetEntity_isFavorite", "AssetEntity"."isArchived" AS "AssetEntity_isArchived", "AssetEntity"."isReadOnly" AS "AssetEntity_isReadOnly", "AssetEntity"."checksum" AS "AssetEntity_checksum", "AssetEntity"."duration" AS "AssetEntity_duration", "AssetEntity"."isVisible" AS "AssetEntity_isVisible", "AssetEntity"."livePhotoVideoId" AS "AssetEntity_livePhotoVideoId", "AssetEntity"."originalFileName" AS "AssetEntity_originalFileName", "AssetEntity"."sidecarPath" AS "AssetEntity_sidecarPath", "AssetEntity__AssetEntity_exifInfo"."assetId" AS "AssetEntity__AssetEntity_exifInfo_assetId", "AssetEntity__AssetEntity_exifInfo"."description" AS "AssetEntity__AssetEntity_exifInfo_description", "AssetEntity__AssetEntity_exifInfo"."exifImageWidth" AS "AssetEntity__AssetEntity_exifInfo_exifImageWidth", "AssetEntity__AssetEntity_exifInfo"."exifImageHeight" AS "AssetEntity__AssetEntity_exifInfo_exifImageHeight", "AssetEntity__AssetEntity_exifInfo"."fileSizeInByte" AS "AssetEntity__AssetEntity_exifInfo_fileSizeInByte", "AssetEntity__AssetEntity_exifInfo"."orientation" AS "AssetEntity__AssetEntity_exifInfo_orientation", "AssetEntity__AssetEntity_exifInfo"."dateTimeOriginal" AS "AssetEntity__AssetEntity_exifInfo_dateTimeOriginal", "AssetEntity__AssetEntity_exifInfo"."modifyDate" AS "AssetEntity__AssetEntity_exifInfo_modifyDate", "AssetEntity__AssetEntity_exifInfo"."timeZone" AS "AssetEntity__AssetEntity_exifInfo_timeZone", "AssetEntity__AssetEntity_exifInfo"."latitude" AS "AssetEntity__AssetEntity_exifInfo_latitude", "AssetEntity__AssetEntity_exifInfo"."longitude" AS "AssetEntity__AssetEntity_exifInfo_longitude", "AssetEntity__AssetEntity_exifInfo"."projectionType" AS "AssetEntity__AssetEntity_exifInfo_projectionType", "AssetEntity__AssetEntity_exifInfo"."city" AS "AssetEntity__AssetEntity_exifInfo_city", "AssetEntity__AssetEntity_exifInfo"."livePhotoCID" AS "AssetEntity__AssetEntity_exifInfo_livePhotoCID", "AssetEntity__AssetEntity_exifInfo"."state" AS "AssetEntity__AssetEntity_exifInfo_state", "AssetEntity__AssetEntity_exifInfo"."country" AS "AssetEntity__AssetEntity_exifInfo_country", "AssetEntity__AssetEntity_exifInfo"."make" AS "AssetEntity__AssetEntity_exifInfo_make", "AssetEntity__AssetEntity_exifInfo"."model" AS "AssetEntity__AssetEntity_exifInfo_model", "AssetEntity__AssetEntity_exifInfo"."lensModel" AS "AssetEntity__AssetEntity_exifInfo_lensModel", "AssetEntity__AssetEntity_exifInfo"."fNumber" AS "AssetEntity__AssetEntity_exifInfo_fNumber", "AssetEntity__AssetEntity_exifInfo"."focalLength" AS "AssetEntity__AssetEntity_exifInfo_focalLength", "AssetEntity__AssetEntity_exifInfo"."iso" AS "AssetEntity__AssetEntity_exifInfo_iso", "AssetEntity__AssetEntity_exifInfo"."exposureTime" AS "AssetEntity__AssetEntity_exifInfo_exposureTime", "AssetEntity__AssetEntity_exifInfo"."fps" AS "AssetEntity__AssetEntity_exifInfo_fps", "AssetEntity__AssetEntity_exifInfo"."exifTextSearchableColumn" AS "AssetEntity__AssetEntity_exifInfo_exifTextSearchableColumn", "AssetEntity__AssetEntity_smartInfo"."assetId" AS "AssetEntity__AssetEntity_smartInfo_assetId", "AssetEntity__AssetEntity_smartInfo"."tags" AS "AssetEntity__AssetEntity_smartInfo_tags", "AssetEntity__AssetEntity_smartInfo"."objects" AS "AssetEntity__AssetEntity_smartInfo_objects", "AssetEntity__AssetEntity_smartInfo"."clipEmbedding" AS "AssetEntity__AssetEntity_smartInfo_clipEmbedding", "AssetEntity__AssetEntity_tags"."id" AS "AssetEntity__AssetEntity_tags_id", "AssetEntity__AssetEntity_tags"."type" AS "AssetEntity__AssetEntity_tags_type", "AssetEntity__AssetEntity_tags"."name" AS "AssetEntity__AssetEntity_tags_name", "AssetEntity__AssetEntity_tags"."userId" AS "AssetEntity__AssetEntity_tags_userId", "AssetEntity__AssetEntity_tags"."renameTagId" AS "AssetEntity__AssetEntity_tags_renameTagId", "AssetEntity__AssetEntity_faces"."assetId" AS "AssetEntity__AssetEntity_faces_assetId", "AssetEntity__AssetEntity_faces"."personId" AS "AssetEntity__AssetEntity_faces_personId", "AssetEntity__AssetEntity_faces"."embedding" AS "AssetEntity__AssetEntity_faces_embedding", "AssetEntity__AssetEntity_faces"."imageWidth" AS "AssetEntity__AssetEntity_faces_imageWidth", "AssetEntity__AssetEntity_faces"."imageHeight" AS "AssetEntity__AssetEntity_faces_imageHeight", "AssetEntity__AssetEntity_faces"."boundingBoxX1" AS "AssetEntity__AssetEntity_faces_boundingBoxX1", "AssetEntity__AssetEntity_faces"."boundingBoxY1" AS "AssetEntity__AssetEntity_faces_boundingBoxY1", "AssetEntity__AssetEntity_faces"."boundingBoxX2" AS "AssetEntity__AssetEntity_faces_boundingBoxX2", "AssetEntity__AssetEntity_faces"."boundingBoxY2" AS "AssetEntity__AssetEntity_faces_boundingBoxY2", "8258e303a73a72cf6abb13d73fb592dde0d68280"."id" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_id", "8258e303a73a72cf6abb13d73fb592dde0d68280"."createdAt" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_createdAt", "8258e303a73a72cf6abb13d73fb592dde0d68280"."updatedAt" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_updatedAt", "8258e303a73a72cf6abb13d73fb592dde0d68280"."ownerId" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_ownerId", "8258e303a73a72cf6abb13d73fb592dde0d68280"."name" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_name", "8258e303a73a72cf6abb13d73fb592dde0d68280"."birthDate" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_birthDate", "8258e303a73a72cf6abb13d73fb592dde0d68280"."thumbnailPath" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_thumbnailPath", "8258e303a73a72cf6abb13d73fb592dde0d68280"."isHidden" AS "8258e303a73a72cf6abb13d73fb592dde0d68280_isHidden" FROM "assets" "AssetEntity" LEFT JOIN "exif" "AssetEntity__AssetEntity_exifInfo" ON "AssetEntity__AssetEntity_exifInfo"."assetId"="AssetEntity"."id" LEFT JOIN "smart_info" "AssetEntity__AssetEntity_smartInfo" ON "AssetEntity__AssetEntity_smartInfo"."assetId"="AssetEntity"."id" LEFT JOIN "tag_asset" "AssetEntity_AssetEntity__AssetEntity_tags" ON "AssetEntity_AssetEntity__AssetEntity_tags"."assetsId"="AssetEntity"."id" LEFT JOIN "tags" "AssetEntity__AssetEntity_tags" ON "AssetEntity__AssetEntity_tags"."id"="AssetEntity_AssetEntity__AssetEntity_tags"."tagsId" LEFT JOIN "asset_faces" "AssetEntity__AssetEntity_faces" ON "AssetEntity__AssetEntity_faces"."assetId"="AssetEntity"."id" LEFT JOIN "person" "8258e303a73a72cf6abb13d73fb592dde0d68280" ON "8258e303a73a72cf6abb13d73fb592dde0d68280"."id"="AssetEntity__AssetEntity_faces"."personId" WHERE ("AssetEntity"."id" IN ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, $37, $38, $39, $40, $41, $42, $43, $44, $45, $46, $47, $48, $49, $50, $51, $52, $53, $54, $55, $56, $57, $58, $59, $60, $61, $62, $63, $64, $65, $66, $67, $68, $69, $70, $71, $72, $73, $74, $75, $76, $77, $78, $79, $80, $81, $82, $83, $84, $85, $86, $87, $88, $89, $90, $91, $92, $93, $94, $95, $96, $97, $98, $99, $100, $101, $102, $103, $104, $105, $106, $107, $108, $109, $110, $111, $112, $113, $114, $115, $116, $117, $118, $119, $120, $121, $122, $123, $124, $125, $126, $127, $128, $129, $130, $131, $132, $133, $134, $135, $136, $137, $138, $139, $140, $141, $142, $143, $144, $145, $146, $147, $148, $149, $150, $151, $152, $153, $154, $155, $156, $157, $158, $159, $160, $161, $162, $163, $164, $165, $166, $167, $168, $169, $170, $171, $172, $173, $174, $175, $176, $177, $178, $179, $180, $181, $182, $183, $184, $185, $186, $187, $188, $189, $190, $191, $192, $193, $194, $195, $196, $197, $198, $199, $200, $201, $202, $203, $204, $205, $206, $207, $208, $209, $210, $211, $212, $213, $214, $215, $216, $217, $218, $219, $220, $221, $222, $223, $224, $225, $226, $227, $228, $229, $230, $231, $232, $233, $234, $235, $236, $237, $238, $239, $240, $241, $242, $243, $244, $245, $246, $247, $248, $249, $250, $251, $252, $253, $254, $255, $256, $257, $258, $259, $260, $261, $262, $263, $264, $265, $266, $267, $268, $269, $270, $271, $272, $273, $274, $275, $276, $277, $278, $279, $280, $281, $282, $283, $284, $285, $286, $287, $288, $289, $290, $291, $292, $293, $294, $295, $296, $297, $298, $299, $300, $301, $302, $303, $304, $305, $306, $307, $308, $309, $310, $311, $312, $313, $314, $315, $316, $317, $318, $319, $320, $321, $322, $323, $324, $325, $326, $327, $328, $329, $330, $331, $332, $333, $334, $335, $336, $337, $338, $339, $340, $341, $342, $343, $344, $345, $346, $347, $348, $349, $350, $351, $352, $353, $354, $355, $356, $357, $358, $359, $360, $361, $362, $363, $364, $365, $366, $367, $368, $369, $370, $371, $372, $373, $374, $375, $376, $377, $378, $379, $380, $381, $382, $383, $384, $385, $386, $387, $388, $389, $390, $391, $392, $393, $394, $395, $396, $397, $398, $399, $400, $401, $402, $403, $404, $405, $406, $407, $408, $409, $410, $411, $412, $413, $414, $415, $416, $417, $418, $419, $420, $421, $422, $423, $424, $425, $426, $427, $428, $429, $430, $431, $432, $433, $434, $435, $436, $437, $438, $439, $440, $441, $442, $443, $444, $445, $446, $447, $448, $449, $450, $451, $452, $453, $454, $455, $456, $457, $458, $459, $460, $461, $462, $463, $464, $465, $466, $467, $468, $469, $470, $471, $472, $473, $474, $475, $476, $477, $478, $479, $480, $481, $482, $483, $484, $485, $486, $487, $488, $489, $490, $491, $492, $493, $494, $495, $496, $497, $498, $499, $500, $501, $502, $503, $504, $505, $506, $507, $508, $509, $510, $511, $512, $513, $514, $515, $516, $517, $518, $519, $520, $521, $522, $523, $524, $525, $526, $527, $528, $529, $530, $531, $532, $533, $534, $535, $536, $537, $538, $539, $540, $541, $542, $543, $544, $545, $546, $547, $548, $549, $550, $551, $552, $553, $554, $555, $556, $557, $558, $559, $560, $561, $562, $563, $564, $565, $566, $567, $568, $569, $570, $571, $572, $573, $574, $575, $576, $577, $578, $579, $580, $581, $582, $583, $584, $585, $586, $587, $588, $589, $590, $591, $592, $593, $594, $595, $596, $597, $598, $599, $600, $601, $602, $603, $604, $605, $606, $607, $608, $609, $610, $611, $612, $613, $614, $615, $616, $617, $618, $619, $620, $621, $622, $623, $624, $625, $626, $627, $628, $629, $630, $631, $632, $633, $634, $635, $636, $637, $638, $639, $640, $641, $642, $643, $644, $645, $646, $647, $648, $649, $650, $651, $652, $653, $654, $655, $656, $657, $658, $659, $660, $661, $662, $663, $664, $665, $666, $667, $668, $669, $670, $671, $672, $673, $674, $675, $676, $677, $678, $679, $680, $681, $682, $683, $684, $685, $686, $687, $688, $689, $690, $691, $692, $693, $694, $695, $696, $697, $698, $699, $700, $701, $702, $703, $704, $705, $706, $707, $708, $709, $710, $711, $712, $713, $714, $715, $716, $717, $718, $719, $720, $721, $722, $723, $724, $725, $726, $727, $728, $729, $730, $731, $732, $733, $734, $735, $736, $737, $738, $739, $740, $741, $742, $743, $744, $745, $746, $747, $748, $749, $750, $751, $752, $753, $754, $755, $756, $757, $758, $759, $760, $761, $762, $763, $764, $765, $766, $767, $768, $769, $770, $771, $772, $773, $774, $775, $776, $777, $778, $779, $780, $781, $782, $783, $784, $785, $786, $787, $788, $789, $790, $791, $792, $793, $794, $795, $796, $797, $798, $799, $800, $801, $802, $803, $804, $805, $806, $807, $808, $809, $810, $811, $812, $813, $814, $815, $816, $817, $818, $819, $820, $821, $822, $823, $824, $825, $826, $827, $828, $829, $830, $831, $832, $833, $834, $835, $836, $837, $838, $839, $840, $841, $842, $843, $844, $845, $846, $847, $848, $849, $850, $851, $852, $853, $854, $855, $856, $857, $858, $859, $860, $861, $862, $863, $864, $865, $866, $867, $868, $869, $870, $871, $872, $873, $874, $875, $876, $877, $878, $879, $880, $881, $882, $883, $884, $885, $886, $887, $888, $889, $890, $891, $892, $893, $894, $895, $896, $897, $898, $899, $900, $901, $902, $903, $904, $905, $906, $907, $908, $909, $910, $911, $912, $913, $914, $915, $916, $917, $918, $919, $920, $921, $922, $923, $924, $925, $926, $927, $928, $929, $930, $931, $932, $933, $934, $935, $936, $937, $938, $939, $940, $941, $942, $943, $944, $945, $946, $947, $948, $949, $950, $951, $952, $953, $954, $955, $956, $957, $958, $959, $960, $961, $962, $963, $964, $965, $966, $967, $968, $969, $970, $971, $972, $973, $974, $975, $976, $977, $978, $979, $980, $981, $982, $983, $984, $985, $986, $987, $988, $989, $990, $991, $992, $993, $994, $995, $996, $997, $998, $999, $1000, $1001, $1002, $1003, $1004, $1005, $1006, $1007, $1008, $1009, $1010, $1011, $1012, $1013, $1014, $1015, $1016, $1017, $1018, $1019, $1020, $1021, $1022, $1023, $1024, $1025, $1026, $1027, $1028, $1029, $1030, $1031, $1032, $1033, $1034, $1035, $1036, $1037, $1038, $1039, $1040, $1041, $1042, $1043, $1044, $1045, $1046, $1047, $1048, $1049, $1050, $1051, $1052, $1053, $1054, $1055, $1056, $1057, $1058, $1059, $1060, $1061, $1062, $1063, $1064, $1065, $1066, $1067, $1068, $1069, $1070, $1071, $1072, $1073, $1074, $1075, $1076, $1077, $1078, $1079, $1080, $1081, $1082, $1083, $1084, $1085, $1086, $1087, $1088, $1089, $1090, $1091, $1092, $1093, $1094, $1095, $1096, $1097, $1098, $1099, $1100, $1101, $1102, $1103, $1104, $1105, $1106, $1107, $1108, $1109, $1110, $1111, $1112, $1113, $1114, $1115, $1116, $1117, $1118, $1119, $1120, $1121, $1122, $1123, $1124, $1125, $1126, $1127, $1128, $1129, $1130, $1131, $1132, $1133, $1134, $1135, $1136, $1137, $1138, $1139, $1140, $1141, $1142, $1143, $1144, $1145, $1146, $1147, $1148, $1149, $1150, $1151, $1152, $1153, $1154, $1155, $1156, $1157, $1158, $1159, $1160, $1161, $1162, $1163, $1164, $1165, $1166, $1167, $1168, $1169, $1170, $1171, $1172, $1173, $1174, $1175, $1176, $1177, $1178, $1179, $1180, $1181, $1182, $1183, $1184, $1185, $1186, $1187, $1188, $1189, $1190, $1191, $1192, $1193, $1194, $1195, $1196, $1197, $1198, $1199, $1200, $1201, $1202, $1203, $1204, $1205, $1206, $1207, $1208, $1209, $1210, $1211, $1212, $1213, $1214, $1215, $1216, $1217, $1218, $1219, $1220, $1221, $1222, $1223, $1224, $1225, $1226, $1227, $1228, $1229, $1230, $1231, $1232, $1233, $1234, $1235, $1236, $1237, $1238, $1239, $1240, $1241, $1242, $1243, $1244, $1245, $1246, $1247, $1248, $1249, $1250, $1251, $1252, $1253, $1254, $1255, $1256, $1257, $1258, $1259, $1260, $1261, $1262, $1263, $1264, $1265, $1266, $1267, $1268, $1269, $1270, $1271, $1272, $1273, $1274, $1275, $1276, $1277, $1278, $1279, $1280, $1281, $1282, $1283, $1284, $1285, $1286, $1287, $1288, $1289, $1290, $1291, $1292, $1293, $1294, $1295, $1296, $1297, $1298, $1299, $1300, $1301, $1302, $1303, $1304, $1305, $1306, $1307, $1308, $1309, $1310, $1311, $1312, $1313, $1314, $1315, $1316, $1317, $1318, $1319, $1320, $1321, $1322, $1323, $1324, $1325, $1326, $1327, $1328, $1329, $1330, $1331, $1332, $1333, $1334, $1335, $1336, $1337, $1338, $1339, $1340, $1341, $1342, $1343, $1344, $1345, $1346, $1347, $1348, $1349, $1350, $1351, $1352, $1353, $1354, $1355, $1356, $1357, $1358, $1359, $1360, $1361, $1362, $1363, $1364, $1365, $1366, $1367, $1368, $1369, $1370, $1371, $1372, $1373, $1374, $1375, $1376, $1377, $1378, $1379, $1380, $1381, $1382, $1383, $1384, $1385, $1386, $1387, $1388, $1389, $1390, $1391, $1392, $1393, $1394, $1395, $1396, $1397, $1398, $1399, $1400, $1401, $1402, $1403, $1404, $1405, $1406, $1407, $1408, $1409, $1410, $1411, $1412, $1413, $1414, $1415, $1416, $1417, $1418, $1419, $1420, $1421, $1422, $1423, $1424, $1425, $1426, $1427, $1428, $1429, $1430, $1431, $1432, $1433, $1434, $1435, $1436, $1437, $1438, $1439, $1440, $1441, $1442, $1443, $1444, $1445, $1446, $1447, $1448, $1449, $1450, $1451, $1452, $1453, $1454, $1455, $1456, $1457, $1458, $1459, $1460, $1461, $1462, $1463, $1464 immich_postgres | , $1465, $1466, $1467, $1468, $1469, $1470, $1471, $1472, $1473, $1474, $1475, $1476, $1477, $1478, $1479, $1480, $1481, $1482, $1483, $1484, $1485, $1486, $1487, $1488, $1489, $1490, $1491, $1492, $1493, $1494, $1495, $1496, $1497, $1498, $1499, $1500, $1501, $1502, $1503, $1504, $1505, $1506, $1507, $1508, $1509, $1510, $1511, $1512, $1513, $1514, $1515, $1516, $1517, $1518, $1519, $1520, $1521, $1522, $1523, $1524, $1525, $1526, $1527, $1528, $1529, $1530, $1531, $1532, $1533, $1534, $1535, $1536, $1537, $1538, $1539, $1540, $1541, $1542, $1543, $1544, $1545, $1546, $1547, $1548, $1549, $1550, $1551, $1552, $1553, $1554, $1555, $1556, $1557, $1558, $1559, $1560, $1561, $1562, $1563, $1564, $1565, $1566, $1567, $1568, $1569, $1570, $1571)) ```

The OS that Immich Server is running on

Ubuntu 22.04

Version of Immich Server

1.74.0

Version of Immich Mobile App

N/A

Platform with the issue

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    user: 1000:1001
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    user: 1000:1001
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: [ "start.sh", "microservices" ]
    volumes:
      - /disks/storage/immich/volumes/geocoding:/usr/src/app/.reverse-geocoding-dump
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-machine-learning:
    user: 1000:1001
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - /disks/storage/immich/volumes/model-cache:/cache
    env_file:
      - .env
    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - .env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
    volumes:
      - /disks/storage/immich/volumes/tsdata:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
    restart: always

  database:
    user: 1000:1001
    container_name: immich_postgres
    image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - /disks/storage/immich/volumes/pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
    depends_on:
      - immich-server
      - immich-web
    restart: always

Your .env content

PUID=1000
GUID=1001

# The location where your uploaded files are stored
UPLOAD_LOCATION=/disks/storage/immich/library

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secrets for postgres and typesense. You should change these to random passwords
TYPESENSE_API_KEY=<>
DB_PASSWORD=<>

TZ=America/New_York

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

### Reproduction steps

```bash
Run extract metadata

Additional information

No response

mertalev commented 1 year ago

Seems like a memory leak in the handler for Pixel motion photos. My hunch is that the buffers for extracting videos are using too much memory, possibly not being freed if there's an error, or not being freed quickly enough anyway.

jrasm91 commented 1 year ago

I'm thinking it is recursively creating assets 😅

francesquini commented 1 year ago

Not sure if the problem is exactly the same, but I'm getting a similar error followed by a core dump:

immich_microservices     | [Nest] 8  - 10/03/2023, 7:23:47 PM   ERROR [JobService] Unable to run job handler (search/search-index-assets): Error: Connection terminated due to connection timeout
immich_microservices     | [Nest] 8  - 10/03/2023, 7:23:47 PM   ERROR [JobService] Error: Connection terminated due to connection timeout
immich_microservices     |     at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
immich_microservices     |     at Object.onceWrapper (node:events:628:28)
immich_microservices     |     at Connection.emit (node:events:514:28)
immich_microservices     |     at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
immich_microservices     |     at Socket.emit (node:events:514:28)
immich_microservices     |     at TCP.<anonymous> (node:net:323:12)
immich_microservices     | [Nest] 8  - 10/03/2023, 7:23:47 PM   ERROR [JobService] Object:
immich_microservices     | {}
immich_microservices     | 
immich_microservices     | 
immich_microservices     | <--- Last few GCs --->
immich_microservices     | 
immich_microservices     | [8:0x5f254840000]   329587 ms: Scavenge (reduce) 1941.5 (1983.0) -> 1940.7 (1983.0) MB, 15.1 / 0.0 ms  (average mu = 0.261, current mu = 0.209) allocation failure; 
immich_microservices     | [8:0x5f254840000]   329612 ms: Scavenge (reduce) 1941.6 (1983.0) -> 1940.7 (1983.0) MB, 19.8 / 0.0 ms  (average mu = 0.261, current mu = 0.209) allocation failure; 
immich_microservices     | [8:0x5f254840000]   329623 ms: Scavenge (reduce) 1941.6 (1983.0) -> 1940.8 (1983.2) MB, 6.8 / 0.0 ms  (average mu = 0.261, current mu = 0.209) allocation failure; 
immich_microservices     | 
immich_microservices     | 
immich_microservices     | <--- JS stacktrace --->
immich_microservices     | 
immich_microservices     | FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
immich_microservices     |  1: 0xb83f50 node::Abort() [immich_microservices]
immich_microservices     |  2: 0xa94834  [immich_microservices]
immich_microservices     |  3: 0xd647c0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [immich_microservices]
immich_microservices     |  4: 0xd64b67 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [immich_microservices]
immich_microservices     |  5: 0xf42265  [immich_microservices]
immich_microservices     |  6: 0xf43168 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]
immich_microservices     |  7: 0xf53673  [immich_microservices]
immich_microservices     |  8: 0xf544e8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]
immich_microservices     |  9: 0xf2ee4e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
immich_microservices     | 10: 0xf30217 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
immich_microservices     | 11: 0xf113ea v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices]
immich_microservices     | 12: 0x12d674f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]
immich_microservices     | 13: 0x17035b9  [immich_microservices]
immich_postgres          | 2023-10-03 19:25:46.778 UTC [624] LOG:  could not send data to client: Connection reset by peer
immich_postgres          | 2023-10-03 19:25:46.778 UTC [624] STATEMENT:  SELECT "AssetFaceEntity"."assetId" AS "AssetFaceEntity_assetId", "AssetFaceEntity"."personId" AS "AssetFaceEntity_personId", "AssetFaceEntity"."embedding" AS "AssetFaceEntity_embedding", "AssetFaceEntity"."imageWidth" AS "AssetFaceEntity_imageWidth", "AssetFaceEntity"."imageHeight" AS "AssetFaceEntity_imageHeight", "AssetFaceEntity"."boundingBoxX1" AS "AssetFaceEntity_boundingBoxX1", "AssetFaceEntity"."boundingBoxY1" AS "AssetFaceEntity_boundingBoxY1", "AssetFaceEntity"."boundingBoxX2" AS "AssetFaceEntity_boundingBoxX2", "AssetFaceEntity"."boundingBoxY2" AS "AssetFaceEntity_boundingBoxY2", "AssetFaceEntity__AssetFaceEntity_asset"."id" AS "AssetFaceEntity__AssetFaceEntity_asset_id", "AssetFaceEntity__AssetFaceEntity_asset"."deviceAssetId" AS "AssetFaceEntity__AssetFaceEntity_asset_deviceAssetId", "AssetFaceEntity__AssetFaceEntity_asset"."ownerId" AS "AssetFaceEntity__AssetFaceEntity_asset_ownerId", "AssetFaceEntity__AssetFaceEntity_asset"."libraryId" AS "AssetFaceEntity__AssetFaceEntity_asset_libraryId", "AssetFaceEntity__AssetFaceEntity_asset"."deviceId" AS "AssetFaceEntity__AssetFaceEntity_asset_deviceId", "AssetFaceEntity__AssetFaceEntity_asset"."type" AS "AssetFaceEntity__AssetFaceEntity_asset_type", "AssetFaceEntity__AssetFaceEntity_asset"."originalPath" AS "AssetFaceEntity__AssetFaceEntity_asset_originalPath", "AssetFaceEntity__AssetFaceEntity_asset"."resizePath" AS "AssetFaceEntity__AssetFaceEntity_asset_resizePath", "AssetFaceEntity__AssetFaceEntity_asset"."webpPath" AS "AssetFaceEntity__AssetFaceEntity_asset_webpPath", "AssetFaceEntity__AssetFaceEntity_asset"."thumbhash" AS "AssetFaceEntity__AssetFaceEntity_asset_thumbhash", "AssetFaceEntity__AssetFaceEntity_asset"."encodedVideoPath" AS "AssetFaceEntity__AssetFaceEntity_asset_encodedVideoPath", "AssetFaceEntity__AssetFaceEntity_asset"."createdAt" AS "AssetFaceEntity__AssetFaceEntity_asset_createdAt", "AssetFaceEntity__AssetFaceEntity_asset"."updatedAt" AS "AssetFaceEntity__AssetFaceEntity_asset_updatedAt", "AssetFaceEntity__AssetFaceEntity_asset"."fileCreatedAt" AS "AssetFaceEntity__AssetFaceEntity_asset_fileCreatedAt", "AssetFaceEntity__AssetFaceEntity_asset"."fileModifiedAt" AS "AssetFaceEntity__AssetFaceEntity_asset_fileModifiedAt", "AssetFaceEntity__AssetFaceEntity_asset"."isFavorite" AS "AssetFaceEntity__AssetFaceEntity_asset_isFavorite", "AssetFaceEntity__AssetFaceEntity_asset"."isArchived" AS "AssetFaceEntity__AssetFaceEntity_asset_isArchived", "AssetFaceEntity__AssetFaceEntity_asset"."isExternal" AS "AssetFaceEntity__AssetFaceEntity_asset_isExternal", "AssetFaceEntity__AssetFaceEntity_asset"."isReadOnly" AS "AssetFaceEntity__AssetFaceEntity_asset_isReadOnly", "AssetFaceEntity__AssetFaceEntity_asset"."isOffline" AS "AssetFaceEntity__AssetFaceEntity_asset_isOffline", "AssetFaceEntity__AssetFaceEntity_asset"."checksum" AS "AssetFaceEntity__AssetFaceEntity_asset_checksum", "AssetFaceEntity__AssetFaceEntity_asset"."duration" AS "AssetFaceEntity__AssetFaceEntity_asset_duration", "AssetFaceEntity__AssetFaceEntity_asset"."isVisible" AS "AssetFaceEntity__AssetFaceEntity_asset_isVisible", "AssetFaceEntity__AssetFaceEntity_asset"."livePhotoVideoId" AS "AssetFaceEntity__AssetFaceEntity_asset_livePhotoVideoId", "AssetFaceEntity__AssetFaceEntity_asset"."originalFileName" AS "AssetFaceEntity__AssetFaceEntity_asset_originalFileName", "AssetFaceEntity__AssetFaceEntity_asset"."sidecarPath" AS "AssetFaceEntity__AssetFaceEntity_asset_sidecarPath" FROM "asset_faces" "AssetFaceEntity" LEFT JOIN "assets" "AssetFaceEntity__AssetFaceEntity_asset" ON "AssetFaceEntity__AssetFaceEntity_asset"."id"="AssetFaceEntity"."assetId"
immich_postgres          | 2023-10-03 19:25:46.779 UTC [624] FATAL:  connection to client lost
immich_postgres          | 2023-10-03 19:25:46.779 UTC [624] STATEMENT:  SELECT "AssetFaceEntity"."assetId" AS "AssetFaceEntity_assetId", "AssetFaceEntity"."personId" AS "AssetFaceEntity_personId", "AssetFaceEntity"."embedding" AS "AssetFaceEntity_embedding", "AssetFaceEntity"."imageWidth" AS "AssetFaceEntity_imageWidth", "AssetFaceEntity"."imageHeight" AS "AssetFaceEntity_imageHeight", "AssetFaceEntity"."boundingBoxX1" AS "AssetFaceEntity_boundingBoxX1", "AssetFaceEntity"."boundingBoxY1" AS "AssetFaceEntity_boundingBoxY1", "AssetFaceEntity"."boundingBoxX2" AS "AssetFaceEntity_boundingBoxX2", "AssetFaceEntity"."boundingBoxY2" AS "AssetFaceEntity_boundingBoxY2", "AssetFaceEntity__AssetFaceEntity_asset"."id" AS "AssetFaceEntity__AssetFaceEntity_asset_id", "AssetFaceEntity__AssetFaceEntity_asset"."deviceAssetId" AS "AssetFaceEntity__AssetFaceEntity_asset_deviceAssetId", "AssetFaceEntity__AssetFaceEntity_asset"."ownerId" AS "AssetFaceEntity__AssetFaceEntity_asset_ownerId", "AssetFaceEntity__AssetFaceEntity_asset"."libraryId" AS "AssetFaceEntity__AssetFaceEntity_asset_libraryId", "AssetFaceEntity__AssetFaceEntity_asset"."deviceId" AS "AssetFaceEntity__AssetFaceEntity_asset_deviceId", "AssetFaceEntity__AssetFaceEntity_asset"."type" AS "AssetFaceEntity__AssetFaceEntity_asset_type", "AssetFaceEntity__AssetFaceEntity_asset"."originalPath" AS "AssetFaceEntity__AssetFaceEntity_asset_originalPath", "AssetFaceEntity__AssetFaceEntity_asset"."resizePath" AS "AssetFaceEntity__AssetFaceEntity_asset_resizePath", "AssetFaceEntity__AssetFaceEntity_asset"."webpPath" AS "AssetFaceEntity__AssetFaceEntity_asset_webpPath", "AssetFaceEntity__AssetFaceEntity_asset"."thumbhash" AS "AssetFaceEntity__AssetFaceEntity_asset_thumbhash", "AssetFaceEntity__AssetFaceEntity_asset"."encodedVideoPath" AS "AssetFaceEntity__AssetFaceEntity_asset_encodedVideoPath", "AssetFaceEntity__AssetFaceEntity_asset"."createdAt" AS "AssetFaceEntity__AssetFaceEntity_asset_createdAt", "AssetFaceEntity__AssetFaceEntity_asset"."updatedAt" AS "AssetFaceEntity__AssetFaceEntity_asset_updatedAt", "AssetFaceEntity__AssetFaceEntity_asset"."fileCreatedAt" AS "AssetFaceEntity__AssetFaceEntity_asset_fileCreatedAt", "AssetFaceEntity__AssetFaceEntity_asset"."fileModifiedAt" AS "AssetFaceEntity__AssetFaceEntity_asset_fileModifiedAt", "AssetFaceEntity__AssetFaceEntity_asset"."isFavorite" AS "AssetFaceEntity__AssetFaceEntity_asset_isFavorite", "AssetFaceEntity__AssetFaceEntity_asset"."isArchived" AS "AssetFaceEntity__AssetFaceEntity_asset_isArchived", "AssetFaceEntity__AssetFaceEntity_asset"."isExternal" AS "AssetFaceEntity__AssetFaceEntity_asset_isExternal", "AssetFaceEntity__AssetFaceEntity_asset"."isReadOnly" AS "AssetFaceEntity__AssetFaceEntity_asset_isReadOnly", "AssetFaceEntity__AssetFaceEntity_asset"."isOffline" AS "AssetFaceEntity__AssetFaceEntity_asset_isOffline", "AssetFaceEntity__AssetFaceEntity_asset"."checksum" AS "AssetFaceEntity__AssetFaceEntity_asset_checksum", "AssetFaceEntity__AssetFaceEntity_asset"."duration" AS "AssetFaceEntity__AssetFaceEntity_asset_duration", "AssetFaceEntity__AssetFaceEntity_asset"."isVisible" AS "AssetFaceEntity__AssetFaceEntity_asset_isVisible", "AssetFaceEntity__AssetFaceEntity_asset"."livePhotoVideoId" AS "AssetFaceEntity__AssetFaceEntity_asset_livePhotoVideoId", "AssetFaceEntity__AssetFaceEntity_asset"."originalFileName" AS "AssetFaceEntity__AssetFaceEntity_asset_originalFileName", "AssetFaceEntity__AssetFaceEntity_asset"."sidecarPath" AS "AssetFaceEntity__AssetFaceEntity_asset_sidecarPath" FROM "asset_faces" "AssetFaceEntity" LEFT JOIN "assets" "AssetFaceEntity__AssetFaceEntity_asset" ON "AssetFaceEntity__AssetFaceEntity_asset"."id"="AssetFaceEntity"."assetId"
immich_microservices     | Aborted (core dumped)
immich_microservices exited with code 0

This is consistent and happens at every execution of the migration job.

yodatak commented 12 months ago

I got the same issues in k3s for the microservices parts for truechart

[Nest] 7  - 10/11/2023, 7:13:18 PM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed
2023-10-11T19:13:18.789042417+02:00 [Nest] 7  - 10/11/2023, 7:13:18 PM   ERROR [JobService] TypeError: fetch failed
2023-10-11T19:13:18.789057486+02:00     at Object.fetch (node:internal/deps/undici/undici:13220:11)
2023-10-11T19:13:18.789069085+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-10-11T19:13:18.789114401+02:00     at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
2023-10-11T19:13:18.789133310+02:00     at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:120:31)
2023-10-11T19:13:18.789144668+02:00     at async /usr/src/app/dist/domain/job/job.service.js:109:37
2023-10-11T19:13:18.789159909+02:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
2023-10-11T19:13:18.789170851+02:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
2023-10-11T19:13:18.789213775+02:00 [Nest] 7  - 10/11/2023, 7:13:18 PM   ERROR [JobService] Object:
2023-10-11T19:13:18.789226330+02:00 {
2023-10-11T19:13:18.789238126+02:00   "id": "400de142-13c7-4424-9fa6-cce0e10d7a73"
2023-10-11T19:13:18.789249081+02:00 }
2023-10-11T19:13:18.789259087+02:00 
[Nest] 7  - 10/11/2023, 7:13:19 PM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 5fe9c003-3506-44d4-9264-16e1266abdc9
[Nest] 7  - 10/11/2023, 7:13:19 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset db23aeb2-e35f-4c9e-8bc2-0af4b4476bc1
[Nest] 7  - 10/11/2023, 7:13:19 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset 8cfb274e-3c99-45f6-9678-68d84aa3944c
[Nest] 7  - 10/11/2023, 7:13:20 PM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset b84207e4-ab96-41a0-92c6-855e19fbf857
[Nest] 7  - 10/11/2023, 7:13:21 PM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset fba4826a-11e6-493b-ab27-6a83ad3d536e
[Nest] 7  - 10/11/2023, 7:13:21 PM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset a988e32a-84d9-4cfb-9610-79220147de94
[Nest] 7  - 10/11/2023, 7:13:21 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout of 10000ms exceeded
[Nest] 7  - 10/11/2023, 7:13:21 PM   ERROR [JobService] Error: timeout of 10000ms exceeded
    at createError (/usr/src/app/node_modules/typesense/node_modules/axios/lib/core/createError.js:16:15)
    at RedirectableRequest.handleRequestTimeout (/usr/src/app/node_modules/typesense/node_modules/axios/lib/adapters/http.js:369:16)
    at RedirectableRequest.emit (node:events:514:28)
    at Timeout.<anonymous> (/usr/src/app/node_modules/follow-redirects/index.js:169:12)
2023-10-11T19:13:21.786904883+02:00     at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 10/11/2023, 7:13:21 PM   ERROR [JobService] Object:
{
  "id": "83b72b63-b869-40f0-80f4-d1aea477b1d5"
}
2023-10-11T19:13:21.786923019+02:00 
[Nest] 7  - 10/11/2023, 7:13:24 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset bd25f755-3da6-4a76-ac31-ca58581dc380
[Nest] 7  - 10/11/2023, 7:13:27 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset 7ba9f06f-b342-4dfe-bbf1-c77217894f27
[Nest] 7  - 10/11/2023, 7:13:27 PM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed
[Nest] 7  - 10/11/2023, 7:13:27 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:13220:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
    at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:120:31)
    at async /usr/src/app/dist/domain/job/job.service.js:109:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
2023-10-11T19:13:27.613029534+02:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
[Nest] 7  - 10/11/2023, 7:13:27 PM   ERROR [JobService] Object:
{
2023-10-11T19:13:27.613045059+02:00   "id": "ea8e152f-42b6-43d2-9ad8-a595e6b61798"
2023-10-11T19:13:27.613048055+02:00 }
2023-10-11T19:13:27.613050837+02:00 
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:13220:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
    at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:120:31)
    at async /usr/src/app/dist/domain/job/job.service.js:109:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Object:
{
  "id": "a56c1370-5bd1-450b-8de1-e4da00e79672"
}

[Nest] 7  - 10/11/2023, 7:13:28 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset 2b794421-20c3-40b6-a98b-8d6a63e6fe32
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] TypeError: fetch failed
2023-10-11T19:13:28.431743273+02:00     at Object.fetch (node:internal/deps/undici/undici:13220:11)
2023-10-11T19:13:28.431746795+02:00     at processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-10-11T19:13:28.431749391+02:00     at runNextTicks (node:internal/process/task_queues:64:3)
2023-10-11T19:13:28.431752058+02:00     at process.processImmediate (node:internal/timers:449:9)
2023-10-11T19:13:28.431754629+02:00     at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
2023-10-11T19:13:28.431757205+02:00     at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:77:22)
2023-10-11T19:13:28.431759672+02:00     at async /usr/src/app/dist/domain/job/job.service.js:109:37
2023-10-11T19:13:28.431762635+02:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
2023-10-11T19:13:28.431765171+02:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
2023-10-11T19:13:28.432117622+02:00 [Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Object:
2023-10-11T19:13:28.432129611+02:00 {
2023-10-11T19:13:28.432133520+02:00   "id": "e58d5d60-9ea3-44b9-a3c9-fe7defa6d867"
2023-10-11T19:13:28.432136263+02:00 }
2023-10-11T19:13:28.432138727+02:00 
2023-10-11T19:13:28.432719161+02:00 [Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed
2023-10-11T19:13:28.432823783+02:00 [Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] TypeError: fetch failed
2023-10-11T19:13:28.432830041+02:00     at Object.fetch (node:internal/deps/undici/und
ici:13220:11)
2023-10-11T19:13:28.432833106+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-10-11T19:13:28.432835650+02:00     at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
2023-10-11T19:13:28.432838258+02:00     at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:205:23)
2023-10-11T19:13:28.432840714+02:00     at async /usr/src/app/dist/domain/job/job.service.js:109:37
2023-10-11T19:13:28.432843403+02:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
2023-10-11T19:13:28.432845819+02:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
2023-10-11T19:13:28.432936251+02:00 [Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Object:
2023-10-11T19:13:28.432941285+02:00 {
2023-10-11T19:13:28.432944105+02:00   "id": "7994bfd3-53ac-4f7e-9649-0829250f9d41"
2023-10-11T19:13:28.432946595+02:00 }
2023-10-11T19:13:28.432948911+02:00 
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] TypeError: fetch failed
2023-10-11T19:13:28.579157643+02:00     at Object.fetch (node:internal/deps/undici/undici:13220:11)
2023-10-11T19:13:28.579161423+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
    at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:205:23)
    at async /usr/src/app/dist/domain/job/job.service.js:109:37
2023-10-11T19:13:28.579171899+02:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Object:
2023-10-11T19:13:28.579295756+02:00 {
  "id": "5b396809-39dd-4c09-a369-177bebd81a27"
2023-10-11T19:13:28.579302705+02:00 }

[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] TypeError: fetch failed
2023-10-11T19:13:28.624674626+02:00     at Object.fetch (node:internal/deps/undici/undici:13220:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-10-11T19:13:28.624681721+02:00     at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
2023-10-11T19:13:28.624684526+02:00     at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:77:22)
2023-10-11T19:13:28.624696932+02:00     at async /usr/src/app/dist/domain/job/job.service.js:109:37
2023-10-11T19:13:28.624700461+02:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
2023-10-11T19:13:28.624703093+02:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
2023-10-11T19:13:28.624823264+02:00 [Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Object:
2023-10-11T19:13:28.624829223+02:00 {
2023-10-11T19:13:28.624832882+02:00   "id": "7120cb7b-6163-4dae-82bc-1ac29914b138"
2023-10-11T19:13:28.624835929+02:00 }
2023-10-11T19:13:28.624838648+02:00 
[Nest] 7  - 10/11/2023, 7:13:28 PM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset cf4ec11f-c6d6-49c5-9064-58cb3d1be707
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] TypeError: fetch failed
2023-10-11T19:13:28.783314243+02:00     at Object.fetch (node:internal/deps/undici/undici:13220:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
2023-10-11T19:13:28.783350662+02:00     at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:77:22)
2023-10-11T19:13:28.783363858+02:00     at async /usr/src/app/dist/domain/job/job.service.js:109:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
2023-10-11T19:13:28.783387750+02:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Object:
{
2023-10-11T19:13:28.783424842+02:00   "id": "4589c464-8307-455e-83f6-e19b76a3a424"
}

[Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed
2023-10-11T19:13:28.840767205+02:00 [Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] TypeError: fetch failed
2023-10-11T19:13:28.840794602+02:00     at Object.fetch (node:internal/deps/undici/undici:13220:11)
2023-10-11T19:13:28.840808677+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-10-11T19:13:28.840820622+02:00     at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
2023-10-11T19:13:28.840831492+02:00     at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:120:31)
2023-10-11T19:13:28.840842777+02:00     at async /usr/src/app/dist/domain/job/job.service.js:109:37
2023-10-11T19:13:28.840854979+02:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
2023-10-11T19:13:28.841024661+02:00 [Nest] 7  - 10/11/2023, 7:13:28 PM   ERROR [JobService] Object:
2023-10-11T19:13:28.841045353+02:00 {
2023-10-11T19:13:28.841059213+02:00   "id": "e81f8a58-0e8a-491d-a9cc-f28bb4eea2b6"
2023-10-11T19:13:28.841070781+02:00 }
2023-10-11T19:13:28.841110845+02:00 
[Nest] 7  - 10/11/2023, 7:13:29 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset 80a79908-c49b-487f-bead-a224b3453cd7
[Nest] 7  - 10/11/2023, 7:13:30 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset 5b471cd1-3933-4ca4-ab60-229807e3e860
[Nest] 7  - 10/11/2023, 7:13:31 PM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-thumbhash-thumbnail): Error: Connection terminated due to connection timeout
[Nest] 7  - 10/11/2023, 7:13:31 PM   ERROR [JobService] Error: Connection terminated due to connection timeout
2023-10-11T19:13:31.482596099+02:00     at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
    at Object.onceWrapper (node:events:628:28)
2023-10-11T19:13:31.482603215+02:00     at Connection.emit (node:events:514:28)
2023-10-11T19:13:31.482606685+02:00     at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
2023-10-11T19:13:31.482609283+02:00     at Socket.emit (node:events:514:28)
2023-10-11T19:13:31.482611884+02:00     at TCP.<anonymous> (node:net:337:12)
2023-10-11T19:13:31.482651487+02:00 [Nest] 7  - 10/11/2023, 7:13:31 PM   ERROR [JobService] Object:
{
2023-10-11T19:13:31.482660906+02:00   "id": "627e6394-f677-429b-b444-9399354d24b3"
2023-10-11T19:13:31.482663691+02:00 }

[Nest] 7  - 10/11/2023, 7:13:31 PM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 627e6394-f677-429b-b444-9399354d24b3
[Nest] 7  - 10/11/2023, 7:13:32 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed
[Nest] 7  - 10/11/2023, 7:13:32 PM   ERROR [JobService] TypeError: fetch failed
2023-10-11T19:13:32.198116303+02:00     at Object.fetch (node:internal/deps/undici/undici:13220:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
    at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:205:23)
    at async /usr/src/app/dist/domain/job/job.service.js:109:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
[Nest] 7  - 10/11/2023, 7:13:32 PM   ERROR [JobService] Object:
{
  "id": "3dd9c9a9-349e-4c1e-94c3-44d1b7076a29"
}

[Nest] 7  - 10/11/2023, 7:13:32 PM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset f6dccf33-3230-4c2b-9031-b966c884d613
[Nest] 7  - 10/11/2023, 7:13:32 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset 024b537c-c30c-4e27-9234-8138f0338861
[Nest] 7  - 10/11/2023, 7:13:32 PM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 288eb4c7-366d-4b0b-8ba0-a4f420a45882
[Nest] 7  - 10/11/2023, 7:13:33 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset 986f85df-01c5-4e48-88c3-e3e84e353c1b
[Nest] 7  - 10/11/2023, 7:13:37 PM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 72935da0-66e4-46ef-8825-8c77d18c37a5
[Nest] 7  - 10/11/2023, 7:13:39 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset 355746ab-8b70-49d1-af8a-90f9d7712b91
[Nest] 7  - 10/11/2023, 7:13:39 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset 6b59b01c-aaa1-4f53-adc5-4a271ae15f50
[Nest] 7  - 10/11/2023, 7:13:40 PM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed
2023-10-11T19:13:40.431657438+02:00 [Nest] 7  - 10/11/2023, 7:13:40 PM   ERROR [JobService] TypeError: fetch failed
2023-10-11T19:13:40.431662824+02:00     at Object.fetch (node:internal/deps/undici/undici:13220:11)
2023-10-11T19:13:40.431666226+02:00     at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
2023-10-11T19:13:40.431671922+02:00     at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:120:31)
2023-10-11T19:13:40.431674737+02:00     at async /usr/src/app/dist/domain/job/job.service.js:109:37
2023-10-11T19:13:40.431677687+02:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
2023-10-11T19:13:40.431680727+02:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
2023-10-11T19:13:40.431713510+02:00 [Nest] 7  - 10/11/2023, 7:13:40 PM   ERROR [JobService] Object:
2023-10-11T19:13:40.431719093+02:00 {
2023-10-11T19:13:40.431722445+02:00   "id": "641554c5-1502-4e1f-8559-591c844acce7"
2023-10-11T19:13:40.431725012+02:00 }
2023-10-11T19:13:40.431727377+02:00 
[Nest] 7  - 10/11/2023, 7:13:40 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset 984e81dc-4b5d-4d89-89d5-afc48563b3d5
[Nest] 7  - 10/11/2023, 7:13:42 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset db3f355f-b4f3-481d-98b7-5f919261b7dd
[Nest] 7  - 10/11/2023, 7:13:42 PM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 56141756-dca1-4f43-917b-c0f0f2c01327
[Nest] 7  - 10/11/2023, 7:13:44 PM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed
2023-10-11T19:13:44.972488668+02:00 [Nest] 7  - 10/11/2023, 7:13:44 PM   ERROR [JobService] TypeError: fetch failed
2023-10-11T19:13:44.972501987+02:00     at Object.fetch (node:internal/deps/undici/undici:13220:11)
2023-10-11T19:13:44.972506083+02:00     at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
2023-10-11T19:13:44.972508814+02:00     at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:77:22)
2023-10-11T19:13:44.972511586+02:00     at async /usr/src/app/dist/domain/job/job.service.js:109:37
2023-10-11T19:13:44.972514865+02:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
2023-10-11T19:13:44.972517493+02:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
2023-10-11T19:13:44.972615981+02:00 [Nest] 7  - 10/11/2023, 7:13:44 PM   ERROR [JobService] Object:
2023-10-11T19:13:44.972622001+02:00 {
2023-10-11T19:13:44.972625531+02:00   "id": "62f8ee52-90b3-4251-a1fa-81b84436fde6"
2023-10-11T19:13:44.972628553+02:00 }
2023-10-11T19:13:44.972631259+02:00 
[Nest] 7  - 10/11/2023, 7:13:47 PM     LOG [MediaService] Successfully generated WEBP video thumbnail for asset c27a8827-c65e-48c0-a7fc-4bece1c49ae1
/usr/src/app/dist/microservices/app.service.js:85
2023-10-11T19:13:48.529060637+02:00                 throw error;
2023-10-11T19:13:48.529065008+02:00                 ^
2023-10-11T19:13:48.529067863+02:00 
2023-10-11T19:13:48.529070587+02:00 Error: Error downloading GeoNames cities500 data: Error: aborted
2023-10-11T19:13:48.529092264+02:00     at PassThrough.<anonymous> (/usr/src/app/node_modules/local-reverse-geocoder/index.js:251:19)
2023-10-11T19:13:48.529098259+02:00     at PassThrough.emit (node:events:526:35)
2023-10-11T19:13:48.529100933+02:00     at emitErrorNT (node:internal/streams/destroy:151:8)
2023-10-11T19:13:48.529103442+02:00     at emitErrorCloseNT (node:internal/streams/destroy:116:3)
2023-10-11T19:13:48.529108595+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
2023-10-11T19:13:48.529110877+02:00 
2023-10-11T19:13:48.529113559+02:00 Node.js v20.8.0
yodatak commented 12 months ago

I cannot use immich anymore to make thumbernails or anythings that need microservices because of this bug , so immich its unusable

jrasm91 commented 12 months ago

Can you try adjusting some of the job concurrencies to be lower values? It seems something is taking up all the available resources (possible disk i/o).

yodatak commented 12 months ago

Whith only one in concurent job I got the same issue

alextran1502 commented 12 months ago

@yodatak how much resource are you allocating for the Immich instance?

yodatak commented 12 months ago

Some more error: with tccr.io/truecharts/immich-server:v1.81.1@sha256:ca84ec775d436a83fa5f15631fed75da2dcd1bb01ccea6a3071068649886cc7c 4000 mcpu limit and 8GO limit of ram


apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    meta.helm.sh/release-name: immich
    meta.helm.sh/release-namespace: photos
  labels:
    app: immich-8.1.24
    app.kubernetes.io/instance: immich
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: immich
    app.kubernetes.io/version: 1.81.1
    helm-revision: '7'
    helm.sh/chart: immich-8.1.24
    helm.toolkit.fluxcd.io/name: immich
    helm.toolkit.fluxcd.io/namespace: photos
    release: immich
  name: immich-microservices
  namespace: XXXX
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app.kubernetes.io/instance: immich
      app.kubernetes.io/name: immich
      pod.name: microservices
  strategy:
    type: Recreate
  template:
    metadata:
      annotations:
        rollme: EDIHZ
      creationTimestamp: null
      labels:
        app: immich-8.1.24
        app.kubernetes.io/instance: immich
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: immich
        app.kubernetes.io/version: 1.81.1
        helm-revision: '7'
        helm.sh/chart: immich-8.1.24
        pod.name: microservices
        release: immich
    spec:
      automountServiceAccountToken: false
      containers:
        - args:
            - start-microservices.sh
          env:
            - name: TZ
              value: Europe/Paris
            - name: UMASK
              value: '0022'
            - name: UMASK_SET
              value: '0022'
            - name: S6_READ_ONLY_ROOT
              value: '1'
          envFrom:
            - secretRef:
                name: immich-secret
            - secretRef:
                name: immich-deps-secret
            - configMapRef:
                name: immich-common-config
            - configMapRef:
                name: immich-micro-config
          image: >-
            tccr.io/truecharts/immich-server:v1.81.1@sha256:ca84ec775d436a83fa5f15631fed75da2dcd1bb01ccea6a3071068649886cc7c
          imagePullPolicy: IfNotPresent
          livenessProbe:
            failureThreshold: 5
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 10004
            timeoutSeconds: 5
          name: immich
          ports:
            - containerPort: 10004
              name: microservices
              protocol: TCP
          readinessProbe:
            failureThreshold: 5
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 2
            tcpSocket:
              port: 10004
            timeoutSeconds: 5
          resources:
            limits:
              cpu: '4'
              memory: 8Gi
            requests:
              cpu: 10m
              memory: 50Mi
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              add:
                - ALL
            privileged: true
            readOnlyRootFilesystem: false
            runAsGroup: 1000
            runAsNonRoot: true
            runAsUser: 1000
            seccompProfile:
              type: RuntimeDefault
          startupProbe:
            failureThreshold: 60
            initialDelaySeconds: 10
            periodSeconds: 5
            successThreshold: 1
            tcpSocket:
              port: 10004
            timeoutSeconds: 2
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /dev/shm
              name: devshm
            - mountPath: /media
              name: externallibrary
            - mountPath: /usr/src/app/upload/library
              name: library
            - mountPath: /microcache
              name: microcache
            - mountPath: /mnt/revolution/Photos Videos
              name: originals-photos
            - mountPath: /usr/src/app/upload/profile
              name: profile
            - mountPath: /shared
              name: shared
            - mountPath: /usr/src/app/upload/thumbs
              name: thumbs
            - mountPath: /tmp
              name: tmp
            - mountPath: /usr/src/app/upload/upload
              name: uploads
            - mountPath: /var/logs
              name: varlogs
            - mountPath: /var/run
              name: varrun
            - mountPath: /usr/src/app/upload/encoded-video
              name: video
      dnsConfig:
        options:
          - name: ndots
            value: '1'
      dnsPolicy: ClusterFirst
      enableServiceLinks: false
      initContainers:
        - command:
            - /bin/ash
            - '-c'
            - >
              echo "Pinging [http://immich-server:10001/server-info/ping] until
              it is ready..."

              until wget --spider --quiet
              "http://immich-server:10001/server-info/ping"; do
                echo "Waiting for [http://immich-server:10001/server-info/ping] to be ready..."
                sleep 2
              done

              echo "URL [http://immich-server:10001/server-info/ping] is ready!"
          env:
            - name: TZ
              value: Europe/Paris
            - name: UMASK
              value: '0022'
            - name: UMASK_SET
              value: '0022'
            - name: S6_READ_ONLY_ROOT
              value: '1'
          image: >-
            tccr.io/truecharts/alpine:v3.18.2@sha256:a661597195a991150580f4915018663dd18bc36f5fef1e88e00c7a3e7b93e127
          imagePullPolicy: IfNotPresent
          name: immich-init-wait-server
          resources:
            limits:
              cpu: '4'
              memory: 8Gi
            requests:
              cpu: 10m
              memory: 50Mi
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              add:
                - ALL
            privileged: true
            readOnlyRootFilesystem: false
            runAsGroup: 1000
            runAsNonRoot: true
            runAsUser: 1000
            seccompProfile:
              type: RuntimeDefault
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /dev/shm
              name: devshm
            - mountPath: /mnt/XXXX/Photos Videos
              name: originals-photos
            - mountPath: /shared
              name: shared
            - mountPath: /tmp
              name: tmp
            - mountPath: /var/logs
              name: varlogs
            - mountPath: /var/run
              name: varrun
      restartPolicy: Always
      runtimeClassName: nvidia
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 100
        fsGroupChangePolicy: OnRootMismatch
        supplementalGroups:
          - 568
          - 1000
          - 0
          - 568
      serviceAccount: default
      serviceAccountName: default
      shareProcessNamespace: false
      terminationGracePeriodSeconds: 60
      volumes:
        - emptyDir:
            medium: Memory
          name: devshm
        - name: externallibrary
          persistentVolumeClaim:
            claimName: immich-externallibrary
        - hostPath:
            path: /mnt/XXXXX/immich/library
            type: Directory
          name: library
        - emptyDir: {}
          name: microcache
        - hostPath:
            path: /mnt/XXXX/Photos
            type: Directory
          name: originals-photos
        - hostPath:
            path: /mnt/XXXX/immich/profile
            type: Directory
          name: profile
        - emptyDir: {}
          name: shared
        - hostPath:
            path: /opt/XXXX/thumbs
            type: Directory
          name: thumbs
        - emptyDir: {}
          name: tmp
        - hostPath:
            path: /mnt/XXXXX/immich/upload
            type: Directory
          name: uploads
        - emptyDir: {}
          name: varlogs
        - emptyDir:
            medium: Memory
          name: varrun
        - hostPath:
            path: /mnt/XXXX/immich/encoded-video
            type: Directory
          name: video

Container: immich Disconnected [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [NestFactory] Starting Nest application... 2023-10-15T16:58:55.056797643+02:00 [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +28ms [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] ConfigModule dependencies initialized +4ms [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +110ms [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] InfraModule dependencies initialized +5ms 2023-10-15T16:58:55.186795380+02:00 [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] DomainModule dependencies initialized +9ms [Nest] 8 - 10/15/2023, 4:58:55 PM LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms [Nest] 8 - 10/15/2023, 4:59:05 PM ERROR [JobService] Unable to run job handler (thumbnailGeneration/undefined): TypeError: handler is not a function [Nest] 8 - 10/15/2023, 4:59:05 PM ERROR [JobService] TypeError: handler is not a function at /usr/src/app/dist/domain/job/job.service.js:109:43 at Worker.workerHandler [as processFn] (/usr/src/app/dist/infra/repositories/job.repository.js:27:46) 2023-10-15T16:59:05.580804341+02:00 at Worker.callProcessJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:106:21) at Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:39) at /usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:165:70 at Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:30) at Worker.run (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:165:45) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) [Nest] 8 - 10/15/2023, 4:59:05 PM ERROR [JobService] Object: {} 2023-10-15T16:59:05.580845846+02:00

yodatak commented 12 months ago

I found for me that the issue is i have PREFER WIDE GAMUT enabled if i disabled it it work for me my issue is dependend of https://github.com/immich-app/immich/pull/4438

yodatak commented 11 months ago

i still have the issue :/

yodatak commented 11 months ago

[Nest] 8 - 10/26/2023, 8:36:54 PM ERROR [JobService] Unable to run job handler (search/search-index-assets): Error: Connection terminated due to connection timeout [Nest] 8 - 10/26/2023, 8:36:54 PM ERROR [JobService] Error: Connection terminated due to connection timeout 2023-10-26T20:36:54.847256916+02:00 at Connection. (/usr/src/app/node_modules/pg/lib/client.js:132:73) 2023-10-26T20:36:54.847259004+02:00 at Object.onceWrapper (node:events:628:28) 2023-10-26T20:36:54.847260689+02:00 at Connection.emit (node:events:514:28) at Socket. (/usr/src/app/node_modules/pg/lib/connection.js:63:12) at Socket.emit (node:events:514:28) 2023-10-26T20:36:54.847266141+02:00 at TCP. (node:net:337:12)

traktuner commented 11 months ago

I have the same problem. I clicked "refresh" for the library tasks (400k images) and not long after that:

<--- Last few GCs --->

[1022:0x30904830000]   377636 ms: Mark-sweep 4049.0 (4136.8) -> 4034.4 (4138.1) MB, 5284.4 / 0.0 ms  (average mu = 0.110, current mu = 0.010) allocation failure; scavenge might not succeed

[1022:0x30904830000]   383081 ms: Mark-sweep 4050.3 (4138.1) -> 4035.5 (4139.1) MB, 5382.1 / 0.0 ms  (average mu = 0.063, current mu = 0.011) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

 1: 0xb85bc0 node::Abort() [immich_microservices]

 2: 0xa94834  [immich_microservices]

 3: 0xd66d10 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [immich_microservices]

 4: 0xd670b7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [immich_microservices]

 5: 0xf447c5  [immich_microservices]

 6: 0xf456c8 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]

 7: 0xf55bd3  [immich_microservices]

 8: 0xf56a48 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]

 9: 0xf313ae v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]

10: 0xf32777 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]

11: 0xf1394a v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices]

12: 0x12d8caf v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]

13: 0x1705b39  [immich_microservices]
zackpollard commented 10 months ago

This issue seems to have diverged from the original problem. I'm not sure all the reported problems are the same. Out of memory exceptions are quite often caused by the library we used for reverse geocoding. In the next release we have completely reworked how reverse geocoding will function. We are replacing the old library with our own implementation that uses postgres for doing the reverse geocoding. This should result in much lower memory usage overall, and also should eliminate the microservices container running out of memory. We believe this was happening due to some issue with the logic in the old geocoding implementation which was not written by us. If you still continue to experience these issues after the next release (1.89.0) please open a new issue describing the problem. Cheers!