Getting stuck at 75% - server assets to delete

[kennym]

Version: immich-go 0.18.1, commit abd793e007f5acc58d8a9ceef3e8065d8023b8bb, built at 2024-06-30T19:14:17Z

As mentioned, my upload is getting stuck at 75% - this is the second time I try.

[kennym]

@simulot it does seem like this line never returns: https://github.com/simulot/immich-go/blob/abd793e007f5acc58d8a9ceef3e8065d8023b8bb/cmd/upload/upload.go#L854

[kennym]

I'm not seeing it implemented anywhere

[simulot]

Server's assets are deleted only when the uploaded asset is a better. (same name, same date of capture)

It's implemented, otherwise the program would not compile. It's here: https://github.com/simulot/immich-go/blob/abd793e007f5acc58d8a9ceef3e8065d8023b8bb/immich/asset.go#L190-L200

125K files... A big take! The log says: 306 assets to delete. Regarding the server activity: 41k pending jobs, 100% load... It may timeout

[simulot]

Have you noticed errors during the upload? This is not good... What the log file says about the errors?

[kennym]

The log says: 306 assets to delete. Regarding the server activity: 41k pending jobs, 100% load... It may timeout

Yeah, I think the operation times out - how could we fix that?

[simulot]

https://github.com/simulot/immich-go?tab=readme-ov-file#shared-options-with-all-commands -client-timeout=duration

[kennym]

So, I tried that, but the moment all the active jobs are processed the process terminates.

$ immich-go -server=http://umbrel.local:2283 -key=<REDACTED> upload -create-albums -google-photos -client-timeout=3h Takeout/
immich-go  0.18.1, commit abd793e007f5acc58d8a9ceef3e8065d8023b8bb, built at 2024-06-30T19:14:17Z
. _ _  _ _ . _|_     _  _
|| | || | ||(_| | ─ (_|(_)
            v 0.18.1 _)

Input analysis:
---------------
scanned image file                      :  114940
scanned video file                      :   10890
scanned sidecar file                    :  118704
discarded file                          :      10
unsupported file                        :       6
file duplicated in the input            :   22999
associated metadata file                :  125814
missing associated metadata file        :       0

Uploading:
----------
uploaded                                :    2877
upload error                            :     156
file not selected                       :     472
server's asset upgraded with the input  :     404
server has same asset                   :   90465
server has a better asset               :    1191

Check the log file:  /Users/kenny/Library/Caches/immich-go/immich-go_2024-07-01_15-01-45.log
context canceled

Log output:

 INFO | 403 server assets to delete. | time="2024-07-01T03:39.48 -04"
 INFO |
Input analysis:
---------------
scanned image file                      :  114940
scanned video file                      :   10890
scanned sidecar file                    :  118704
discarded file                          :      10
unsupported file                        :       6
file duplicated in the input            :   22999
associated metadata file                :  125814
missing associated metadata file        :       0

Uploading:
----------
uploaded                                :    2877
upload error                            :     156
file not selected                       :     472
server's asset upgraded with the input  :     404
server has same asset                   :   90465
server has a better asset               :    1191
 | time="2024-07-01T04:31.35 -04"
 INFO | context canceled | time="2024-07-01T04:31.35 -04"

This is the version of my Immich Server:

[kennym]

I'm re-running the script with a packet sniffer and some more logging. Will report back results

[kennym]

[kennym]

I added some more logging:

INFO | Starting to delete server assets with IDs: [256ef3fd-fc96-4ac6-8a42-585023877f9f af9bd568-6671-48e4-bcab-5efe5765c139 6bf05136-c343-434b-88f1-e8fc13f3344e 70742812-85ca-4ece-878e-8e6d4a3ee4b3 e5e0cd0d-3e4b-49f0-b4a1-b0316816f02d 4045d23a-1438-4c4c-b2de-f9e9faa397a8 077441b8-b1c5-4712-9495-e0c5154964e6 574ad108-30e0-47d1-a6f5-be13e58a1d74 9cdbfc23-392a-4857-a699-504371d82bbe 64f0cbf4-5a0b-4bb6-a794-2fc91df3c0e1 b9ef4fd8-b0cf-49c6-b487-f7c7bb857430 0c1b0fb8-4e31-4be2-90a6-5de86d82b812 1302c2bf-c6b5-4303-80d0-684fd1d7504e e2cc88b6-857f-4e71-95b8-f8842eea8f64 077d9391-c9fb-4f8c-a844-9af15c0270e1 c7ac2c9a-d85f-4d6d-ab35-7d80323ef553 1f0ca54b-b086-43c3-a466-9d6c8fb3c4e9 c2df253a-1c4a-42f4-afbd-2ab29812762d 939da0c9-7725-4f2e-bc7b-d0438a9698df 6d9ecb98-ae82-4cb6-ac78-a37d6aec84b1 109d8efa-9654-43fa-87a2-ff0ca0581e8a b98ab216-bef8-4d0d-9db2-8d2ddcf43ff0 1314e09d-4db2-4bf3-b9db-5882df3dc7ac 1bd6e7ec-28d4-4d37-97b5-aa4d1cd256ad 63bfb091-396f-4f28-83d8-7ffdf760274c c616f54a-bcda-4d51-b5a7-f5b8abddba6e a4dd181f-b878-456f-a85c-37d5d459312e e37a48cd-49f4-4e7b-9df0-04ac804eb3ed d7620ae3-d480-46b0-a3f4-e219736e8244 01cdedd7-cedc-43b2-93c8-d9c9802781aa 7bb7eca7-cf9d-4812-a8eb-66eb95be6d57 d0840fac-9db9-4935-b85e-7e9ebc367366 602a7bdc-28cf-4d41-b978-952567edd840 a7e2fc72-41e7-488d-a9b3-56058fc5da55 8e347033-bfd6-4668-87b7-04d1cb845af4 c736084d-b295-4515-a2f7-04f22ee80885 062aaeb0-4e1a-48f8-a5e2-c611ecdbc0cf ab5af2eb-df13-4d5f-b86c-ec8d743b3f82 38e773b7-3d1f-4872-b431-f9a08cdf089f 6dd51290-1a85-4712-be8d-fd85998b27dc 4b5676d2-d808-4831-8558-cbb3c9047b12 8c4dd381-4965-4375-b45c-a983fb8da581 22e64d92-938e-4cf1-b2cc-14630f85ab48 ae253cd2-d6fd-481a-a302-cc1003e58e12 4c575551-0a83-46bf-9779-da5bd198b26f a2a5accb-2f1d-4440-a974-66e2058f37b0 4fc6a293-89ca-47f0-b295-0ffdef242196 90d4c82c-c09d-47be-93e8-f86ea34826e9 432f1dfd-e4e2-4591-915e-c8e5dbaa40df c055de25-c76e-4fec-9220-4031ec281913 a4aa4aff-c1f8-4c9c-9a6d-b298fe7cd351 378485e8-a237-4633-ac9e-646943adeae9 dfb20ccf-e5c9-4650-ab2c-7d078be50c7e 35266e3c-9aa7-4097-a75a-12aa2e325168 0c6a7e1c-1a14-477d-995a-16953b054fc5 44d103a2-fcc3-447c-a1d9-6b041e9de722 dbc39ee3-b59f-4f37-8116-055bb3983686 ce1e6746-d426-4deb-a6fa-cb8fb161be8e 4873f163-aa4e-4329-8005-e5e35b1367da 1f0f1cb0-462d-4fa5-9888-dfac8900e6ca 4c67bb80-0506-4de7-bf1d-0b88c4b1e2ec a8921961-072b-4ca9-8f28-454832d181e5 488c316a-86e5-4f34-b1e7-3bce36bdaa53 45af3441-fc75-4556-978a-5e543f79ed62 ce53f25a-2de3-40de-9cf8-3732a70056dc 1cc8fb39-ec54-42eb-8c8b-fc9647fd2026 444e5be5-cb5d-4929-8b01-347e5d5a8162 30689729-91f2-4c52-afc8-3262f914ccd8 ab375e51-2e7a-45f4-a0df-52fcac03bbd1 43e3eb63-dd7a-4abd-9d59-81c96f20da32 a15f939e-d102-42af-a235-0f6fecbfddd6 7109096d-a212-48f9-9b5e-77a12787c796 6134818c-2ec4-4646-a4f6-4fdea4b33396 71f21c43-4b38-4732-ac3e-e5dfd53da337 39b19340-db52-4d38-af28-a53d1115be67 89910c05-ab7b-41d6-a88d-eaa69f8ddb79 30136aec-b4b0-4f44-bef7-036d75d573fa ca0c3324-340f-4817-a5d2-ee69afb6a4d7 8fce3082-9102-4780-9e1f-0606c5a35d9c bb5fdbec-374d-4886-a75e-3a7a0547a110 9ee93bb2-6169-469d-a53d-ebf30c761147 d513ce72-965f-4238-9e7d-9563235029b9 8fdd20bc-47c5-4ee6-9869-5eed6888abb6 3e14f681-b5f0-409b-9bfc-2c1fc8c101fc 88fae300-02da-4c1b-8af0-73b4bed8e577 6cb3330c-1ca9-48de-91b6-bda23e3cc703 e3d4fd93-d206-4cff-972b-d806eaaf5149 b724062a-10c2-4ff3-9c23-050281d277b2 e3dfd19b-cab4-4b95-bed6-58f3f1899a6d c6b8aede-0fd2-4e95-8c4b-c00fcb1f1bbb ba06e901-43ec-4c9d-946b-135c8d1cfad4 352a0ed5-3fec-4107-9f50-bcf6727a847d 352bf8fe-764f-44e2-a828-395f2b2ffbef e9863b80-508f-4281-9c4d-81742947fee3 07742e82-c786-4a75-80d6-02a9b5944f35 6d115ded-4eed-4bd0-95cd-e9812f22a279 0834d57d-63ed-4959-aa0e-bedfa38903e9 1c7f8d39-7898-4bac-b562-824e464e2d88 1c0a7740-2515-4621-a176-ae5257cfa5e3 9af0104a-d5d2-48cc-b670-81618942ab50 bf8a8785-b9df-48c6-8e32-b101ed07c4a0 9386fbc1-39fa-452d-94d2-f1747917e59c de0ad098-7111-43c1-a7d9-058026979444 e2d33942-dfb5-46f1-addf-01eec0d373c3 b483447c-1c13-4494-8227-65ae1c003e5c 44788f81-69de-4300-b20e-0f6a42ae5545 011a1ec6-8878-4a56-a39c-df2096c17245 6e75578a-090a-4da9-bbc3-d37f1f0d655a 0c2b1658-bf45-4f66-b988-02286ac81667 a12c8554-ae3a-4c74-b5a9-48a03bb6775e 8364608b-e930-4315-a572-0a6879f0f1e1 51066e36-7899-4a5c-8b6e-987a85b5f146 2c9056fa-fcbb-44aa-a44a-18b3d11786b7 b88a79ea-9427-4a92-8025-b47f055f24a8 9fc75b9a-ce4a-4dc5-a704-3e6a94dbdb65 2a283e32-424a-4474-9ee2-abee68242f2f 3e1d51dd-9ab5-4cd4-9715-b8f71cd8de2e 71469fee-d3fb-49ef-8b27-5b03d2ec7ee6 61e79fce-1b46-4cbe-8be9-6f52828af6fe 4266fd5d-920b-438d-b625-2c517dfc4ed4 cfbe10a9-7c48-4dc4-b49f-66cdceb5bef3 1554a8ea-a25a-4d4e-b283-69cba8edf17f 11915db6-50c0-4fa7-ae4c-fb486d566b27 c1fdd55b-a001-44f9-a5c3-2e0483697ca2 052f095a-eec8-41ad-a21b-5341495a5d91 21e381f8-afa3-40c9-9a3f-13300391fcc7 1b09d0ee-9eb0-405c-80b0-a3d0c4b2a936 7d8a1ab9-b205-4616-bb48-2c479cb5ebfc a776a49a-fe94-4cb5-9e5b-a7ca4363fdc4 090774e8-fc97-4c3f-aeb1-2f9302d5aad5 139abeb5-e639-4782-b442-edb963892f6b f71e223a-610a-49db-9271-b5c28236acfc 1a16e4d4-7655-44b5-8adc-6beca959e23d 81394439-1895-42cc-a2f2-badc9961eb04 ce2ad902-80e3-4391-afe4-6be8f5a10c62 f90f2453-198f-4ebc-8b3d-578158fe5437 5bc47ace-1998-4585-9225-c52bfc916e50 4bd6f16c-b798-449b-8610-3f172294125d 0f45b4cd-77cf-46de-a296-7391122fe88f c478f020-1189-4617-b7ac-5eed7967e510 7f88c53a-7243-4fae-8648-1044aced2181 b9bed2cf-b88c-4876-8df4-704e31819000 9e851579-1b5e-45e6-b4ed-9c1d69d37109 5730c521-298a-41ce-beaf-9bea9a3f2ea5 5092b981-e5e9-4a8d-a53e-1ca7ae81bd26 2d6d7e6b-43bd-4c1e-b6e7-8da1d4cdb12b 3e2e686e-3676-45fd-8918-e4a4adcb1835 b6b4f64b-a5c2-48ba-80c2-6fae473da0d7 b6b4f64b-a5c2-48ba-80c2-6fae473da0d7 8c9508d6-0fc4-4eda-ac4a-c0c8dd55bf2b 44683f84-fc59-4bc4-855b-1293af67954d 16be5aae-faf5-4c3f-86c2-775b9febdf4c 0943494e-3c04-43ef-a0e3-a1709cffcfeb 78f2dbfd-4a32-4202-a154-3a4ed79a9d3b 1540b6ff-e365-4b2a-a2ee-8c5a83b07cc8 2a316575-d508-4512-9127-1cb4c31527b3 e129c735-9a98-4b48-9efd-8046adde346b 63f249ec-f8bd-4560-8749-4a9a7b1bb4c4 46623ae4-c416-4efb-8ff0-9e27058a8d63 bb512d99-f872-43bf-a64a-eddbeed6aac9 a6682d2a-9242-4e36-a82b-ba5726f52062 639183b2-6f81-4e18-b492-a90468d07211 a8b15e99-0fc7-4418-b523-36215ea2e3d0 544a0b35-aa91-4c96-b36d-5c42df317f60 680ce6e6-6344-4f16-8a0e-841c50620774 fa8a6c31-b1e5-462b-8662-5162207a3d28 f0c7e800-4c3b-4977-8b03-7fb41969ecb9 a6017047-2347-48ce-9bf4-a4078edbfbec 218b401d-c4e3-4e32-9483-0db899e5340d 03262140-dc01-4cea-817b-f79776f9f2cc d111e9d2-ec30-4906-99e0-371ebef1a3f4 c9a60c08-8582-4cea-ade7-2d1fa0ba7579 2611e9fd-de7a-4cc6-aa00-7487d59d49a2 d89129bd-1e1c-43d5-a0db-3d39e15ac9f3 11d4fff7-d701-4f3c-b14b-86437a01d68e dedd508b-3e6b-44e9-abb6-ccf9b51b35c5 71deaaf4-e7e4-4178-9f93-fe8ffb66e488 54ddbf08-5e57-4cf1-895f-8b41b8171b1a 97affe4a-5017-4909-ba02-ce8a10dcf3b6 76c32f3e-bfbb-4954-85fb-4d4e9c8bd87f 9fb0ce8e-37c5-4f70-8d03-870ba082a01f a94aaa2c-bd3e-4142-a407-efc37808d718 b9f7e47a-dd7d-402a-8335-8d85313c1943 32cb9046-52ac-43c4-8914-2341ea20aa9a e8e25b41-4010-4150-bc0b-d95949855105 2ae0a991-a967-4699-8e09-cc04af812293 0c9d5b3e-7ab5-45da-98fe-7673d370555a 33eeddbd-8aa2-4a20-b610-519be4fb34dc 0e90a917-84ce-4f06-bc0d-d7bada86f5ef a2ebd9ee-c91f-48e2-b67b-f709ce1e920a 197deb81-c65a-4aad-b348-92cff9fc22c4 44512a46-7791-42bc-8051-9bf1bb454339 55846f71-b0c5-4d2d-9fb8-f5757beb52cd 71d91d74-b944-4ab6-9ca0-8ec3d2558144 61ae0a29-8207-4421-af47-8a7d73164ff2 043635d3-2a7b-4cf8-aabf-d79707231404 98283057-db92-4485-80b5-2dd647e91c30 dd6a5ab2-142d-4187-93d3-1f3e967ca383 7f89dce2-0ba0-4279-8bb6-b0dfa285081d c38c5276-33c4-4a3a-b341-7fd15316053e 196538f6-8d11-4aa8-bae7-dc36cd0c8732 f7af30df-1b60-48e6-83e6-1227652706e4 358a3edd-3261-4c50-9589-adf4f594c29c 15894dc6-cca0-422b-8ab0-ef02416d2620 e5236f75-c765-49b8-ac23-02ba57f45437 14298dd1-d7eb-4b77-9677-44d90a0829da cf37a848-eb85-401b-9832-236daeeec153 d4991285-01a2-4ccf-9c50-1721eed94956 1a25c4b7-aad7-4f8d-

It looks like the line got cut off! Maybe the payload is too big?

[kennym]

I'm trying deleting in batches of 50 UUIDs - I wonder if this failure is on the client or server side.

[kennym]

@simulot any ideas?

[simulot]

No. How beefy is your server?

[kennym]

No. How beefy is your server?

Umbrel Home - 2.9 GhZ Quad Core 16GB memory

[simulot]

Definitively not a PI4 Which OS?

[simulot]

I'm working on create a trace file

[simulot]

I'm re-running the script with a packet sniffer and some more logging. Will report back results

Oops, I haven't seen your trace on my mobile. I have implemented a build in trace. That will be used anyway. You can check out the branch simulot/issue343 and use the -api-trace=TRUE to enable traces.

On delete assets... Could you get the request body?

[kennym]

Request:

DELETE http://umbrel.local:2283/api/assets HTTP/1.1
Host: umbrel.local:2283
User-Agent: Go-http-client/1.1
Transfer-Encoding: chunked
Accept: application/json
Content-Type: application/json
X-Api-Key: <REDACTED>
Accept-Encoding: gzip

Payload:

{
    "force": false,
    "ids": [
        "21e381f8-afa3-40c9-9a3f-13300391fcc7",
        "7d8a1ab9-b205-4616-bb48-2c479cb5ebfc",
        "1554a8ea-a25a-4d4e-b283-69cba8edf17f",
        "11915db6-50c0-4fa7-ae4c-fb486d566b27",
        "cfbe10a9-7c48-4dc4-b49f-66cdceb5bef3",
        "1b09d0ee-9eb0-405c-80b0-a3d0c4b2a936",
        "a4aa4aff-c1f8-4c9c-9a6d-b298fe7cd351",
        "30689729-91f2-4c52-afc8-3262f914ccd8",
        "bb5fdbec-374d-4886-a75e-3a7a0547a110",
        "30136aec-b4b0-4f44-bef7-036d75d573fa",
        "c6b8aede-0fd2-4e95-8c4b-c00fcb1f1bbb",
        "38e773b7-3d1f-4872-b431-f9a08cdf089f",
        "d513ce72-965f-4238-9e7d-9563235029b9",
        "dfb20ccf-e5c9-4650-ab2c-7d078be50c7e",
        "8c4dd381-4965-4375-b45c-a983fb8da581",
        "e51a214a-4b5c-479e-9dc9-e57c360ef281",
        "ab5af2eb-df13-4d5f-b86c-ec8d743b3f82",
        "ce53f25a-2de3-40de-9cf8-3732a70056dc",
        "c736084d-b295-4515-a2f7-04f22ee80885",
        "8fce3082-9102-4780-9e1f-0606c5a35d9c",
        "90d4c82c-c09d-47be-93e8-f86ea34826e9",
        "8e347033-bfd6-4668-87b7-04d1cb845af4",
        "ae253cd2-d6fd-481a-a302-cc1003e58e12",
        "1bade132-cc7b-4674-85f6-3d53e0dd6699",
        "ce1e6746-d426-4deb-a6fa-cb8fb161be8e",
        "1637e4b5-8c3e-4db0-b7e8-02881aa32221",
        "cca5fefa-9750-484a-9b9e-90a36d815451",
        "ca0c3324-340f-4817-a5d2-ee69afb6a4d7",
        "45af3441-fc75-4556-978a-5e543f79ed62",
        "22e64d92-938e-4cf1-b2cc-14630f85ab48",
        "c6a50045-bd60-44d1-b7d6-ec543bb9cd43",
        "a15f939e-d102-42af-a235-0f6fecbfddd6",
        "1d39d7f5-ea97-4950-9571-bbd58ffa9af6",
        "0c6a7e1c-1a14-477d-995a-16953b054fc5",
        "444e5be5-cb5d-4929-8b01-347e5d5a8162",
        "9fe84b07-111c-4ab5-a287-559c43232750",
        "e3dfd19b-cab4-4b95-bed6-58f3f1899a6d",
        "4c575551-0a83-46bf-9779-da5bd198b26f",
        "6dd51290-1a85-4712-be8d-fd85998b27dc",
        "89910c05-ab7b-41d6-a88d-eaa69f8ddb79",
        "432f1dfd-e4e2-4591-915e-c8e5dbaa40df",
        "8fdd20bc-47c5-4ee6-9869-5eed6888abb6",
        "6134818c-2ec4-4646-a4f6-4fdea4b33396",
        "4b5676d2-d808-4831-8558-cbb3c9047b12",
        "4873f163-aa4e-4329-8005-e5e35b1367da",
        "6cb3330c-1ca9-48de-91b6-bda23e3cc703",
        "9ee93bb2-6169-469d-a53d-ebf30c761147",
        "ab375e51-2e7a-45f4-a0df-52fcac03bbd1",
        "88fae300-02da-4c1b-8af0-73b4bed8e577",
        "71f21c43-4b38-4732-ac3e-e5dfd53da337"
    ]
}

This seriously looks like something's wrong with Immich not the client.

[kennym]

I think with force=True it worked!

[simulot]

I'm wondering if one of the IDs isn't valid anymore when trying to delete as stated in the error message. This delete step happens at the end of th process and this isn't coherent with a progression of 75%

[kennym]

yes, I think you're right.

Later tonight, I'll try deleting in batches of 10, and continue on failure. It shouldn't get stuck only because of a 400