search

simulot / immich-go

An alternative to the immich-CLI command that doesn't depend on nodejs installation. It tries its best for importing google photos takeout archives.
GNU Affero General Public License v3.0
1.5k stars 47 forks source link

Albums need multiple tries to get created #3

Closed Deathproof76 closed 1 year ago

Deathproof76 commented 1 year ago

Thank you for this migration tool!

I seem to have a problem: Everything uploads in full and the exif-data is mostly there (maybe .1% of photos don't have a date etc, but there may never was one to begin with), but after the upload when albums should get created, this happens:

 Uploading "Takeout/Google Fotos/Photos from 2023/Screenshot_20230603-144125.png"... already exists on the server
 Uploading "Takeout/Google Fotos/Photos from 2023/Screenshot_20230604-014959.png"... already exists on the server
 Uploading "Takeout/Google Fotos/Photos from 2023/original_64b53217-891e-4c00-b1bb-e17d512f0d10_S.png"... already exists on the server
33311 media scanned
Create the album Schweiz 2021
Update the album Tropen-Aquarium 2022
1 400 Bad Request
Done.

On a different try there was this error:

 Uploading "Takeout/Google Fotos/Photos from 2023/Screenshot_20230603-144125.png"... already exists on the server
 Uploading "Takeout/Google Fotos/Photos from 2023/Screenshot_20230604-014959.png"... already exists on the server
 Uploading "Takeout/Google Fotos/Photos from 2023/original_64b53217-891e-4c00-b1bb-e17d512f0d10_S.png"...Done, total 17101 uploaded
33357 media scanned
Update the album Disneyland Paris 2022
1 400 Bad Request
2 server assets to delete.
1 500 Internal Server Error
Can't delete server's assets: %!!(MISSING)v(PANIC=Error method: runtime error: invalid memory address or nil pointer dereference)

sadly I don't have the related logs of the immich docker container, because I already recreated it. Maybe next time

this is my cmd: ./immich-go -server=https://immich.xxxx-xxx.lala -key=XXXXXXXXXXXXXXXXXXXXXXX -create-albums -google-photos /mnt/Dockerspace/gphotoback/takeout-20230604T075014Z-001.zip /mnt/Dockerspace/gphotoback/takeout-20230604T075014Z-002.zip /mnt/Dockerspace/gphotoback/takeout-20230604T075014Z-003.zip /mnt/Dockerspace/gphotoback/takeout-20230604T075014Z-004.zip

If I try to run it multiple times sometimes it does work for one more album, but same error after the creation of one or two.

I'm using this image: https://github.com/imagegenius/docker-immich could be related, maybe. The upload itself is more than 180GB, maybe also related

edit: another run, two additional albums created, still some missing. absolutely nothing in the immich container logs, same error:

 Uploading "Takeout/Google Fotos/Photos from 2023/Screenshot_20230603-144125.png"... already exists on the server
 Uploading "Takeout/Google Fotos/Photos from 2023/Screenshot_20230604-014959.png"... already exists on the server
 Uploading "Takeout/Google Fotos/Photos from 2023/original_64b53217-891e-4c00-b1bb-e17d512f0d10_S.png"... already exists on the server
33311 media scanned
Create the album Laeiszhalle 2023
Create the album Standesamt 
Update the album Timmendorf 2023
1 400 Bad Request
Done.
simulot commented 1 year ago

There is an error 400 on album creation. I'll check that asap

simulot commented 1 year ago

Thanks for your report. I have found errors during album updating. This is fixed in the release 0.1.3.

rhrpr commented 1 year ago

Firstly awesome work and thank you for what you've made!

Couple of things here (and happy to be told I was on the old release, went from using three zips to one and that's the culprit);

  1. Invalid Memory Address or Nil Pointer (Albums not created)

TLDR: I was able to work around these errors which occurred with the old and new release, by simply creating the albums in the immich app manually.

I got this can't create the album list from the server: %!!(MISSING)v(PANIC=Error method: runtime error: invalid memory address or nil pointer dereference) earlier today, at first I tried three zips at once and needed what I thought was more memory on the vm holding docker (I'd given it very little memory to run on beforehand). Since then I've upgraded immich app and immich-go by a version each to their respective latest.

https://github.com/simulot/immich-go/releases/tag/v0.1.3 https://github.com/immich-app/immich/releases/tag/v1.74.0

I then ran against a single zip only (as the issue was with the first zip).

Multiple retries added more new albums, but there were two albums that seemed to be stuck which was then resolved via attempt of the second zip. I was then able to have have the second and third of my three zips uploaded successfully (using upload command).

The command I was using; ./immich-go -server http://localhost:2283 -key xyz upload -create-albums -google-photos -keep-partner ~/path/to/takeout001.zip > output-1.log

There remained only the issue on retry of only the first zip where immich-go struggled to create two albums - repeated runs against the first zip will flip between these with the failed album create & null pointer reference

  1. With an ampersand (&) and and emoji (❤️) in the name (similar to #1 ).
  2. With just two letters, a word and 4 numbers at the end each split by spaces i.e. 'AB Cdef 2023'

I was able to work around these by simply creating the albums in the immich app manually.

  1. Large volume of 'File does not exist'

TLDR : Running all three of my takeout zips at once solved this issue Perhaps the docs could be more explicit that all need to be run it at once.

I had a fair number of 'file does not exist' as well with each zip run one at a time; 

$ cat output-1.log | grep not | wc -l
4449
$ cat output-2.log | grep not | wc -l
1609
$ cat output-3.log | grep not | wc -l
26

Running all three of my takeout zips at once solved this issue and created a final two albums. 

 cat output-all.log | grep not | wc -l 22

On the whole this is a super useful tool and it's impressive that it is as rapid as it is. Thanks again!

simulot commented 1 year ago

Firstly awesome work and thank you for what you've made!

Thank you. Your analysis is really helpful!

First, a precision on the tool. There is no need to run the tool inside a docker container on the server. This adds difficulties for the installation, and add execution constraints. This is possible because immich-go is doesn't need anything else than itself to run.

I haven't yet measured precisely the memory consumption or searched for memory leaks, but I have seen that used memory wasn't ballooning during the execution. I don't think that memory shortage is the cause of the panic.

It happens that the photo's json is in another zip part, giving "file not found" errors. To avoid them, all parts of the takeout archive must be handled at the same time as you have mentioned. Another solution, is to unzip all of them in a merged directory and process that directory.

I have used the tool from my laptop to upload my 2 zip archives totaling 70Gb and 20000 items. The immich server and my laptop are in the same network.

I'm glad you have found a workaround.

1/ Panic message I think about a miss use of a pointer when processing the albums. I never had the problem... This need to analyze the code to find a possible cause. Also, I have to change the message to get the line of code that triggers the panic.

2/ Names with emojis and or containing special chars Who is using "&" in file names? just kidding! I read photo's json and try to infer the photo's actual name. But the file name changed to avoid problems when unzipping archives on a disk. The used rules for this aren't clear nor stable.

I have some rules to determine the name of the JPG. When the rule is incorrect, the JGP isn't found, giving "file not found" errors

Could you give me following if possible for specials cases you have:

  1. The json file name and its content
  2. The name of the corresponding jpeg file (not the content) I hope to find how JPGs names are altered in such cases.

Thank for your patient analysis.

Deathproof76 commented 1 year ago

Thanks for your report. I have found errors during album updating. This is fixed in the release 0.1.3.

Thank you for answering and fixing. Almost all albums have been created by now, just two are missing still:

VID_20190718_165620.mp4: An asset with the same name:"VID_20190718_165620", date:"2019-07-18 14:56:20" and size:19.1 MB exists on the server. No need to upload.
VID_20190718_170512.mp4: An asset with the same name:"VID_20190718_170512", date:"2019-07-18 15:05:12" and size:42.8 MB exists on the server. No need to upload.
VID_20190718_173904.mp4: An asset with the same name:"VID_20190718_173904", date:"2019-07-18 15:39:04" and size:19.7 MB exists on the server. No need to upload.
0 asset(s) added to the album "Lübeck Weihnachtsmarkt 2022"
0 asset(s) added to the album "Mallorca 2021 "
0 asset(s) added to the album "Schweiz 2021"
Create the album Vivi + Alex Standesamt 
0 asset(s) added to the album "Bayern 2022"
0 asset(s) added to the album "Familia Mallorca 21"
0 asset(s) added to the album "Kopenhagen 2019"
Create the album Kolumbien 2022
can't create the album list from the server: CreateAlbum, POST,  https://immich.xxxxx.xxxx/api/album, 400 Bad Request
Bad Request
each value in assetIds must be a UUID

5 server assets to delete.
Done.

This was on immich 1.73, will try 1.74 next.

rhrpr commented 1 year ago

Thank you. Your analysis is really helpful!

No problem!

Who is using "&" in file names?

I guess I'm just an anarchist! Seems the takeout zip's album folder name just puts this _ in its place. Maybe anarchy isn't a great idea :)

Could you give me following if possible for specials cases you have:

  1. The json file name and its content
  2. The name of the corresponding jpeg file (not the content) I hope to find how JPGs names are altered in such cases.

Will get back to you on this later in the week based on the output I had, those logs are still there so I can refer to them easily. It'll take me a bit of time to try and build up some pattern rather than just give a random example but hopefully worth the wait.

Thank for your patient analysis.

You're welcome, least I could do!

simulot commented 1 year ago

I guess I'm just an anarchist!

:-) I have added the handling of "&". This should be sufficient to keep anarchist quiet.

In have reworked on the file name. I have done some tests with insane names full of emojis. So, I hope this problem is close.

The panic was ironically triggered in the error handling on the CreateAlbum. I have fixed the panic, revealing the actual error: 400 Bad Request.

Use the release 0.1.6. Tell me if the problem persists.

twau commented 1 year ago

I also get panics when trying to create albums with the 0.1.6 release

Create the album Hovdala slott
can't create the album list from the server: %!!(MISSING)v(PANIC=Error method: runtime error: invalid memory address or nil pointer dereference)

Also got this one time when trying to create album:

Create the album Ystad djurpark 20230717
can't create the album list from the server: CreateAlbum, POST, http://immich.localdomain:2283/api/album, 400 Bad Request
Bad Request
each value in assetIds must be a UUID

So the error messages differs each run it seems. Sometimes it also successfully creates albums. Anything else I can provide that is helpful?

simulot commented 1 year ago

Well, I should rework on Album handling!

leowinterde commented 1 year ago

Same here, with ~95GB of photos: ./immich-go -server SERVER -key KEY upload -google-photos -create-albums takeout-20230821T193117Z-001.zip takeout-20230821T193117Z-002.zip takeout-20230821T193117Z-003.zip takeout-20230821T193117Z-004.zip takeout-20230821T193117Z-005.zip takeout-20230821T193117Z-006.zip takeout-20230821T193117Z-007.zip takeout-20230821T193117Z-008.zip takeout-20230821T193117Z-009.zip takeout-20230821T193117Z-010.zip

already exists on the serverotos/Untitled(4)/IMG_20200101_003650.jpg"...
already exists on the serverotos/Untitled(4)/IMG_20200101_003657.jpg"...
Create the album Schule
can't create the album list from the server: %!!(MISSING)v(PANIC=Error method: runtime error: invalid memory address or nil pointer dereference)
27 server assets to delete.
Can't delete server's assets: %!!(MISSING)v(PANIC=Error method: runtime error: invalid memory address or nil pointer dereference)

but adding only one album (Schule) manually into Immich, the import works fine after a re-run:

already exists on the serverotos/Romfahrt - 2013/IMG_3092-COLLAGE.jpg"...
935 asset(s) added to the album "Familienbilder - 2010"
1704 asset(s) added to the album "Fam"
688 asset(s) added to the album "Familienbilder - 2008"
453 asset(s) added to the album "Familienbilder - 2013"
182 asset(s) added to the album "Familienbilder - 2016"
618 asset(s) added to the album "Grundschule"
Create the album Abschlussfest - 2014
563 asset(s) added to the album "Schule"
5050 asset(s) added to the album "Familienbilder"
128 asset(s) added to the album "Familienbilder - 2006"
308 asset(s) added to the album "Romfahrt - 2013"
662 asset(s) added to the album "Familienbilder - 2009"
24 asset(s) added to the album "Familienbilder - 2014"
414 asset(s) added to the album "2012"
516 asset(s) added to the album "Familienbilder - 2007"
243 asset(s) added to the album "Familienbilder - 2011"
337 asset(s) added to the album "Familienbilder - 2012"
198 asset(s) added to the album "Familienbilder - 2015"
Done.
simulot commented 1 year ago

To help the bug tracking, I have released a debug version downloadable #7

Thanks you for giving it a try and reports findings.

dolfje commented 1 year ago

I have the same problem, but didn't find the debug version in #7.

I started the application with -log-level info.

...
Create the album Amerika!!
can't create the album list from the server: %!!(MISSING)v(PANIC=Error method: runtime error: invalid memory address or nil pointer dereference)
Done.

So I guess I'm the anarchist for using an exclamation mark? I guess that is the problem.

simulot commented 1 year ago

I have the same problem, but didn't find the debug version in https://github.com/simulot/immich-go/discussions/7

The linux binary is here: https://github.com/simulot/immich-go/releases/tag/untagged-bd302c88a27212def45f

So I guess I'm the anarchist for using an exclamation mark? I guess that is the problem.

@rhrpr qualify himself as an anarchist... His profile mentions system integration and it's well known that file names with &,*, : ... give always problems... Funny! By the way album's title isn't a file. No risk here

dolfje commented 1 year ago

That link gives for me a 404

simulot commented 1 year ago

That link gives for me a 404

I'm not familiar with all github subtleties. This one looks to be accessible: https://github.com/simulot/immich-go/releases/tag/v0.1.7debug

dolfje commented 1 year ago

Thank you! I didn't see the debug version inside the releases yesterday. Here the log:

...
--- BODY ---
Create the album Gemiste kurken
--- API CALL ---
POST https://***/api/album
Accept : application/json
Content-Type : application/json
X-Api-Key : ***
--- JSON ---
{
 "albumName": "Gemiste kurken",
 "assetIds": [
  "2ae45205-dd55-4db8-9508-723de06eb362",
  "3572b11f-fc4e-44fd-886f-8817e9ec05ee",
  "aeb33647-d8b9-495d-ad52-ee97dbbcaf74",
  "db697fed-cdb4-438e-b478-7a9dd35b0241",
  "69ee7ee0-fd81-4c2a-a26b-b3f9c29aa58e",
  "45fc3aa3-cb61-42f5-8e21-348ba7902da7",
  "4b506873-207d-4764-8a92-e7e138148cd8",
  "3de29963-347c-4797-a40a-5b4da8cc3c99",
  "25a3f3a1-508a-48b5-9717-55f631f30ee6",
  "f1a99294-a3d5-4e0f-bf0c-add083da524b",
  "8b717623-3365-4125-beb3-4879f9a39346",
  "23d4affa-eb14-46a0-bb6c-dfb9fabe8da4",
  "99778ab0-cfc3-418f-9d73-78889dae8c84",
  "498482c3-5477-43ca-9e7b-b4df8ce803ec",
  "c345c142-e40b-49d7-bfc6-e0feb4a896d5",
  "a175a3c0-0e78-49ce-9090-6c34458715a3",
  "3eb4280f-6a1e-4271-9a65-d698cc9ee89e",
  "47fff1b4-10c4-4630-b5af-097adfd368a2",
  "ccc368d2-9f05-4bbd-ac38-f49c07a82a58",
  "841d48a9-eea0-461a-ae03-4544df2df3bb",
  "c2d47057-804a-4c5e-b5ac-8608e90716b3",
  "8f21206b-cf8f-4754-b219-11c64fcb976a",
  "c866846e-3b5e-4f43-8905-37de75c8094f",
  "c866846e-3b5e-4f43-8905-37de75c8094f",
  "778ffbb4-02a5-4b79-b227-8b73ed0bcf95",
  "cc02cc80-db04-4cb5-9b62-4404a3b56993",
  "b65708f1-aabf-4547-b495-66e73f83ae7b",
  "f2fb93c7-461c-489b-b25f-0d234db1562a",
  "2ac1b780-d5ec-4148-89ff-9679b1652248",
  "3e334f0f-babf-4c8a-85aa-30a0bcf67780",
  "3302e1de-bb74-4ac5-8906-c1181b9d90e1",
  "73949f9d-7fa0-4a70-a4cb-dff3f00caa97",
  "0de8ce92-546e-4618-9e92-535d463c5f4d",
  "6b131e3d-6a1b-43ec-abd6-dbc2b081594a",
  "bc129cf7-6a8e-48af-ada8-93b1a5ea6b92",
  "c2dbe4f3-9463-4143-a414-5266ec76ded9",
  "09d6accd-fc18-4e0d-ae0b-c963956c9c58",
  "559e0485-79aa-496f-8ae9-4ee977be38ba",
  "19cd061a-290c-42e1-9c70-a44271b9e6ef",
  "a7c75f62-35d7-44ef-9333-3f6c4180f5f0",
  "ec314a89-f142-4697-a3fb-b87dd1e64484",
  "e355572f-0f6b-4bb9-a9aa-8d060b4f666c",
  "e355572f-0f6b-4bb9-a9aa-8d060b4f666c",
  "67e2d461-b3c7-4441-a973-e34ac6ee0daa",
  "a017caed-102e-4f15-84ec-1dbf60a15f4f",
  "86656afc-6315-4bf8-b480-a4f8004ae718",
  "5d994157-64ac-46e0-86f4-859ccb5c7cf7",
  "bd9d61f6-8370-442d-8fa1-bb4e405b0460",
  "d02eed9f-8cf4-4e64-b3a3-40fe0428b96d",
  "f9c7711d-b847-48ca-9c90-16c115eac25e",
  "eaaec64c-6cc8-44e9-b05e-758d50cb4817",
  "8f56c75b-bf46-498a-a19c-20c837691357",
  "602415fb-7456-4b87-984f-5ab2d3444539",
  "52f90b75-e3d8-41c3-b965-650bff07ef8c",
  "ab84b8cc-019a-443e-ab27-59b68efaf208",
  "0b1f99a4-33ab-4fa6-8bf0-d45ec32b535c",
  "afa3c4b0-873f-4fce-94e0-af5f59cf2971",
  "5f906cc5-f993-4f07-b526-5d62a9a7c3f5",
  "b45ac4c0-bcb5-4a78-a836-a22726851d1a",
  "89bfbe27-4d0a-4f8b-a8b6-c5e81b57dbf3",
  "cf8fb94b-79e3-4725-be1d-014f4631866e",
  "8847f1f4-5b5b-4f35-8099-ec54d5b21cea",
  "110c7d99-8c34-40bd-9713-fa7ddc30e25a",
  "2b895675-eefb-47e9-b606-8f9a613ca462",
  "2875109c-1b1b-4c28-bd5b-014e3477ba08",
  "929da8ef-cfeb-46b5-9e56-d137fbb23746"
 ]
}

--- BODY ---
can't create the album list from the server: %!!(MISSING)v(PANIC=Error method: runtime error: invalid memory address or nil pointer dereference)
Done.
simulot commented 1 year ago

Thanks. Let me check it

dolfje commented 1 year ago

Maybe the problem lies repeating the same asset multiple time when creating the album?

Create the album Copy of Profile pictures
--- API CALL ---
POST https://***/api/album
Content-Type : application/json
X-Api-Key : ***
Accept : application/json
--- JSON ---
{
 "albumName": "Copy of Profile pictures",
 "assetIds": [
  "863da73d-baaf-4551-99bb-5b7b000aa366",
  "863da73d-baaf-4551-99bb-5b7b000aa366",
  "863da73d-baaf-4551-99bb-5b7b000aa366",
  "863da73d-baaf-4551-99bb-5b7b000aa366",
  "863da73d-baaf-4551-99bb-5b7b000aa366",
  "863da73d-baaf-4551-99bb-5b7b000aa366"
 ]
}

--- BODY ---
can't create the album list from the server: %!!(MISSING)v(PANIC=Error method: runtime error: invalid memory address or nil pointer dereference)
Done.
simulot commented 1 year ago

Ho... I don't know how you end up with duplicate ids. How did you do?

I have forged a takeout zip to have some duplicates. Album creations fails, be with a message. This need a fix anyway

--- BODY ---
can't create the album list from the server: CreateAlbum, POST, http://localhost:2283/api/album, 400 Bad Request
Bad Request
each value in assetIds must be a UUID

Done.
dolfje commented 1 year ago

I guess it was an import tool that imported all photos from facebook into my Google Photos. It created the following files, that are all different. But probably because of your deduplication (or the immich deduplication) are converted into the same UUID.

ls Takeout/Google Photos/Copy of Profile pictures -al:
drwxrwxr-x  2 nikos nikos    15 Aug 30 16:28  .
drwxrwxr-x 42 nikos nikos    45 Aug 30 16:30  ..
-rw-rw-r--  1 nikos nikos 21337 Aug 28 11:46 '2020-06-04(1).jpg'
-rw-rw-r--  1 nikos nikos 21290 Aug 28 11:46 '2020-06-04(2).jpg'
-rw-rw-r--  1 nikos nikos 11772 Aug 28 11:46 '2020-06-04(3).jpg'
-rw-rw-r--  1 nikos nikos 40179 Aug 28 11:46 '2020-06-04(4).jpg'
-rw-rw-r--  1 nikos nikos 20698 Aug 28 11:46 '2020-06-04(5).jpg'
-rw-rw-r--  1 nikos nikos  9445 Aug 28 11:46  2020-06-04.jpg
-rw-rw-r--  1 nikos nikos   847 Aug 28 11:46 '2020-06-04.jpg(1).json'
-rw-rw-r--  1 nikos nikos   794 Aug 28 11:46 '2020-06-04.jpg(2).json'
-rw-rw-r--  1 nikos nikos   844 Aug 28 11:46 '2020-06-04.jpg(3).json'
-rw-rw-r--  1 nikos nikos   794 Aug 28 11:46 '2020-06-04.jpg(4).json'
-rw-rw-r--  1 nikos nikos   844 Aug 28 11:46 '2020-06-04.jpg(5).json'
-rw-rw-r--  1 nikos nikos   846 Aug 28 11:46  2020-06-04.jpg.json
-rw-rw-r--  1 nikos nikos   337 Jun  4  2020  metadata.json
simulot commented 1 year ago

Ok, this defeats my pre-detection of duplicates based on file name and file size. Immich-cli send all copies, and the server reject duplicates.

I need to work on a better duplicate handling. I'd prefer to not upload something that will be discarded immediatly.

simulot commented 1 year ago

Yes, it was due to poor handling of duplicate images. I think I have fixed it.

Could you confirm how it works on your files?

https://github.com/simulot/immich-go/releases/tag/v0.1.7debug4

Thanks a lot

dolfje commented 1 year ago

I understand the trying reducing upload part. I'm actually happy to come across this bug. Else I would never have known that a previous tool made such strange filenames. And would probably lost some files of my library.

Maybe add a flag that disables the deduplication? As for my use case, the uploading is not a concern. I'm uploading from the same server. Thus there is no bandwidth used.

The thanks goes primarily to you! For creating this tool!

I have tested the debug4, but got the following error:

Uploading "Takeout/Google Photos/Photos from 2015/IMG_20150726_231620_nopm_.jpg"...--- API CALL ---
POST https://***/api/asset/upload
X-Api-Key : ***
Accept : application/json
Content-Type : multipart/form-data; boundary=782650cc2469e7caf464607f742782639f2e4095b72160143516230df0e8
--- API RESPONSE -- 
Server Caddy,nginx/1.25.0
Content-Length 62
Accept-Ranges bytes
Content-Type application/json; charset=utf-8
Date Wed, 30 Aug 2023 18:28:22 GMT
Alt-Svc h3=":2283"; ma=2592000
Etag "3e-2GvAdOOIpCxCm7nPR7vOuJWs9CA"
X-Powered-By Express
--- RESPONSE BODY ---
{"id":"d5a216da-d6a9-42f4-858c-6e6e2ed17d3a","duplicate":true}
--- BODY ---
already exists on the server
Uploading "Takeout/Google Photos/Photos from 2015/IMG_20150812_213143_nopm_.jpg"...--- API CALL ---
POST https://***/api/asset/upload
Accept : application/json
Content-Type : multipart/form-data; boundary=9d63fa0ba517f7546fa9f8b4e070924b7c423b36f1c2360d9b23db2f086d
X-Api-Key : ***
--- API RESPONSE -- 
Accept-Ranges bytes
Alt-Svc h3=":2283"; ma=2592000
X-Powered-By Express
Content-Type application/json; charset=utf-8
Date Wed, 30 Aug 2023 18:28:22 GMT
Etag "3e-Synu08O9XRcnjW+GfWxvoZYNzRo"
Server Caddy,nginx/1.25.0
Content-Length 62
--- RESPONSE BODY ---
{"id":"365fec50-c4c1-4117-b23d-6cc103170c5d","duplicate":true}
--- BODY ---
already exists on the server
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7e58a7]

goroutine 1 [running]:
immich-go/cmdupload.(*AssetIndex).adviceSameOnServer(0x835d40?, 0xc0088bee00)
    /home/jfcassan/dev/github.com/simulot/immich-go/cmdupload/upload.go:507 +0x27
immich-go/cmdupload.(*AssetIndex).ShouldUpload(0xc000a6cc80, 0xc00544a3c0)
    /home/jfcassan/dev/github.com/simulot/immich-go/cmdupload/upload.go:554 +0x21f
immich-go/cmdupload.(*UpCmd).handleAsset(0xc000132300, {0x95ceb8, 0xc00007e3c0}, 0xc00544a3c0)
    /home/jfcassan/dev/github.com/simulot/immich-go/cmdupload/upload.go:198 +0x3bd
immich-go/cmdupload.UploadCommand({0x95ceb8, 0xc00007e3c0}, 0x8b140c?, 0x13?, {0xc000022120?, 0x5?, 0xc00035aca5?})
    /home/jfcassan/dev/github.com/simulot/immich-go/cmdupload/upload.go:132 +0xa2d
main.Run({0x95ceb8, 0xc00007e3c0}, 0x1?)
    /home/jfcassan/dev/github.com/simulot/immich-go/main.go:122 +0xd3e
main.main()
    /home/jfcassan/dev/github.com/simulot/immich-go/main.go:44 +0x2a5
simulot commented 1 year ago

I would never have known that a previous tool made such strange filenames. And would probably lost some files of my library.

I'll do my best to improve my tool. With patience you won't lost any files.

The log shows that the error comes after handling the file "Takeout/Google Photos/Photos from 2015/IMG_20150812_213143nopm.jpg"

I was able to panic the program when the json name and the actual image name differs.

I have added a debug output to fine the json file. https://github.com/simulot/immich-go/releases/tag/v0.1.7debug6

./immich-go -server=http://localhost:2283 -key=XXXXX --no-colors-log -api-trace -log-level=debug upload -google-photos "/home/XXXX/duplicates/cat-001.zip" | tee log.log

Could you post the json content, and the name of the images that could match?
Thank you

dolfje commented 1 year ago

I have run your command and it generated a lot of data. When grepping, I found these:

1900694435-     {
1900694436:      "Name": "Takeout/Google Photos/Photos from 2015/IMG_20150812_213143_nopm_.jpg.json",
1900694437-      "Comment": "",
1900694438-      "NonUTF8": false,
1900694439-      "CreatorVersion": 45,
1900694440-      "ReaderVersion": 45,
1900694441-      "Flags": 2056,
1900694442-      "Method": 8,
1900694443-      "Modified": "2023-08-28T13:16:08Z",
1900694444-      "ModifiedTime": 27140,
1900694445-      "ModifiedDate": 22300,
1900694446-      "CRC32": 1529201351,
1900694447-      "CompressedSize": 404,
1900694448-      "UncompressedSize": 791,
1900694449-      "CompressedSize64": 404,
1900694450-      "UncompressedSize64": 791,
1900694451-      "Extra": "AQAIABO47HAMAAAAdXBOAAGWhEEOVGFrZW91dC9Hb29nbGUgUGhvdG9zL1Bob3RvcyBmcm9tIDIwMTUvSU1HXzIwMTUwODEyXzIxMzE0M19ub3BtXy5qcGcuanNvbg==",
1900694452-      "ExternalAttrs": 0
1900694453-     },
...
1901060323-     {
1901060324:      "Name": "Takeout/Google Photos/Photos from 2015/IMG_20150812_213143_nopm_.jpg",
1901060325-      "Comment": "",
1901060326-      "NonUTF8": false,
1901060327-      "CreatorVersion": 45,
1901060328-      "ReaderVersion": 45,
1901060329-      "Flags": 2056,
1901060330-      "Method": 8,
1901060331-      "Modified": "2023-08-28T12:41:40Z",
1901060332-      "ModifiedTime": 25908,
1901060333-      "ModifiedDate": 22300,
1901060334-      "CRC32": 572305839,
1901060335-      "CompressedSize": 4294967295,
1901060336-      "UncompressedSize": 4294967295,
1901060337-      "CompressedSize64": 648582,
1901060338-      "UncompressedSize64": 648482,
1901060339-      "Extra": "AQAcACLlCQAAAAAAhuUJAAAAAAAWF9coBwAAAAAAAAB1cEkAAUhTI09UYWtlb3V0L0dvb2dsZSBQaG90b3MvUGhvdG9zIGZyb20gMjAxNS9JTUdfMjAxNTA4MTJfMjEzMTQzX25vcG1fLmpwZw==",
1901060340-      "ExternalAttrs": 0
1901060341-     },

The command itself completed with: 1084 media scanned...

But maybe this is related with that my takeout zip is split into multiple file that I call with immich-go upload -google-photos -create-albums takeout-001.zip takeout-002.zip takeout-001.zip : Google Photos/Photos from 2015/IMG_20150812_213143nopm.jpg.json takeout-002.zip : Google Photos/Photos from 2015/IMG_20150812_213143nopm.jpg

simulot commented 1 year ago

Indeed, too much data! I haven't anticipate it will also dump the zips' directories, sorry for that. I need to The initial issue about albums is no solved, thanks to @dolfje report.

I propose to continue on #8 I'll make a version with a better debugging log.