Lomoc arguments: no-move option missing

[darrepac]

Hi

Seems the doc is outdated. Contrary to what is written here , when I press lomoc -h, here is what I have:

root@ec3d0cddb52b:/opt/lomorage/bin# ./lomoc -h
NAME:
   lomoc - A new cli application

USAGE:
   lomoc [global options] command [command options] [arguments...]

VERSION:
   2022-05-01.07-14-31.0.0d586d5

COMMANDS:
   reset, r        reset password, or home directory
   discover, d     Discover lomo backend
   import, i       Import all photos from given directory into lomo backend with given username and password
   consistency, c  consistent check and fix btw db and local file system
   frame, f        lomo-frame related commands
   help, h         Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h     show help
   --version, -v  print the version

So I don't find the "no-move" option... and as I don't want to move the files that I import, I don't know what to do. thanks for your help

[fuji246]

The latest one should be "2022-06-02.00-39-36.0.c8d7547", you can run /opt/lomorage/bin/lomoc import [username] [password] [import_dir] -n, given that the import_dir is accessible on the host where lomod running.

lomoware@lomorage:~$ /opt/lomorage/bin/lomoc -h
NAME:
   lomoc - A new cli application

USAGE:
   lomoc [global options] command [command options] [arguments...]

VERSION:
   2022-06-02.00-39-36.0.c8d7547

COMMANDS:
   reset, r        reset password, or home directory
   discover, d     Discover lomo backend
   import, i       Import all photos from given directory into lomo backend with given username and password
   consistency, c  consistent check and fix btw db and local file system
   frame, f        lomo-frame related commands
   help, h         Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h     show help
   --version, -v  print the version
lomoware@lomorage:~$ /opt/lomorage/bin/lomoc import -h
NAME:
   lomoc import - Import all photos from given directory into lomo backend with given username and password

USAGE:
   lomoc import [command options] [username] [password] [directory]

OPTIONS:
   --port value, -p value  (default: 8000)
   --no-move, -n           not moving original photos/videos, and only insert record in db

[darrepac]

Thanks for the reply. I am using image: lomorage/amd64-lomorage:latest, I pulled the latest image but I have still a lomoc version of May 1st 2022... Is there a beta image that is more recent?

[fuji246]

Could you please try update again, just pushed new image

[darrepac]

Ok thanks I have the right version now

[darrepac]

Here is what I did... I have 40k photos and it finished in few seconds, so obviously something was going wrong:

root@b89920c0943c:/# opt/lomorage/bin/lomoc import -p 8001 -n user password /PHOTOS
Login localhost successfully
import photos/videos from /PHOTOS
please check import log /opt/lomorage/var/log/import_PHOTOS.log

Unfortunately, there is not such log file. /opt/lomorage/var doesn't contain log directory

root@b89920c0943c:/# cat /opt/lomorage/var/log/import_PHOTOS.log
cat: /opt/lomorage/var/log/import_PHOTOS.log: No such file or directory

[dwebfan]

@darrepac sorry to hear that, can you please find logs around that period in /opt/lomorage/var/log/lomod.log and /opt/lomorage/var/log/lomod_access.log, and post here? or you can send log from mobile app.

[darrepac]

In fact, the import was running but it crashes after a while:

asset(user_id, hash, year, month, day, ext_id, device_id, latitude, longitude, size, status, create_time, upload_time)
    values(1, 'f2453c38f80152f'/*+25 bytes*/, 2011, 6, 17, 1, 14, 0.0, 0.0, 947690, 1, '2011-06-17 20:4'/*+10 bytes*/, '2022-06-24 11:0'/*+20 bytes*/)
time="2022-06-24T11:01:14Z" level=info msg="created asset 20110617_8193.jpg: f2453c38f80152f6d391cdf39a6b21fcd68f5e22 @ 2011-6-17 from /PHOTOS/Photos_canon/2011/IMG_0689.JPG"
DB [1:14]: insert into asset_album (asset_id, album_id, orig_filename, create_time) values (8193,31,'img_0689.jpg', '2022-06-24 11:0'/*+20 bytes*/)
time="2022-06-24T11:01:15Z" level=warning msg="User 's /media/primary/pascal/Photos/master/2018/01/22/20180122_7005.mov is not found in file system, remove from DB"
DB [1:15]: delete from metadata_similarity where asset_id = 7005
DB [1:15]: delete from metadata_tag where asset_id = 7005
DB [1:15]: delete from metadata_encrypt where asset_id = 7005
DB [1:15]: delete from metadata_geo where asset_id = 7005
DB [1:15]: delete from metadata_scene where asset_id = 7005
DB [1:15]: delete from metadata_face where asset_id = 7005
DB [1:16]: delete from metadata_text where asset_id = 7005
DB [1:18]: delete from metadata_human where asset_id = 7005
DB [1:18]: delete from asset_album_geo where asset_id=7005
DB [1:18]: delete from asset_album_label where asset_id=7005
DB [1:18]: delete from asset_album_camera_make where asset_id=7005
DB [1:18]: delete from asset where id = 7005 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xb30a65]
goroutine 7182 [running]:
bitbucket.org/lomoware/lomo-backend/handler.(*Handler).deleteAsset1.func1({0x17d4180, 0xc003da95c0}, 0xc0000aa000)
    /media/KingFast_e672fee1-90d2-4b78-966c-7cda33684599/Jenkins/workspace/build-lomod-docker-amd64/handler/asset.go:632 +0x225
bitbucket.org/lomoware/lomo-backend/common/dbx.runQuery({0x17d4180, 0xc003da95c0}, 0x203000, 0xc00084f7c8)
    /media/KingFast_e672fee1-90d2-4b78-966c-7cda33684599/Jenkins/workspace/build-lomod-docker-amd64/common/dbx/db.go:209 +0xd7
bitbucket.org/lomoware/lomo-backend/common/dbx.InQuery(0x0, 0xc00084f790)
    /media/KingFast_e672fee1-90d2-4b78-966c-7cda33684599/Jenkins/workspace/build-lomod-docker-amd64/common/dbx/db.go:188 +0x9c
bitbucket.org/lomoware/lomo-backend/handler.(*Handler).deleteAsset1(0xc000348000, 0x1, {0xc000331240, 0x8}, 0x0, 0x2, 0x0)
    /media/KingFast_e672fee1-90d2-4b78-966c-7cda33684599/Jenkins/workspace/build-lomod-docker-amd64/handler/asset.go:620 +0xca
bitbucket.org/lomoware/lomo-backend/handler.(*Handler).fixMissMasterPreviewAssets(0xc000348000, {0xc000874000, 0x4, 0x1})
    /media/KingFast_e672fee1-90d2-4b78-966c-7cda33684599/Jenkins/workspace/build-lomod-docker-amd64/handler/system.go:704 +0xb8b
bitbucket.org/lomoware/lomo-backend/handler.(*Handler).maintenanceScan(0xc000348000, 0x0, 0x0)
    /media/KingFast_e672fee1-90d2-4b78-966c-7cda33684599/Jenkins/workspace/build-lomod-docker-amd64/handler/system.go:655 +0x751
bitbucket.org/lomoware/lomo-backend/handler.(*Handler).maintenance(0xc000348000, 0xc0001ea500, 0xa, 0x0, 0xc0006ae660, 0xc0006ae6c0, 0x0)
    /media/KingFast_e672fee1-90d2-4b78-966c-7cda33684599/Jenkins/workspace/build-lomod-docker-amd64/handler/system.go:569 +0xb1
created by bitbucket.org/lomoware/lomo-backend/handler.NewHandler
    /media/KingFast_e672fee1-90d2-4b78-966c-7cda33684599/Jenkins/workspace/build-lomod-docker-amd64/handler/handler.go:369 +0xd9d

[dwebfan]

sorry, it is one bug. Will fix. Thanks for reporting!

[darrepac]

sorry, it is one bug. Will fix. Thanks for reporting!

Tell me when you have an updated container so I can give a new try

[fuji246]

@darrepac , could you try update docker image? already upload the fixed version.

[darrepac]

Thanks, it is runing now. Is there a way to see the progress. I have 40k photos to scan and would like to see a least a counter 434 / 40 000 or something on the logs (or UI) to know where I am.

[darrepac]

It crashed again, here is the end of the logs:

2022-06-27T07:15:12.180307618Z   asset(user_id, hash, year, month, day, ext_id, device_id, latitude, longitude, size, status, create_time, upload_time)
2022-06-27T07:15:12.180324127Z  values(1, 'c5504a13c38eb2d'/*+25 bytes*/, 2005, 7, 8, 1, 14, 0.0, 0.0, 2836560, 1, '2005-07-08 14:1'/*+10 bytes*/, '2022-06-27 07:1'/*+20 bytes*/)
2022-06-27T07:15:12.220228156Z time="2022-06-27T07:15:12Z" level=info msg="created asset 20050708_9032.jpg: c5504a13c38eb2d2080d10eac0a3d77c7f5d27ac @ 2005-7-8 from /PHOTOS/Photos Pyrenees/Best Off/DSC_5729.JPG"
2022-06-27T07:15:12.402887850Z DB [15:12]: insert into asset_album (asset_id, album_id, orig_filename, create_time) values (9032,18,'dsc_5729.jpg', '2022-06-27 07:1'/*+20 bytes*/)
2022-06-27T07:15:14.202971849Z time="2022-06-27T07:15:14Z" level=info msg="start create asset: /PHOTOS/Photos Pyrenees/DVD/Etape6/DSC_5087.JPG (jpg) for user: 1 from 14"
2022-06-27T07:15:14.203680371Z DB [15:14]: insert into 
2022-06-27T07:15:14.203713635Z   asset(user_id, hash, year, month, day, ext_id, device_id, latitude, longitude, size, status, create_time, upload_time)
2022-06-27T07:15:14.203728074Z  values(1, '1e27033626717f7'/*+25 bytes*/, 2005, 7, 2, 1, 14, 0.0, 0.0, 2815933, 1, '2005-07-02 07:0'/*+10 bytes*/, '2022-06-27 07:1'/*+20 bytes*/)
2022-06-27T07:15:14.204341226Z time="2022-06-27T07:15:14Z" level=info msg="created asset 20050702_9033.jpg: 1e27033626717f788c2b63dc8cbe67a7836cf1d3 @ 2005-7-2 from /PHOTOS/Photos Pyrenees/DVD/Etape6/DSC_5087.JPG"
2022-06-27T07:15:14.417016599Z DB [15:14]: insert into asset_album (asset_id, album_id, orig_filename, create_time) values (9033,46,'dsc_5087.jpg', '2022-06-27 07:1'/*+20 bytes*/)
2022-06-27T07:15:14.678267783Z time="2022-06-27T07:15:14Z" level=info msg="finish scan from 2022-06-27T06:15:14Z to 0001-01-01T00:00:00Z"
2022-06-27T07:15:14.679479355Z time="2022-06-27T07:15:14Z" level=info msg="FINISH: totally scanned 23 directories, and imported 840 media files"
2022-06-27T07:15:15.170915500Z panic: send on closed channel
2022-06-27T07:15:15.171144732Z 
2022-06-27T07:15:15.172295963Z goroutine 34467 [running]:
2022-06-27T07:15:15.225105601Z bitbucket.org/lomoware/lomo-backend/common/scan.(*Runner).reportStats(0xc000494000, 0x34630b8a000, 0xc000fdf6e0, 0x1873df0, 0xc00103c5c0)
2022-06-27T07:15:15.233939546Z  /media/KingFast_e672fee1-90d2-4b78-966c-7cda33684599/Jenkins/workspace/build-lomod-docker-amd64/common/scan/scan.go:180 +0x58e
2022-06-27T07:15:15.233988555Z created by bitbucket.org/lomoware/lomo-backend/common/scan.(*Runner).Start
2022-06-27T07:15:15.234003641Z  /media/KingFast_e672fee1-90d2-4b78-966c-7cda33684599/Jenkins/workspace/build-lomod-docker-amd64/common/scan/scan.go:121 +0x325

[dwebfan]

@darrepac sorry to hear that. It is caused by one race condition btw asynchronous channel communication. I'll provide a fix this week. Thanks for all feedback and support!

[fuji246]

@darrepac, we pushed the fix , could you try again with the latest version?

[darrepac]

ok So far the log is full of such message, I don't know why (tried to restart but was the same). I did not even try to import anything...I will wait and see....

2022-07-04T06:57:25.681990598Z time="2022-07-04T06:57:25Z" level=warning msg="[/media/primary/pascal/Photos/master/2005/07/08/20050708_8577.jpg /media/primary/pascal/Photos/preview/2005/07/08/20050708_8577] - /media/primary/pascal/Photos/preview/2005/07/08/20050708_8577_640_0.webp exif analysis got: exiftool analysis /media/primary/pascal/Photos/preview/2005/07/08/20050708_8577_640_0.webp got Error: File not found - /media/primary/pascal/Photos/preview/2005/07/08/20050708_8577_640_0.webp\n 1 files could not be read\n 0 output files created\n: exit status 1"

[darrepac]

Seems import failed:

/# /opt/lomorage/bin/lomoc import -p 8001 -n user password /PHOTOS/
2022/07/04 09:56:38 {"id": "0", "text": "sql: no rows in result set"}

[dwebfan]

@darrepac thanks for trying. If possible, could you please send log at mobile app's own setting page? We can analysis and find the root cause.

[darrepac]

Don't know why but now it imports photos ok...let see if it crashes after a while

[dwebfan]

thanks for update. Please let us know if you see any problems.