Exclude video - Githubissues

darrepac commented 2 years ago

Hi,

It seems that I will have to import again my 40k assets library. It is quite slow and particularly due to video files (transcoding). Is there a way to:

exclude video files (in order to speed up first scan) and include it later
do not transcode video files. I could want to stream directly my video files instead of having a transcoded version of it (and it would speed up the scan by avoiding ffmpeg transcoding)

thanks

fuji246 commented 2 years ago

Currently not supported yet, will discuss with team on this.

fuji246 commented 2 years ago

We will provide cmd line option to skip video transcoding when import.

fuji246 commented 2 years ago

@darrepac , exclude video is added, you can update lomoc to try again.

+               cli.BoolFlag{
+                   Name:  "no-video, nv",
+                   Usage: "not scan video files. This is to speed up the first import process",
+               },

darrepac commented 2 years ago

Hi,

Problem is that even if I stop the container and update it, when I launch it, it starts again the previously stopped process... How can I stop the job and start a new one with the no-video option?

thanks

Le dim. 28 août 2022 à 10:20, Jiantao Fu @.***> a écrit :

@darrepac https://github.com/darrepac , exclude video is added, you can update lomoc to try again.

cli.BoolFlag{

Name: "no-video, nv",

Usage: "not scan video files. This is to speed up the first import process",

},

— Reply to this email directly, view it on GitHub https://github.com/lomorage/homepage/issues/24#issuecomment-1229397250, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABWMPIVIPIQVNXVLPVOGF5TV3MHNHANCNFSM53J5LOTA . You are receiving this because you were mentioned.Message ID: @.***>

dwebfan commented 2 years ago

@darrepac sorry to hear that. Can you please describe the problem more? Do you mean restarting container will resume last import automatically?

Importer is a little weak on our side now. It is the top priority now, and will update you shortly.

Regarding this suggestion, just wonder if you use our mobile app to watch video before, and what is your user experience? How is your network speed? and how powerful is your server CPU? Is it enough to download original resolution video? Do you like the idea to do just in time transcoding using ffmpeg while downloading?

do not transcode video files. I could want to stream directly my video files instead of having a transcoded version of it (and it would speed up the scan by avoiding ffmpeg transcoding)

darrepac commented 2 years ago

@darrepac sorry to hear that. Can you please describe the problem more? Do you mean restarting container will resume last import automatically?

Well, I don't really understand what's going on. I just restarted the container to check the logs. I see: time="2022-08-30T18:51:48Z" level=info msg="start generate missed preview" then a lot of message like this (still counting): time="2022-08-30T18:52:28Z" level=warning msg="[/media/primary/pascal/Photos/master/2018/07/25/20180725_77850.jpg /media/primary/pascal/Photos/preview/2018/07/25/20180725_77850] - /media/primary/pascal/Photos/preview/2018/07/25/20180725_77850_640_0.webp exif analysis got: exiftool analysis /media/primary/pascal/Photos/preview/2018/07/25/20180725_77850_640_0.webp got Error: File not found - /media/primary/pascal/Photos/preview/2018/07/25/20180725_77850_640_0.webp\n 1 files could not be read\n 0 output files created\n: exit status 1"

darrepac commented 2 years ago

Importer is a little weak on our side now. It is the top priority now, and will update you shortly.

Regarding this suggestion, just wonder if you use our mobile app to watch video before, and what is your user experience? How is your network speed? and how powerful is your server CPU? Is it enough to download original resolution video? Do you like the idea to do just in time transcoding using ffmpeg while downloading?

do not transcode video files. I could want to stream directly my video files instead of having a transcoded version of it (and it would speed up the scan by avoiding ffmpeg transcoding)

First: My server is a dell latitude laptop E6410 (intel Core i5 M560 and ssd for database and cache, the original photos being on an hdd). Obivously it is not the most powerful machine, but it is not a Raspberry either.

Second: I did not really get the occasion to well test the lomorage app due to 2 main problems: import is running since weeks (so while it is importing I cannot do much things on the server). Could make sense to lower the priority of the import process by the way in order to let the server run is other tasks well. Second problem was my wan network which is quite weak but I can test on my lan network for the moment (I should get a gigabit fiber internet at home the Sept 9).

My suggestion is may-be not well written and should be more written from user point of view instead of technical point of view: lomorage is too too slow importing video. All other solutions I tested were faster (from not transcoding, like plex) to may-be 1 transcoding but lomorage is doing many resolution transcoding... or may-be ffmpeg is less optimized, I don't know.

Hope it helps :)

darrepac commented 2 years ago

@darrepac sorry to hear that. Can you please describe the problem more? Do you mean restarting container will resume last import automatically?

Well, I don't really understand what's going on. I just restarted the container to check the logs. I see: time="2022-08-30T18:51:48Z" level=info msg="start generate missed preview" then a lot of message like this (still counting): time="2022-08-30T18:52:28Z" level=warning msg="[/media/primary/pascal/Photos/master/2018/07/25/20180725_77850.jpg /media/primary/pascal/Photos/preview/2018/07/25/20180725_77850] - /media/primary/pascal/Photos/preview/2018/07/25/20180725_77850_640_0.webp exif analysis got: exiftool analysis /media/primary/pascal/Photos/preview/2018/07/25/20180725_77850_640_0.webp got Error: File not found - /media/primary/pascal/Photos/preview/2018/07/25/20180725_77850_640_0.webp\n 1 files could not be read\n 0 output files created\n: exit status 1"

I stopped the container waiting for your insights cause I am a bit afraid that it is resetting again all the db…

dwebfan commented 2 years ago

@darrepac Is it possible to tar all log files under /opt/lomorage/var/log, and send to me dwebfan@gmail.com. I guess it is related with the files deleted at original location, but not so sure, so hope to get more logs to analysis. Thanks again.

darrepac commented 2 years ago

There is nothing in /opt/lomorage/var but I sent you by email the log from the app

fuji246 commented 1 year ago

It's already added:

import command line tool use "--no-video" option to skip video import, https://docs.lomorage.com/docs/Usage/lomorage-client/#import-from-one-existing-folders
when running lomod, use --preview-size-video=0 to disable video preview generation , https://github.com/lomorage/lomod#preview-size-video

darrepac commented 1 year ago

👍

Le 7 août 2022 à 05:05, Jiantao Fu @.***> a écrit :

We will provide cmd line option to skip video transcoding when import.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

fuji246 commented 1 year ago

close it now, feel free to reopen if it still happens.

lomorage / homepage

Exclude video #24