AI: Support inference with Coral Edge TPU USB devices and TFlite instead of using the CPU

[yayitazale]

As a user with a Coral Edge TPU USB device, I would like to have it supported by PhotoPrism to speed up AI inference when indexing pictures.

Image Classification using coral edge tpu would improve drastically the use of CPU. Is there any plan to add support of this device to increase the speed of classification reducing the amount of CPU needed?

Thanks for all the amazing work.

Depends On:

• 222

Related Issues:

• https://github.com/photoprism/photoprism/issues/1090
• https://github.com/photoprism/photoprism/issues/3438
• https://github.com/photoprism/photoprism/issues/4356
• https://github.com/photoprism/photoprism/issues/907

[lastzero]

Certainly something we can explore after a stable release!

[Shamshala]

I bought two DualEdgeTPUs recently and have one USB unit as well so i can offer any of them for testing (probably wouldn't be a problem to create VM specific for this and share it remotely). :smiley:

[mannp]

As a new user importing a lot of photos, I would definitely appreciate a speedier initial import :)

Thanks for creating photoprism :)

[srvoets]

Just wanna say +1 for coral, I have a coral USB for frigate and my CPU usage has been DRASTICALLY reduced. Another good option, IMO is to have a variable for detector, I am running deepstack for frigate, would be nice to point it to my deepstack URL.

Im importing all of my icloud photos right now and my CPU is hurting

[TapuGithub]

Could be relevant - https://github.com/photoprism/photoprism/issues/731

[mike-lloyd03]

I too have a TPU for Frigate and would love to see this implemented for PhotoPrism. Thanks!

[DrSpaldo]

As above, I am also using Coral for Frigate and it is amazing the difference it makes. If this is something that would reduce the CPU load on PhotoPrism, that would be great. They are pretty reasonable in price for the benefit it gives

[lastzero]

There now is an optional PHOTOPRISM_INIT environment variable that can be used to install optional dependencies (when starting the container as root for the first time). You can already enable AVX2 support for modern Intel CPUs to improve TensorFlow performance, see https://github.com/photoprism/photoprism/blob/develop/docker/examples/docker-compose.yml#L84.

Feel free to send pull requests for other CPUs and hardware devices (requires a TensorFlow lib compiled with support for the specific hardware and docs if hardware devices need to be mounted):

https://github.com/photoprism/photoprism/blob/develop/docker/scripts/Makefile

[yayitazale]

There now is an optional PHOTOPRISM_INIT environment variable that can be used to install optional dependencies (when starting the container as root for the first time). You can already enable AVX2 support for modern Intel CPUs to improve TensorFlow performance, see https://github.com/photoprism/photoprism/blob/develop/docker/examples/docker-compose.yml#L84.

Feel free to send pull requests for other CPUs and hardware devices (requires a TensorFlow lib compiled with support for the specific hardware and docs if hardware devices need to be mounted):

https://github.com/photoprism/photoprism/blob/develop/docker/scripts/Makefile

I'm not a programmer but I'll try to help:

TPU hardware doesn't work with TF, instead you need to install TFlite:

https://www.tensorflow.org/lite/

Installing the runtime (this is what frigate does):

RUN apt-get -qq update \
    && apt-get -qq install --no-install-recommends -y \
        ## Tensorflow lite
    && wget -q https://github.com/google-coral/pycoral/releases/download/release-frogfish/tflite_runtime-2.5.0-cp38-cp38-linux_x86_64.whl \
    && python3.8 -m pip install tflite_runtime-2.5.0-cp38-cp38-linux_x86_64.whl \
    && rm tflite_runtime-2.5.0-cp38-cp38-linux_x86_64.whl \
    && rm -rf /var/lib/apt/lists/* \
    && (apt-get autoremove -y; apt-get autoclean -y)

The TF lite library for object detection (from https://tfhub.dev/tensorflow/lite-model/ssd_mobilenet_v1/1/metadata/1?lite-format=tflite): https://tfhub.dev/tensorflow/lite-model/ssd_mobilenet_v1/1/metadata/1?lite-format=tflite

The TPU is mounted with this: https://github.com/blakeblackshear/frigate/blob/master/frigate/edgetpu.py

[rickysarraf]

Following up on my comment mistakenly added to #1337

Did you add

PHOTOPRISM_INIT: "intel-graphics tensorflow-amd64-avx2"

to your docker-compose.yml and start the container as root?

Oh!! Yes. I did. That's when I started seeing those (new) messages.

Video transcoding using Intel QSV is working for me after the last commit (not in the Development Preview yet!):

Forgive my ignorance but I was under the impression that this feature is also for tensorflow.

Videos is not an issue to me. All my videos, I manually re-encode before feeding to PhotoPrism.

Note: This comment was for #536 I just mixed up the issues.

I finally got it working.

• Run docker as root so that it can pull in the relevant tensorflow files
• docker commit the built image so that it is not lost in the next normal user run.

With that, I have:

Sep 29 19:20:46 lenovo docker[273902]: 2021-09-29 13:50:46.898444: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /photoprism/assets/nasnet
Sep 29 19:20:47 lenovo docker[273902]: 2021-09-29 13:50:47.022999: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { photoprism }
Sep 29 19:20:47 lenovo docker[273902]: 2021-09-29 13:50:47.680883: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
Sep 29 19:20:49 lenovo docker[273902]: 2021-09-29 13:50:49.787710: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { photoprism }; Status: success. Took 2889282 microseconds.

[janstadt]

@rickysarraf did you have to do everything that @yayitazale mentioned or did you just have to pass in the new PHOTOPRISM_INIT info for tensorflow?

[yayitazale]

@rickysarraf did you have to do everything that @yayitazale mentioned or did you just have to pass in the new PHOTOPRISM_INIT info for tensorflow?

As I'm understanding correctly, he is just launching the complete TensorFlow to use it with nvidia GPUs. My instructions/help tips are to implement the TensorFlow lite runtime witch is needed to run a tflite model over a dedicated hardware like Coral Edge Tpu. That implementation needs to be done apart of the gpu/tensorflow.

[janstadt]

@rickysarraf did you have to do everything that @yayitazale mentioned or did you just have to pass in the new PHOTOPRISM_INIT info for tensorflow?

As I'm understanding correctly, he is just launching the complete TensorFlow to use it with nvidia GPUs. My instructions/help tips are to implement the TensorFlow lite runtime witch is needed to run a tflite model over a dedicated hardware like Coral Edge Tpu. That implementation needs to be done apart of the gpu/tensorflow.

Thanks for the clarification. Have you successfully installed TFLite in the container and is your coral edge working with it? I'd love to have a step by step example showing this if possible (similar to frigate). I am also having some issues on my QNAP with this container using the settings mentioned in this issue: https://github.com/photoprism/photoprism/issues/1337#issuecomment-930172631 so it seems like i need to get that taken care of before i move on to TFLite.

[yayitazale]

Sadly I don't have the skills to make it...

[rickysarraf]

@rickysarraf did you have to do everything that @yayitazale mentioned or did you just have to pass in the new PHOTOPRISM_INIT info for tensorflow?

I haven't really spent any time exploring how PhotoPrism leverages the TensorFlow stuff. Only when I saw @lastzero commit about support for x86, I checked my machine's capability.

My expectation is that with the proper cpu support, object detection and classification using Tensorflow, should now be improved.

[janstadt]

Frigate is a python app but there looks to be a tensorflow lite binding in golang (with TPU support) here: https://github.com/mattn/go-tflite I am unfamiliar with golang but i'll take a closer look. Somone with more experience might be faster to produce something useful.

[lastzero]

You are asking us to maintain additional TF lite models plus corresponding test suites so that you can use entry level TPUs instead of standard hardware like GPUs / Intel AVX?

[janstadt]

No im not asking for maintenance but to provide extensibility points and examples for folks who have other hardware. The PHOTOPRISM_INIT seems like a good starting point but it would be nice to provide the ability to pass in our own makefile (for this) on top of the existing makefile maybe. Or i guess some way to offload the TF functionality to a different container with some api like deepstack which i believe has the anility to leverage tflite and coral. I understand this is OSS and you are doing great work and have other priorities so i figure id see if anyone else has the ability and time to investigate.

[lastzero]

You're welcome to send a pull request. For private use, you can simply create a custom Dockerfile extending our image. If you want to use our existing models, we need to convert them. If not, we are not talking about TPU support but a general API to use external services for indexing. Doesn't sound like a quick win.

[srett]

There now is an optional PHOTOPRISM_INIT environment variable that can be used to install optional dependencies (when starting the container as root for the first time). You can already enable AVX2 support for modern Intel CPUs to improve TensorFlow performance, see https://github.com/photoprism/photoprism/blob/develop/docker/examples/docker-compose.yml#L84.

Feel free to send pull requests for other CPUs and hardware devices (requires a TensorFlow lib compiled with support for the specific hardware and docs if hardware devices need to be mounted):

https://github.com/photoprism/photoprism/blob/develop/docker/scripts/Makefile

Maybe this is a little overkill, but I just had the idea to automatically detect the CPU capabilities and then download a properly optimized libtensorflow.
I wrote a bash script that could replace the Makefile in this regard. It would have to be called from the entrypoint.sh in some way (maybe even by default?).

Assumptions this script makes: /proc/cpuinfo is accessible and contains correct information, and uname -m properly returns x86_64 when running on according hardware.

Call the script with no arguments or "auto" to download the best libtensorflow version available, or pass "avx2" "avx" etc. as the first argument to explicitly download the according flavor.
The script remembers which version was downloaded last time, and does nothing if you request the same version again. So you could have another env var like PHOTOPRISM_TF_FLAVOR that defaults to auto (or cpu to be safe), and in entrypoint.sh just unconditionally call download-tf-script.sh "$PHOTOPRISM_TF_FLAVOR".

With this, new optimized builds can be added easily: Add a single line to the script and upload the according precompiled lib. I already added a line for the old Athlon X2 I'm running PhotoPrism on, so I could check the cpu flags as returned by /proc/cpuinfo, and to verify that the script would continue with the next-best version if curl/tar fails (because obviously no such build exists for a 12 year old CPU). Feel free to remove that line. :-)

#!/usr/bin/env bash

if [ "$( uname -m )" != "x86_64" ]; then
    # TODO....?
    echo "Not running on AMD64"
    exit 1
fi

declare -rg TF_VERSION="1.15.2"
declare -rg VERSION_FILE="/root/.tf-version"

tf_install()
{
    local url dst flavor ret
    flavor="$1"
    # Do nothing if already installed; delete $VERSION_FILE to force re-download
    [[ "$( cat "$VERSION_FILE" 2> /dev/null )" = "${flavor}-${TF_VERSION}" ]] && return 0
    url="https://dl.photoprism.org/tensorflow/linux/libtensorflow-linux-${flavor}-${TF_VERSION}.tar.gz"
    dst="/tmp/${url##*/}"
    ret=1
    if ! curl -L --retry 4 --retry-max-time 60 --max-time 60 --retry-connrefused --connect-timeout 3 \
            --fail --show-error "$url" > "$dst" 2> "${dst}.err"; then
        echo "Error downloading ${url}:"
        cat "${dst}.err"
    else
        echo "Downloaded TensorFlow flavor: $flavor"
        if tar --overwrite -C "/usr" -xf "$dst"; then
            ldconfig
            # Remember version for next time
            echo "${flavor}-${TF_VERSION}" > "$VERSION_FILE"
            ret=0
        fi
    fi
    rm -f -- "${dst}" "${dst}.err"
    return "$ret"
}

tf_install_best()
{
    local cpu key ok
    declare -a flavors a

    cpu=" $( awk -F: '$1 ~ /^flags/ {print $2; exit}' /proc/cpuinfo ) "
    # Ordered from best to worst
    # First element is the flavor; cannot use assoc array as they don't guarantee any order
    # Required flags collected from https://gcc.gnu.org/onlinedocs/gcc-4.8.5/gcc/i386-and-x86-64-Options.html
    flavors+=("avx2      avx aes sse4_1 sse4_2 fsgsbase rdrand pclmulqdq f16c avx2 fma bmi1 bmi2")
    flavors+=("avx       avx aes sse4_1 sse4_2 fsgsbase rdrand pclmulqdq f16c")
    flavors+=("amdfam10  sse4a 3dnow 3dnowext abm")
    flavors+=("cpu")

    for flavor in "${flavors[@]}"; do
        a=($flavor)
        key="${a[0]}"
        ok=1
        for flag in "${a[@]:1}"; do
            if [[ "$cpu" != *" $flag "* ]]; then
                #echo "CPU doesn't have $flag, required for $key"
                ok=
                break
            fi
        done
        if [ -n "$ok" ]; then
            tf_install "$key" && return 0
            # On failure, the loop continues with the next flavor
        fi
    done
}

if [ -n "$1" ] && [ "$1" != "auto" ]; then
    tf_install "$1"
else
    tf_install_best
fi

[rickysarraf]

I get this message with TF, when indexing. I hope this is normal message.

2021-10-05 08:20:07.910243: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /photoprism/assets/facenet
2021-10-05 08:20:07.985143: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2021-10-05 08:20:08.129693: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2021-10-05 08:20:08.129766: I tensorflow/cc/saved_model/loader.cc:212] The specified SavedModel has no variables; no checkpoints were restored. File does not exist: /photoprism/assets/facenet/variables/variables.index
2021-10-05 08:20:08.129796: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 219559 microseconds.

[carck]

Frigate is a python app but there looks to be a tensorflow lite binding in golang (with TPU support) here: https://github.com/mattn/go-tflite I am unfamiliar with golang but i'll take a closer look. Somone with more experience might be faster to produce something useful.

I have integrated go-tflite in my fork, things work, need to verify the accuracy, but it is far awary from a pull request.

[m0ngr31]

Would love to get this added

[lastzero]

Would love to add this: https://docs.photoprism.app/funding/

[ppriorfl]

Would LOVE to have coral TPU support - I have 30,000 photos to import and I started the process and was very slow and maxing all cores on my server so I stopped. Offloading to TPU would be awesome.

[IssueHuntBot]

An anonymous user has funded $50.00 to this issue.

---

• Submit pull request via IssueHunt to receive this reward.
• Want to contribute? Chip in to this issue via IssueHunt.
• Checkout the IssueHunt Issue Explorer to see more funded issues.
• Need help from developers? Add your repository on IssueHunt to raise funds.

[yayitazale]

Would love to add this: https://docs.photoprism.app/funding/

Donated 50€ to help.

[lastzero]

I'm looking forward to start a battle with TensorFlow as soon as I have some time ;) Not sure when that will be....

[mike-lloyd03]

I just looked around for a Go library that interfaces with the Coral TPU and it looks like there currently isn't any support. Just Python and C++. :(

[graciousgrey]

Please don't add funds to IssueHunt anymore! 🎁

While we like IssueHunt and are grateful for the donations we've received so far, it hasn't proven to be a sustainable funding option for us as we spend much of our time maintaining existing features and providing support.

If we don't have enough resources to provide support and bugfixes, we can't start working on new features.

[ivorip]

Looking around, there aren't many TPU integration other than qnap's qumagie. I personally don't have a Coral, but seems very useful to enable older hardware for faster indexing.

[dotWee]

Please don't add funds to IssueHunt anymore! 🎁

While we like IssueHunt and are grateful for the donations we've received so far, it hasn't proven to be a sustainable funding option for us as we spend much of our time maintaining existing features and providing support.

If we don't have enough resources to provide support and bugfixes, we can't start working on new features.

I would be more than glad to donate a fresh Google Coral Dev Board for future development. They are currently difficult to get in Germany, I got a spare one laying around. Let me know your team could use one!

[ppriorfl]

Not just “older” hardware - I have 30,000+ images that need indexing and running photoprism pegs all 48 cores in my threadripper for hours and hours and hours.

On Jan 19, 2022, at 11:18 AM, ivorip @.***> wrote:

Looking around, there aren't many TPU integration other than qnap's qumagie. I personally don't have a Coral, but seems very useful to enable older hardware for faster indexing.

— Reply to this email directly, view it on GitHub https://github.com/photoprism/photoprism/issues/536#issuecomment-1016630960, or unsubscribe https://github.com/notifications/unsubscribe-auth/APKLQGPTFRKV3O4PTD2M5YTUW3P47ANCNFSM4SHHKW5A. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you commented.

[ivorip]

@ppriorfl Good to know, 30k is about 2% of my catalog. Certainly, average to semi-pro photographers will have more. Currently on capture 1 with prodibi. I'll bookmark photoprism until it can be of more commercial use

[lastzero]

Not just “older” hardware - I have 30,000+ images that need indexing and running photoprism pegs all 48 cores in my threadripper for hours and hours and hours.

I have a very old Core i3 with 2 cores for a library of 100k+ files. It also runs many other services. A full rescan (which is not needed very often) with max quality takes about 8-12h. If 48 threads are slower for 30k pictures, there are other issues than TensorFlow. Could be locking issues or slow I/O. RAW conversion and thumbs generation can be demanding. Possible that some users don't look closely and then blame (traditionally slow) RAW to JPEG conversion on PhotoPrism or TensorFlow... Sure, we could extract low res previews from the RAW headers like "faster" tools, but it's a different quality.

[devova]

Well, having a strong distributed plug-in approach would help with this. Exposing hooks would allow making extensions either in Go or as separate docker containers for those who don't familiar with Golang. So it should look like this:

[ppriorfl]

All my files are RAW. If slow I/O was the issue I would not expect all my cores running 100%. Not denigrating the project by any means, just hoping the extra hardware that I have could be integrated as other applications show amazing improvement with TPU use.

On Jan 21, 2022, at 12:33 AM, Michael Mayer @.***> wrote:

Not just “older” hardware - I have 30,000+ images that need indexing and running photoprism pegs all 48 cores in my threadripper for hours and hours and hours.

I have a very old Core i3 with 2 cores for a library of 100k+ files. It also runs many other services. A full rescan (which is not needed very often) with max quality takes about 8-12h. If 48 threads are slower for 30k pictures, there are other issues than TensorFlow. Could be locking issues or slow I/O. RAW conversion and thumbs generation can be demanding. Possible that some users don't look closely and then blame (traditionally slow) RAW to JPEG conversion on PhotoPrism or TensorFlow... Sure, we could extract low res previews from the RAW headers like "faster" tools, but it's a different quality.

— Reply to this email directly, view it on GitHub https://github.com/photoprism/photoprism/issues/536#issuecomment-1018195987, or unsubscribe https://github.com/notifications/unsubscribe-auth/APKLQGKPJGJZZMDB4QV4E4TUXDV4FANCNFSM4SHHKW5A. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.

[lastzero]

RAWs are very demanding for the CPU too. It's absolutely possible TensorFlow is not the (main) problem. Try by reindexing when JPEGs and thumbnails already exist.

[srett]

Not just “older” hardware - I have 30,000+ images that need indexing and running photoprism pegs all 48 cores in my threadripper for hours and hours and hours.

Mine is close to 40k, and the initial indexing took a few days on a 12 y/o AMD dual core. And frankly, while I was eager to try out PP on my full collection and wanted it to be finished already, in reality, I quickly realized that stuff that happens in the background isn't that performance critical. It's more about the UX when browsing, which should be polished and snappy. You import your collection once and then only incrementally add a comparably small amount of new photos over time. Let's say a couple hundred at once. Do you really care if it takes 1 or 10 minutes until they are fully indexed?

While obviously there's never going to be that one thing to work on that 100% of your users want/need, I think it's safe to say adding TFLite support would benefit a small minority of users only, compared to most things on the roadmap. That isn't to say that someone could step in and just go ahead and implement it and open a PR for it. The problem is that the people who say it's "just using this lib here" seem to completely ignore that you need to add switches/GUI to enable it, configure it, think of all the ways in which it could fail, or be different from your test setup and handle it gracefully, etcpp. I guess that's why there is no such PR yet.

[j0rd]

Not just “older” hardware - I have 30,000+ images that need indexing and running photoprism pegs all 48 cores in my threadripper for hours and hours and hours.

I have a very old Core i3 with 2 cores for a library of 100k+ files. It also runs many other services. A full rescan (which is not needed very often) with max quality takes about 8-12h. If 48 threads are slower for 30k pictures, there are other issues than TensorFlow. Could be locking issues or slow I/O. RAW conversion and thumbs generation can be demanding. Possible that some users don't look closely and then blame (traditionally slow) RAW to JPEG conversion on PhotoPrism or TensorFlow... Sure, we could extract low res previews from the RAW headers like "faster" tools, but it's a different quality.

I'll provide some benchmarking. These images I'm processing are 16MB jpegs. I'll provide my specs below

Creating the 6 thumbs takes around 5-7 seconds Face label takes 200-1200 ms when one is found. Things label takes 12-16 seconds when one is found.

Problem with debug information, is when a face or a label is not found, the time spent doing this calculation is not displayed in debug information. I would suspect that Face and Things Label take time when none are found.

I would recommend outputting the time spent doing Face and Things label, even if none are found. I would also recommend displaying this debug information on the frontend, so people can debug why things are slow.

Here's my logs. 16MB JPEGs

INFO[2022-01-24T18:52:37+07:00] index: 'GooglePhotos/- Check before delete/0 - 19-05-2558-1 - Beau-Nick-Vik/_DSC1479.jpg' has instance_id 59ed0a07-0c1d-1447-a8e1-7139464ac082 
INFO[2022-01-24T18:52:37+07:00] media: _DSC1479.jpg was taken at 2015-05-19 14:40:26 +0000 UTC (meta) 
INFO[2022-01-24T18:52:41+07:00] media: created 6 thumbnails for _DSC1476 [5.703545804s] 
INFO[2022-01-24T18:52:41+07:00] index: added main jpg file 'GooglePhotos/- Check before delete/0 - 19-05-2558-1 - Beau-Nick-Vik/_DSC1476.jpg' 
INFO[2022-01-24T18:52:42+07:00] media: created 6 thumbnails for _DSC1477 [6.816283902s] 
INFO[2022-01-24T18:52:42+07:00] index: added main jpg file 'GooglePhotos/- Check before delete/0 - 19-05-2558-1 - Beau-Nick-Vik/_DSC1477.jpg' 
INFO[2022-01-24T18:52:46+07:00] index: found 1 face in _DSC1480.jpg [509.2115ms] 
INFO[2022-01-24T18:52:48+07:00] index: found 1 face in _DSC1481.jpg [200.431309ms] 
INFO[2022-01-24T18:52:49+07:00] index: 'GooglePhotos/- Check before delete/0 - 19-05-2558-1 - Beau-Nick-Vik/_DSC1480.jpg' has instance_id cd9d5e4c-9318-e74d-8b42-49ba372a15ac 
INFO[2022-01-24T18:52:49+07:00] media: _DSC1480.jpg was taken at 2015-05-19 14:40:37 +0000 UTC (meta) 
INFO[2022-01-24T18:52:52+07:00] media: created 6 thumbnails for _DSC1478 [5.900237575s] 
INFO[2022-01-24T18:52:52+07:00] index: added main jpg file 'GooglePhotos/- Check before delete/0 - 19-05-2558-1 - Beau-Nick-Vik/_DSC1478.jpg' 
INFO[2022-01-24T18:52:52+07:00] index: 'GooglePhotos/- Check before delete/0 - 19-05-2558-1 - Beau-Nick-Vik/_DSC1481.jpg' has instance_id b15b2b79-c5a9-9b4b-b5e9-24157e021a5f 
INFO[2022-01-24T18:52:52+07:00] media: _DSC1481.jpg was taken at 2015-05-19 14:40:57 +0000 UTC (meta) 
INFO[2022-01-24T18:52:54+07:00] index: matched 2 labels with _DSC1479.jpg [17.002481224s] 
INFO[2022-01-24T18:52:57+07:00] index: found 1 face in _DSC1482.jpg [279.82803ms] 
INFO[2022-01-24T18:53:01+07:00] media: created 6 thumbnails for _DSC1479 [6.57828908s] 
INFO[2022-01-24T18:53:01+07:00] index: added main jpg file 'GooglePhotos/- Check before delete/0 - 19-05-2558-1 - Beau-Nick-Vik/_DSC1479.jpg' 
INFO[2022-01-24T18:53:02+07:00] index: 'GooglePhotos/- Check before delete/0 - 19-05-2558-1 - Beau-Nick-Vik/_DSC1482.jpg' has instance_id 8657c44b-b737-8342-89fc-db9e006883ec 
INFO[2022-01-24T18:53:02+07:00] media: _DSC1482.jpg was taken at 2015-05-19 14:41:11 +0000 UTC (meta) 
INFO[2022-01-24T18:53:10+07:00] media: created 6 thumbnails for _DSC1480 [6.11284825s] 
INFO[2022-01-24T18:53:10+07:00] index: added main jpg file 'GooglePhotos/- Check before delete/0 - 19-05-2558-1 - Beau-Nick-Vik/_DSC1480.jpg' 
INFO[2022-01-24T18:53:10+07:00] index: 'GooglePhotos/- Check before delete/0 - 19-05-2558-1 - Beau-Nick-Vik/_DSC1483.jpg' has instance_id 35a3fab6-6b8a-ed48-a5e5-c9a7a800eae6 
INFO[2022-01-24T18:53:10+07:00] media: _DSC1483.jpg was taken at 2015-05-19 14:41:18 +0000 UTC (meta) 
INFO[2022-01-24T18:53:14+07:00] media: created 6 thumbnails for _DSC1481 [7.055038104s] 
INFO[2022-01-24T18:53:14+07:00] index: added main jpg file 'GooglePhotos/- Check before delete/0 - 19-05-2558-1 - Beau-Nick-Vik/_DSC1481.jpg' 
INFO[2022-01-24T18:53:16+07:00] index: found 1 face in _DSC1484.jpg [218.847464ms] 
INFO[2022-01-24T18:53:20+07:00] index: found 1 face in _DSC1485.jpg [207.335542ms] 
INFO[2022-01-24T18:53:20+07:00] index: 'GooglePhotos/- Check before delete/0 - 19-05-2558-1 - Beau-Nick-Vik/_DSC1484.jpg' has instance_id 62935ff9-a44a-6940-a648-fb78208c35ce 
INFO[2022-01-24T18:53:20+07:00] media: _DSC1484.jpg was taken at 2015-05-19 14:41:34 +0000 UTC (meta) 
INFO[2022-01-24T18:53:22+07:00] index: 'GooglePhotos/- Check before delete/0 - 19-05-2558-1 - Beau-Nick-Vik/_DSC1485.jpg' has instance_id f90157ab-49c3-1f45-83a1-70d81d1173a0 
INFO[2022-01-24T18:53:22+07:00] media: _DSC1485.jpg was taken at 2015-05-19 14:41:36 +0000 UTC (meta) 
INFO[2022-01-24T18:53:25+07:00] media: created 6 thumbnails for _DSC1482 [5.735262559s] 
INFO[2022-01-24T18:53:25+07:00] index: added main jpg file 'GooglePhotos/- Check before delete/0 - 19-05-2558-1 - Beau-Nick-Vik/_DSC1482.jpg' 
INFO[2022-01-24T18:53:29+07:00] media: created 6 thumbnails for _DSC1483 [7.287027045s] 
INFO[2022-01-24T18:53:29+07:00] index: added main jpg file 'GooglePhotos/- Check before delete/0 - 19-05-2558-1 - Beau-Nick-Vik/_DSC1483.jpg' 
INFO[2022-01-24T18:53:32+07:00] index: found 1 face in _DSC1486.jpg [269.613917ms] 
INFO[2022-01-24T18:53:35+07:00] index: 'GooglePhotos/- Check before delete/0 - 19-05-2558-1 - Beau-Nick-Vik/_DSC1486.jpg' has instance_id ee7ce166-9382-8647-99f8-0c14314b0495 
INFO[2022-01-24T18:53:35+07:00] media: _DSC1486.jpg was taken at 2015-05-19 14:41:46 +0000 UTC (meta) 
INFO[2022-01-24T18:53:35+07:00] index: matched 2 labels with _DSC1484.jpg [15.293798328s] 
INFO[2022-01-24T18:53:38+07:00] index: 'GooglePhotos/- Check before delete/0 - 19-05-2558-1 - Beau-Nick-Vik/_DSC1487.jpg' has instance_id 20116301-32aa-db42-8c9c-a18a6b96d68b 
INFO[2022-01-24T18:53:38+07:00] media: _DSC1487.jpg was taken at 2015-05-19 14:41:58 +0000 UTC (meta) 
INFO[2022-01-24T18:53:41+07:00] media: created 6 thumbnails for _DSC1484 [5.459616673s] 
INFO[2022-01-24T18:53:41+07:00] index: added main jpg file 'GooglePhotos/- Check before delete/0 - 19-05-2558-1 - Beau-Nick-Vik/_DSC1484.jpg' 
INFO[2022-01-24T18:53:46+07:00] media: created 6 thumbnails for _DSC1485 [7.706669014s] 
INFO[2022-01-24T18:53:46+07:00] index: added main jpg file 'GooglePhotos/- Check before delete/0 - 19-05-2558-1 - Beau-Nick-Vik/_DSC1485.jpg' 
INFO[2022-01-24T18:53:51+07:00] index: 'GooglePhotos/- Check before delete/0 - 19-05-2558-1 - Beau-Nick-Vik/_DSC1488.jpg' has instance_id 197dbae5-603e-d844-bf00-cf4f2f08f364 
INFO[2022-01-24T18:53:51+07:00] media: _DSC1488.jpg was taken at 2015-05-19 14:42:01 +0000 UTC (meta) 
INFO[2022-01-24T18:53:54+07:00] media: created 6 thumbnails for _DSC1486 [5.189432448s] 

I have an older dell r720 server with 64 GB of Ram and 32 cores (2x E5-2670 0 @ 2.60GHz).

Has AVX, but not AVX2. No GPU, but I do have a Coral TPUs (4x TPU cores).

I'm currently playing around with PhotoPrism and attempted to index 2TB (200k photos) photos today. Some JPEG some RAW. Some big, some small.

I have 5x 5400 rpm spinning disks running on zfs-z2. So not super fast IO, but faster than single disk. I do have an SSD and I'm willing to set this ssd up as a cache should it help (i assume it would)

I attempted to do an index today and noticed it was slow. There's not great debugging info on the webpage, so I ran photoprism index via command line via docker shell on the instance.

My docker is running at 400% which would indicate only roughly 4 CPUs cores are being used. Unsure if this is a docker issue or a photoprism issue. Advice on how to get this using all my CPU cores would be appreciated, if that's possible.

@devova while distributed arch is easy, problem is latency. Frigate has explored similar thing for doing AI labeling, but the latency of the HTTP call can outweigh the inference time saving.

As for 1x Coral TPU performance. They're about as fast as a GTX 1080 doing tensorflow lite, but instead of 180W worth of power they run at 2.5W.

Another improvement might be to resize the image, then run facenet and things label AI at them after they're smaller. Not sure if PhotoPrism does this already. I'd assume you don't need to run these on full sized 16MB images.

[lastzero]

Open the photo edit dialog > Files tab. At the bottom you find the indexing time and duration, for example:

Added	Jan 21, 2022, 10:01 AM in 617 ms

16MB JPEGs are huge, so not the average use case. I'd expect indexing such large files takes longer than usual.

Check our docs for how to configure the number of workers:

https://docs.photoprism.app/getting-started/config-options/#index-workers

[alexislefebvre]

I have an older dell r720 server with 64 GB of Ram and 32 cores (2x E5-2670 0 @ 2.60GHz).

Nitpicking: one E5-2670 has 8 cores / 16 threads. So with 2 processors, the server has 16 cores / 32 threads.

I think you should try to use 16 workers (see the link from Mickael above) to start, then increase. 32 workers would probably exceed the power of the CPUs.

[IssueHuntBot]

An anonymous user has cancelled funding for this issue. (Cancelled amount: $50.00) See it on IssueHunt

[kbekus]

I love Photoprism, it's excellent. Just my 2c, having a plug-in framework would be great allowing for modularizing certain components. There are already many applications out there that do this (Frigate, viseron). These NVR applications call external modules for image and face recognition. It would be great if photoprism called Deepstack for example to handle face recognition.

That would be particularly great because then any NVR that uses Deepstack for face recognition (Frigate, Blue Iris) would have access to your already recognised and curated face library.

[srett]

For that specific use case to work, Deepstack had to use the same landmarks for faces as the lib currently used by PP does, otherwise you had to externalize the clustering of faces too. I agree that if some features of PP were pluggable you could build really cool stuff, but I guess everybody has a slightly different idea of how that should work and exactly what parts of PP would be important to modularize.

[lastzero]

Once we have the time/resources to work on it, we are happy to implement support for external APIs, additional models and hardware devices. This could be introduced as a premium feature to help fund development, especially since many users will likely need support for debugging interactions between applications/APIs (as is already the case with hardware video transcoding).

[NexusXe]

Premium feature as in closed source?

While I understand the necessity to make money, I dislike the addition of closed-source modules so close to extremely personal data.

If by "premium" you mean putting a paywall in front of such features in official builds/distributed images, I wouldn't mind this too much as I could simply build it myself with whatever feature enabled.

This is sort of similar to what Aseprite does: $20 for the program (with product support), or build it yourself for free. It's really not a terrible way to do things.

[lastzero]

Since it is Open Source, you are free to donate your time to develop, maintain and support this feature. If it is not contributed/funded in some way ($20 is not enough to pay a full time developer for a week or more...), then it cannot be offered because no one is doing the work.

[j0rd]

I have an older dell r720 server with 64 GB of Ram and 32 cores (2x E5-2670 0 @ 2.60GHz).

Nitpicking: one E5-2670 has 8 cores / 16 threads. So with 2 processors, the server has 16 cores / 32 threads.

I think you should try to use 16 workers (see the link from Mickael above) to start, then increase. 32 workers would probably exceed the power of the CPUs.

I had to increase the amount of cores and memory available in docker to get photoprism to work at a reasonable speed.

I set docker to 24 threads and 42GB of ram. Still took over a day to crunch through all my photos, but this sped up the process a tonne and was my main bottleneck previously.

The label inference times would still be greatly sped up via a Coral integration though, but via the increase in resources available to docker, inference times were sped up.

[sailhobie]

Coral support would be great