UX: Improve performance of infinite scrolling for large photo collections

[williamkray]

I discovered after importing my very large, pre-existing photo library and then trying to create some albums that very large collections will run into limitations in the browser. The first limitation is not terrible but I didn't see it documented: selecting multiple images is limited to 500. The second limitation was much more difficult to work within: I was unable to scroll past 2,222 photos in a single view (using mosaic view). This doesn't take into account the performance hit I saw while browsing and selecting, which I see is already reported in https://github.com/photoprism/photoprism/issues/477.

My entire photo library is around 15,000+ photographs and I presume without being able to move photos out of the primary view, it's not hard to get past that 2,222 image limit. Attempting to create an album for photographs from my wedding, for example (for which I have nearly 1,200 photographs alone in various color palettes and from various sources) became an exercise in finding limiting views to select the photos I wanted, sorting forward and then reverse for the month in question, etc.

Perhaps instead of a hard limit on the number of photos to load, the limit can be a rolling number, and as the limit is reached via the infinite scrolling, the previous ones are kicked out of cached memory. The process would then be reversed for scrolling in the opposite direction.

Thank you for this incredible software, both the web app and the Android mobile applications are far superior to anything else I've tried!

[lastzero]

First, let me thank you for your donation! Much appreciated.

As a quick solution, we can increase the limits to 3333 results and 999 clipboard items. Of course it is also possible to hit those limits and eventually your browser might run out of memory if you try to load all pictures. That happened to a user in the past, so we've added these limits.

Note that we provide a filter for finding pictures that have not been added to an album yet:

https://demo.photoprism.org/unsorted

You can manually apply it everywhere by entering unsorted:true in the main search field.

Implementing a mix of infinite scrolling and pagination, as you suggest, doesn't seem to be a good idea in terms of code complexity and UX. I know Flickr has/had a relatively low limit for infinite scrolling and then forces you to use a pagination. I'd rather provide other ways of dealing with large result sets, like filtering results that have already been added etc.

[williamkray]

yes, i don't think the limits should be raised as this already slows down my reasonably-spec'd computer by loading lots of pictures. i think the issue is that from a user perspective, the library is "infinite scroll" until it isn't. I thought i would be able to just keep scrolling back in time through my entire library, but i only get a fraction of the way through before i can't go any further and have to choose a different method of browsing.

this is in contrast to google photos, where my entire library is shown, it handles infinite scrolling, and there's even a timeline along the left so that i can quick-scroll to a certain date-range. if i do this scrolling too quickly, it takes a moment for the images to cache but otherwise it's a seamless user experience.

i cannot speak to code complexity regarding this, but from a UX standpoint i would say that the current behavior did not match my expectation based on the general functionality of the site, and therefore this is bad UX (this small use case, not the entire site). a site should not use infinite scroll if it cannot scroll infinitely.

EDIT: also, I've tried using the unsorted flag but a) there isn't a search field in many of the views (calendar, folders, etc) and b) applying it manually to the url with ?q=unsorted%3Atrue doesn't work, so I'm unable to use that flag to find photos I may be missing from their respective albums.

[lastzero]

Jumping to a point in a timeline is different. Time effectively is a search filter then, like you can limit results with our filters. Just a different way to visualize it. Many users asked for this, so it's definitely on the list. I personally prefer a real search form and typically don't use it.

Would a timeline also work in a situation where you have lots of photos taken in a very short time, like a wedding? Jumping to a different year is much different from what your initial use case was (selecting wedding photos probably taken on the same date).

Note that Google Search is a prominent example for what you describe as bad user experience. They use a so called "top k" index where you can only actually see the first k results even though there supposedly are millions of results 😉

[lastzero]

EDIT: also, I've tried using the unsorted flag but a) there isn't a search field in many of the views (calendar, folders, etc) and b) applying it manually to the url with ?q=unsorted%3Atrue doesn't work, so I'm unable to use that flag to find photos I may be missing from their respective albums.

You're right, we don't provide a regular search form in albums or moments as the search filter might conflict - in various unforeseen ways - with the scope of the album or moment, like when you open "May 2020" in the calendar and then request results from 2019. Technically, albums and moments (incl calendar and folders) are nothing else than static search filters. So you can filter / find the same results using our regular search if that makes sense.

You can find a list of all filters on https://docs.photoprism.org/user-guide/organize/search/

I agree that those docs need a little more work and better descriptions of what each filter does.

Edit: You especially might want to try after:2006-01-02 and before:2009-01-02. In addition, you can search for day, month and year independently. Filtering by day and month is great for finding birthday pics! Obviously, a custom timeline selector would be better for other use cases, like when you quickly want to browse your collection by time without knowing the exact date. For now, we provide the calendar for this.

[williamkray]

those additional search filters are great, thank you. i didn't seem to find them in my first pass of the documentation.

when i used google photos as an example, i wasn't referring to google search... google search is paginated, hence the famous Goooooooooogle at the bottom. Pagination is a bad UX for a photo album, i agree, and is the primary reason i'm looking for an alternative to Piwigo (which uses pagination) and have happily found Photoprism (which uses infinite scroll).

But google photos (photos.google.com) is an infinite scroll of your entire photo collection which loads on-demand, similar to Photoprism. The timeline along the side is a separate feature but loads in the same interface as the infinite scroll.

If i'm browsing my collection in google photos, I am never told "you've looked at too many photos, we can't load any more, please search or use some other interface to find the thing you're looking for." This is the differentiator i wanted to call out.

if i'm looking through my photo collection for every picture i've taken of my dog over the last 10 years, i can't just scroll through my photo collection (the default landing page of photoprism) to find them and add them to an album because i'll be told there are too many pictures to load and my browser will crash. instead i have to find a different way of browsing my photo collection:

• perhaps month by month (but what if i have too many photos in a given month?)
• search for "dog", and hope that tensorflow identified all photos perfectly (unlikely)
• a combination of technical search terms to specifically identify photos within certain date ranges that have not yet been added to an album (great for power users, but very technical and layman-unfriendly)

This flow is not intuitive because the photo library acts like i should be able to view them all (it's right in the name of the mechanism: infinite scroll, not arbitrarily limited scroll). this is at the heart of the matter: UX requires consistency and predictability by the user. if the user is presented with an interface that says "you can keep scrolling down, and we'll load more pictures as you need to see them" then it should continue to do this, instead of stopping at an arbitrary point and saying "wait, not that far down! use something else".

hopefully this clarifies my initial statement... i'm not suggesting a combination of pagination and infinite scrolling from a user-perspective, i'm suggesting consistent infinite scrolling behavior. using pagination on the backend to make that happen both forward and backward will have to be the only way to do that without requiring as much RAM as there are photos.

[williamkray]

all this is to say: the view already loads the next 60 photos, with a maximum of 2,222. it should probably be limited to 500 to reflect the maximum number you can select, and once it met that limit you just continue loading the next 60 by clearing the first 60 photos out of browser cache, the whole situation would be much smoother and put less pressure on system resources.

[lastzero]

those additional search filters are great, thank you. i didn't seem to find them in my first pass of the documentation.

Excellent!

when i used google photos as an example, i wasn't referring to google search... google search is paginated, hence the famous Goooooooooogle at the bottom

Typically it is possible to jump to the last page where there is a pagination. Not only does Google skip results, you can't even jump to the last page. Users accept this because it is Google and they are used to it. That was my point.

Pagination is a bad UX for a photo album, i agree, and is the primary reason i'm looking for an alternative to Piwigo (which uses pagination) and have happily found Photoprism (which uses infinite scroll).

Maybe we should call it dynamic scrolling instead of infinite, to dampen expectations ;)

This is what Flickr does when you dynamically scroll too far in the Photostream:

They also have a Camera Roll view with infinite scrolling, but it loads really slow.

All in all, I think we have a good enough solution for the start even though there is room to improve. But that requires more budget.

But google photos (photos.google.com) is an infinite scroll of your entire photo collection which loads on-demand, similar to Photoprism. The timeline along the side is a separate feature but loads in the same interface as the infinite scroll.

Seems like this issue is effectively another request for a Timeline view, see #152 and #502 👍

When you do a search in Google Photos, you see the results in a regular view, not the Timeline. They don't even have different result layouts (like List, Card or Mosaic View) to choose from, unless there are settings I haven't found so far?

To properly render a timeline, you need to know how many results you have and what dates they have. Applying a filter slows down performance significantly as you always need to operate on the complete result set. We intentionally don't do this in favor of performance and more metadata in search results, so that we can render our Card and List View (which saves me a lot of time as I don't need to open every picture to see camera, lens, codec, location etc).

When implementing a Timeline view without additional filters (like Google), we will choose an approach similar to Places where all results are loaded instantly, but with less metadata.

My point is: You can't have everything without any tradeoffs. For us, performance and good filters were more important than a Timeline view as part of our first release. Once there is budget, we can provide a Timeline view that is just as nice as what Google has to offer. If you think Google has the perfect product, there is also no reason to use PhotoPrism instead.

[williamkray]

There's a lot of focus on timeline view in this issue, and it's making me regret having brought it up as a side-note because there are separate issues which you've already linked to.

Google doesn't have a perfect product because I am not in control of any of my data by using their service. I have been searching for a functional self-hosted replacement of their photo service for about three years now, migrating my photos constantly from Sigal to Piwigo and now to Photoprism, with countless trials of other products in between. Photoprism has been by far the best.

The only issue I've had with it is that it doesn't clear browser cache of thumbnails when infinite-scrolling in any of the views. I still stand by my recommendations that:

• the number of photos you can "select" should be the limiting factor on how many thumbnails can be loaded
• when loading the next set of N photos into browser cache, the first N should be removed from browser cache
• this process is handled invisibly by the user, making it appear that infinite scroll is in fact infinite in either direction when browsing their collection
• doing this reduces resource consumption on the web client and eliminates arbitrary limitations on scrolling through the photostream
• this issue does not recommend any additional changes to filtering, search criteria, or timelines. any other suggestions are reserved for other issue numbers.

I can only hope that once this issue gets raised to it's eventual priority, this is a change that budget allows for. I will do what I can to help sustain the project.

[lastzero]

I was going to also give more feedback regarding your specific implementation ideas, but didn't have the time so far and wasn't sure if you're interested after all I wrote already...

JavaScript typically doesn't have control over the browser cache, although there is now a CacheStorage API which I haven't used before.

Even if this works, micro-managing the browser cache is often not a good idea:

• On a desktop computer, you typically have lots of RAM, disk storage, and a fast internet flatrate. Difficult to find the best trade-off.
• On a mobile phone, you typically have high latency and pay for traffic on top of it. Also difficult to find the best trade-off, especially since there are major performance differences between smartphone generations.
• Then some users might only use PhotoPrism in their home network, which makes it more feasible to offload caching to the server - unless it's a slow Raspberry Pi. Again, difficult choice that requires lots of testing.

I currently don't have time to upload 100k files to Google Photos for testing, so I don't want to make a statement how exactly they handle this - but I wouldn't be surprised if they only do lazy loading when you scroll to a position but don't remove anything that was already loaded. You typically don't want to do this as scrolling back and forth would cause a lot of traffic and bad UX by itself.

Also keep in mind:

• Google serves preview images with much higher compression and less detail, so that by itself can be a reason it is faster. If best performance / smallest browser cache is a major concern for you, you can already reduce thumbnail quality in our config without waiting for new features.
• Google seems to add less metadata to their (search) results so they can render faster (as their views are less complex) and can return more results with the same response size. You need to be careful not to compare apples with oranges.

Bottom line: Loading too many files is bad, but loading them repeatedly over a network without caching seems worse. What we can and should do is a) optimize our JS so that rendering gets faster and b) add additional views, like a Timeline, for users who like to browse their library as known from Google Photos and other photo apps (see #152 and #502).

[williamkray]

I wouldn't be surprised if they only do lazy loading when you scroll to a position but don't remove anything that was already loaded.

I have verified in my google photos account that the interface reloads thumbnails that have already been loaded when i revisit them, because they have been purged from local memory. This is only with a significantly smaller subset of photos that happen to be in my account, probably only a thousand or so.

On a mobile phone

This sounds like something that is a reasonable thing to consider "best effort" to accommodate, but you also have a mobile application which handles caching differently. Users concerned with data usage would best leverage the mobile app to cache photos locally rather than load them over the internet. I don't think this is unreasonable, and most people conscious of their mobile data usage already do things like this regularly, choosing native caching applications over webapps. On top of that, if a user is concerned with their data usage, they probably should use your alternative suggestions to find the photos they're looking for rather than load every image in their photostream trying to look for it, and then doubling back over photos that they've already passed.

you can already reduce thumbnail quality in our config

I will definitely investigate these settings to see if it improves my local browser performance, but thumbnail size would then have even less to do with the 2,222 image limit.

I recognize the difficulty in identifying a good baseline for performance related issues, and my recommendation would be to always start with a very limited constraint window, and allow for higher-performance options secondary. It's the reason that web development has a "mobile-first" design mentality for responsive display sizes, and tools like https://github.com/tylertreat/comcast exist. I'm not you, so I cannot make those decisions for you, but I am a technical consultant by trade so I'm automatically opinionated and want to tell other people how to do things :smile:

[prog32]

First of all - GOOD JOB! I'm impressed!

@williamkray Suggest that problem is in showing a large number of photos.

I just discovered that project (at night) and immediately noticed, that it's hard to get pictures far far away.

It requires scrolling down, waiting to load, scrolling down, waiting,... It's time-consuming. And I'm expecting to have memory problems, by the way it is done.

I know there is a calendar tab, we can limit dates in the search box. But sometimes I'd like to just scroll and see what I have (I don't know the month, year, name in advance).

So being able to scroll like in google photos will be desired. Using a calendar or filtering for just scrolling is bad UX. It also doesn't solve memory issues.

Solution?

I believe it can be solved by creating a virtual list.

Pagination is already implemented on the backend. The only missing thing on the backend is the 'count of items'. But I believe that "Select Count(*) WHERE..." should not be that expensive.

When we will have: 1) pagination (implemented, API available) 1) count (probably missing, I didn't check codebase)

It will be possible to deliver google photo's like scrolling.

Knowing count, we will now how many rows we need to display. number_of_rows * row_height gives us the height of the whole list.

Then we can load only pictures which are in view (with some margin). Photos out of view (with some margin) will be removed from memory (no need to store DOM elements).

In that way viewing thousands of pictures should be possible.

For example when we will move the scroll bar to the middle. Applications may request just: http://localhost:2342/api/v1/photos?count=60&offset=10000&...

instead of every page to reach offset 10000.

A simple example of such a scroller with 200000+ elements! (JS/React) https://codesandbox.io/s/react-virtual-scrolling-basics-u1svg?file=/src/index.js

Please check out: (description how virtual scrolling works) https://blog.logrocket.com/virtual-scrolling-core-principles-and-basic-implementation-in-react/

It seems that you use VUE. Probably you can use vue-virtual-scroll-list (Just first library from google) https://tangbc.github.io/vue-virtual-scroll-list/#/page-mode https://github.com/tangbc/vue-virtual-scroll-list#live-demo

Currently, I don't know even a basis of VUE, nor Go :/ I'm more into Python/Django & Rect on the frontend. So probbably I wouldn't be able to provide a pull request in a reasonable time.

Using virtual lists can solve several issues.

API returns just array of items. To avoid adding count to every API page request, maybe it can be done on another request. So it will initially load just 60 pictures as usual, but then it will fetch the real number of photos. So scrollbar can be adjusted (for virtual list).

Use of "vue-virtual-scroll-list" (or other library) will still require some work like fetching image details on demand.

Note on Google photos timeline

Note, that even Google photos don't show a timeline on the side for example when you search for some term. It just allows you to scroll. - The thing which can be done with a virtual list (when we will have "count")

I main view (view of all photos) it shows a timeline where you can see "what date you can expect on given scroll position". It's useful but probably to heavy to generated it on request. I think they just store in DB, how many pictures we have in a given month/year. Having it we can compute the percent of photos in a given month/year to all photos, so it is possible to show dates next to the scroll bar. Example: 2020: 40% of all photos 2019: 30% of all photos 2018: 20% of all photos 2017: 10% of all photos Then on top we can show 2020, on 40% of height we can show 2019, on (40%+30%) of height we can show 2018, ... But it will be 'nice to have', but virtual list (memory issues) + scrolling (to any point) will be enough :) to handle large collections.

[ryancc9]

I'd love a timeline view also, but my most desired feature is smoother/better infinite scrolling. I found this article about google photos very helpful, particularly "4. Instantaneous Feel" : https://medium.com/google-design/google-photos-45b714dfbed1 If the UX was loading the next batch of photos as I was scrolling down(not waiting until I hit the very bottom), I think that would make the experience 10x better. As it is now, I have to scroll fully to the bottom, wait 100ms for it to load more, then start scrolling again.

All the lower quality preload isn't even necessary. Just preload 60 more photos when I've scrolled down 20 out of 60, instead of waiting until I'm at 60/60.

[lastzero]

So now that JS and thumbnail API performance was improved by a lot, browsers limit the number of connections thus loading images delayed - sometimes up to 20 seconds after opening a page (or more). Solutions may be to use multiple sub-domains / hostnames or try HTTP2.

[benjaminknauer]

Please consider using content-visibility: auto to increase performance when many images are displayed:

https://web.dev/content-visibility/

[StephenBrown2]

No support in Firefox, yet: https://developer.mozilla.org/en-US/docs/Web/CSS/content-visibility

[benjaminknauer]

Right, but can be implemented with almost no effort and a possibly strong performance improvement for Chrome users.

Our company sees large benifits in performance via web vitals.

https://youtu.be/FFA-v-CIxJQ

[lastzero]

@benmccann @benjaminknauer Just started a new preview build containing this optimization:

#photoprism .search-results .result {
    content-visibility: auto;
    contain: layout style paint;
}

While it makes sense for the file browser which shows up to 999 results at once, I'm not convinced it helps the other views which already use lazy loading. Feels shaky especially on mobile, e.g. because the top toolbar fades in and out even when just scrolling down.

Feedback welcome!

[lastzero]

Had to remove content-visibility as it's not compatible with our auto-hiding top toolbar. Fades in and out all the time.