No content is displayed at all

johannesjh commented 6 months ago

Hi, I wanted to give fotema a try... I like its design and I like that it does not move photos into a closed, app-specific library like some other photo managers do. Unfortunately, I could not get it to work - let me explain. Maybe someone can help me get started?

I am running ubuntu 24.04 with GNOME 46. I installed fotema 1.7.1 from flathub.

When I start the fotema app, its main windows is shown.
A blue headerbar pops up for a short time, but quickly disappears. Something about photos and videos being scanned.
But no photos or videos are shown.

I then started fotema from the terminal in order to see its logs, using the command from #1, i.e., flatpak run --env=RUST_LOG=fotema=DEBUG,glycin=DEBUG --env=RUST_BACKTRACE=1 app.fotema.Fotema. Still no content shown. Not much info in the logs; fotema wrote the following output:

➜  ~ flatpak run --env=RUST_LOG=fotema=DEBUG,glycin=DEBUG --env=RUST_BACKTRACE=1 app.fotema.Fotema
20:57:21.833  INFO fotema::languages: Requested languages: [LanguageIdentifier { language: Language(Some("en")), script: None, region: Some(Region("US")), variants: None }]

(fotema:2): Adwaita-WARNING **: 20:57:21.996: AdwNavigationPage 0x60d9f03dd990 is missing a title. To hide a header bar title, consider using AdwHeaderBar:show-title instead.

(fotema:2): Adwaita-WARNING **: 20:57:21.996: AdwNavigationPage 0x60d9f02d69b0 is missing a title. To hide a header bar title, consider using AdwHeaderBar:show-title instead.
20:57:21.996  INFO fotema::app::background::bootstrap: bootstrap: start
20:57:21.996  INFO fotema::app::background::photo_scan: Scanning file system for pictures...
20:57:21.999  INFO fotema::app::background::bootstrap: bootstrap: scan photos started
20:57:22.660  INFO fotema::app::background::bootstrap: bootstrap: scan photos completed
20:57:22.660  INFO fotema::app::background::video_scan: Scanning file system for videos...
20:57:22.660  INFO fotema::app::background::bootstrap: bootstrap: scan videos started
20:57:22.911  INFO fotema::app::background::bootstrap: bootstrap: scan videos completed
20:57:22.911  INFO fotema::app::background::photo_enrich: Enriching photos...
20:57:22.912  INFO fotema::app::background::bootstrap: bootstrap: photo enrichment completed
20:57:22.912  INFO fotema::app::background::video_enrich: Enriching videos...
20:57:22.912  INFO fotema::app::background::bootstrap: bootstrap: video enrichment completed
20:57:22.912  INFO fotema::app::background::photo_thumbnail: Generating photo thumbnails...
20:57:22.913  INFO fotema::app::background::bootstrap: bootstrap: photo thumbnails completed
20:57:22.913  INFO fotema::app::background::video_thumbnail: Generating video thumbnails...
20:57:22.913  INFO fotema::app::background::bootstrap: bootstrap: video thumbnails completed in Some(916.463467ms)
20:57:22.913  INFO fotema::app::background::photo_clean: Cleaning photos...
20:57:22.913  INFO fotema::app::background::bootstrap: bootstrap: video cleanup completed.
20:57:22.913  INFO fotema::app: Bootstrap completed.

(fotema:2): Adwaita-WARNING **: 20:57:38.686: AdwNavigationPage 0x60d9f04ae5d0 is missing a title. To hide a header bar title, consider using AdwHeaderBar:show-title instead.

My $HOME/Pictures folder contains about 138.000 files, it is about 114gb in size. Yet fotema seems to quickly scan the folder without finding any content to display.

Any ideas how I could analyze this? What could be the problem? Thanks!

blissd commented 6 months ago

Hi @johannesjh. That is very curious. If all the start up process is very quick and finds no photos or videos the I suspect that your XDG_PICTURES_DIR does not point at where you actually store your pictures. To check where Fotema will look for pictures and videos you run the following command in a terminal:

xdg-user-dir PICTURES

blissd commented 6 months ago

For the next release of Fotema I'll add a log line of the XDG_PICTURES_DIR so you can see what directory is being scanned.

johannesjh commented 6 months ago

Thank you, I also thought that fotema somehow did not "see" my files, maybe due to a wrong path. But my XDG_PICTURES_DIR looks normal and correct:

~ xdg-user-dir PICTURES
/home/jh/Pictures

/home/jh is my home directory, and /home/jh/Pictures is where I keep all my photos.

johannesjh commented 6 months ago

I then thought about missing flatpak permissions. But the permissions look normal to me. This is a screenshot of flatseal showing fotema's filesystem permissions:

...all default values, I could not see anything surprising here.

blissd commented 6 months ago

Can you see your pictures directory from inside the running Fotema flatpak? You can enter the flatpak with commands like the following:

$ flatpak ps|grep Fotema
377141664   243829  app.fotema.Fotema   org.gnome.Platform
$ flatpak enter 377141664 sh

You can then see the flatpak's view of your home directory and see if your pictures is present and populated with pictures.

johannesjh commented 6 months ago

I appreciate your help a lot, thank you! It appears to me that the photos are accessible from within the flatpak container:

➜  ~ flatpak ps|grep Fotema
3058908295  118009  app.fotema.Fotema   org.gnome.Platform
➜  ~ flatpak enter 3058908295 sh
sh-5.2$ ls
Pictures
sh-5.2$ cd Pictures/
sh-5.2$ ls
 Camera   Photos   Screenshots   Webcam
sh-5.2$ cd Camera/
sh-5.2$ ls -la
total 3320
drwxr-xr-x 2 jh jh   4096 May  6 17:16  .
drwxr-xr-x 6 jh jh  45056 May  6 18:47  ..
-rw-r--r-- 1 jh jh 192669 May  6 16:58 'Photo from 2024-05-06 16-58-29.668209.jpeg'
-rw-r--r-- 1 jh jh 194063 May  6 16:59 'Photo from 2024-05-06 16-59-01.852894.jpeg'
-rw-r--r-- 1 jh jh 184501 May  6 16:59 'Photo from 2024-05-06 16-59-44.634359.jpeg'
-rw-r--r-- 1 jh jh 140572 May  6 17:04 'Photo from 2024-05-06 17-04-44.504040.jpeg'
-rw-r--r-- 1 jh jh 146462 May  6 17:04 'Photo from 2024-05-06 17-04-48.192220.jpeg'
-rw-r--r-- 1 jh jh 160944 May  6 17:05 'Photo from 2024-05-06 17-05-42.128025.jpeg'
-rw-r--r-- 1 jh jh 159034 May  6 17:06 'Photo from 2024-05-06 17-06-38.491746.jpeg'
-rw-r--r-- 1 jh jh 188895 May  6 17:06 'Photo from 2024-05-06 17-06-47.780696.jpeg'
-rw-r--r-- 1 jh jh 162833 May  6 17:07 'Photo from 2024-05-06 17-07-54.419783.jpeg'
-rw-r--r-- 1 jh jh 202846 May  6 17:08 'Photo from 2024-05-06 17-08-05.519782.jpeg'
-rw-r--r-- 1 jh jh 165605 May  6 17:09 'Photo from 2024-05-06 17-09-13.240409.jpeg'
-rw-r--r-- 1 jh jh 129647 May  6 17:14 'Photo from 2024-05-06 17-14-09.932030.jpeg'
-rw-r--r-- 1 jh jh 186790 May  6 17:14 'Photo from 2024-05-06 17-14-19.109119.jpeg'
-rw-r--r-- 1 jh jh 197059 May  6 17:14 'Photo from 2024-05-06 17-14-33.028175.jpeg'
-rw-r--r-- 1 jh jh 189169 May  6 17:14 'Photo from 2024-05-06 17-14-47.992227.jpeg'
-rw-r--r-- 1 jh jh 200797 May  6 17:15 'Photo from 2024-05-06 17-15-00.480293.jpeg'
-rw-r--r-- 1 jh jh 150215 May  6 17:16 'Photo from 2024-05-06 17-16-12.572117.jpeg'
-rw-r--r-- 1 jh jh 200951 May  6 17:16 'Photo from 2024-05-06 17-16-36.296525.jpeg'

Nonetheless, fotema shows no folders or photos:

Screenshot from 2024-06-01 21-03-30

Screenshot from 2024-06-01 21-03-33

How do you usually run and debug the app?, e.g, do you use GNOME Builder? (I am new to rust, but I could try to check out and run a branch with debugging statements etc), thanks!

blissd commented 6 months ago

I mostly run from GNOME builder. But I also produce builds using just devel and just release depending on what I want to test.

blissd commented 6 months ago

I am baffled. I've installed Ubuntu in a VM, setup Flatpak and Flathub, installed Fotema, added some test pictures, and it worked out of the box.

One difference between our setups is that my picture library is much smaller, at 13,000-ish pictures and videos and totally about 25GB. However, if the size of the library was the problem then I'd expect to see Fotema do some work and then failing with a noisy error.

johannesjh commented 6 months ago

I tested the hypothesis that the (very large) library size could be the problem:

I changed the filesystem path in flatseal from xdg-picures:ro to /home/jh/Pictures/Webcam ...this folder only contains three pictures.
I started fotema and I entered its flatpak container. I could browse to Pictures/Webcam and list the three jpg images from within the container's commandline
But fotema still did not display any content

...this makes me think that the library size rather was not the problem.

blissd commented 6 months ago

If you want to try testing again from scratch, you can delete ~/.var/app/app.fotema.Fotema.

If you want to see what data Fotema has stored in its database, then the SQLite file is at ~/.var/app/app.fotema.Fotema/data/app.fotema/Fotema/pictures.sqlite.

johannesjh commented 6 months ago

The database is empty. I verified this in two ways:

First by adding a debug statement: I cloned the project using GNOME Builder. I can now run and debug fotema. I can now see a lot more output. For example, I stumbled upon this log output: fotema::app::background::photo_scan: Found 0 photos to add to database. This made me wonder if fotema (for whatever reason) the database stays empty? => To verify if the database stays empty, I added a debug statement in repo.rs::Repository::all to print the number of photos in the database using this command info!("Found {} photos in the database", result.len()); => This line was executed twice when I started fotema. The output always read "Found 0 photos in the database".
I then started again from scratch: I deleted fotema's folder, i.e., rm -rf ~/.var/app/app.fotema.Fotema.Devel. Then I started and closed fotema using GNOME Builder.
I then opened ~/.var/app/app.fotema.Fotema.Devel/data/app.fotema.Fotema.Devel/pictures.sqlite in the sqlitebrowser app. I found that the database schema was present but the pictures table was empty.

Thank you for your debugging suggestions, this is very useful! Next steps: I guess I need to dive deeper into the scanning process.

johannesjh commented 6 months ago

As a way of saying thank you, I can contribute a summary of the tips and tricks that we've been using so far to debug Fotema. Feel free to add it to the README.

## Debugging

To run Fotema with additional log output:

```shell
flatpak run --env=RUST_LOG=fotema=DEBUG,glycin=DEBUG --env=RUST_BACKTRACE=1 app.fotema.Fotema
```

To open a shell within Fotema's flatpak container:
```shell
$ flatpak run app.fotema.Fotema
$ flatpak ps|grep Fotema
377141664   243829  app.fotema.Fotema   org.gnome.Platform
$ flatpak enter 377141664 sh
```

To view and edit Fotema's flatpak permissions, you can use the [flatseal](https://flathub.org/apps/com.github.tchx84.Flatseal) app.

To delete all of Fotema's configuration and data (e.g., to start testing from scratch): Close Fotema. Delete the following folders. Then start Fotema again.
```txt
~/.var/app/app.fotema.Fotema
~/.var/app/app.fotema.Fotema.Devel
```

To see what data Fotema has stored in its database:
The SQLite file is at `~/.var/app/app.fotema.Fotema/data/app.fotema/Fotema/pictures.sqlite`. You can use [sqlitebrowser](https://flathub.org/apps/org.sqlitebrowser.sqlitebrowser) to view the database contents.

For interactive step-through debugging:
You can use GNOME Builder to run and debug Fotema.

blissd commented 6 months ago

The code for walking the file system to find pictures is some of the first code I wrote for Fotema, so it doesn't have the most graceful error handling. I see that it swallows any errors it has while either walking the file system or reading file metadata. If something in your setup causes Fotema to fail on accessing every picture, then it will result in no errors logged and no pictures found.

blissd commented 6 months ago

I've added some error logging to the photo and video scanning process. If you have the patience to investigate further, then please pull the latest commits to the main branch and try running again.

johannesjh commented 6 months ago

Thank you, the additional logging now shows what is going on. I get tons of error messages, each with a stacktrace, each with the same error message "ERROR update_with_view: fotema_core::photo::scanner: Failed scanning: creation time is not available for the filesystem".

log.txt

Now the question is of course, why does it fail to retrieve the creation time from the filesystem? Strangely enough, many if not all of my files don't have a file creation date. Even if I look at them using GNOME files, the "Created" attribute is empty for whatever reason that may be.

Running stat on the same file shows no creation date either. I ran the stat command directly on my host system, not from inside a flatpak.

$ stat 2022-02-23-172956.jpg 
  File: 2022-02-23-172956.jpg
  Size: 126134      Blocks: 264        IO Block: 4096   regular file
Device: 0,79    Inode: 19544996    Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/      jh)   Gid: ( 1000/      jh)
Access: 2024-06-03 21:44:19.977210312 +0200
Modify: 2022-02-23 17:29:56.502902359 +0100
Change: 2022-02-23 17:29:56.502902359 +0100
 Birth: -

I am running ubuntu linux with an ext4 filesystem. The home directory is encrypted using ecryptfs, that is "special" on my system, otherwise everything is pretty standard. So yeah, for whatever reason that may be, my files don't seem to have creation dates, and that seems to be causing the problem.

EDIT, related infos from a quick google search:

ext4 does not directly expose file creation dates?, is that plausible?, compare https://forums.linuxmint.com/viewtopic.php?t=343387
workarounds exit, see e.g., https://moiseevigor.github.io/software/2015/01/30/get-file-creation-time-on-linux-with-ext4/

blissd commented 6 months ago

Thank you for identifying the problem! That is very annoying, though, as I have made a bad assumption that Linux file systems always have a file creation timestamp. I'm using BTRFS so I didn't see that my assumption was incorrect.

I'll see what I can do to fix the problem. Probably I'll use the modification timestamp as a fallback for the creation timestamp when ordering items in the photo library.

blissd commented 6 months ago

I'm working on a fix and have pushed some changes to the creation_timestamp branch. If you have the time and patience, please have a go running the branch through GNOME Builder and seeing if any files appear.

blissd commented 6 months ago

Fix is in Fotema v1.8.1 which is now available on Flathub.

johannesjh commented 6 months ago

Thank you! This fixed the issue. The fotos are now shown.

blissd / fotema

No content is displayed at all #52