Updating the lens fun database

[paperdigits]

I got a question from a user over on the pixls.us forum asking how to update the lensfun database. He says it is way out of date. I haven't checked yet, but how do we ensure that we pull the latest databse when building?

[hfiguiere]

Where is the database located? If it is part of lensfun source code, then we pull the latest release available (0.3.95), which is not the latest revision.

In that case we'd need a lensfun release, that is compatible with Darktable, of course.

[matze]

Alternatively, your user might help himself with this answer from last year.

[paperdigits]

The problem is that the latest lensfun is from Jun 29, 2018, which is pretty old. You have to run lensfun-update-data in order to fetch the latest lens profiles, which can be done independent of the source code.

[paperdigits]

Seems like we can grab http://wilson.bronger.org/lensfun-db/version_2.tar.bz2 and extract it to the lensfun database location, that'd at least give us a database as of the build date of the flatpak.

[hfiguiere]

I'll have a PR tonight with a slightly better solution (IMHO). We can't run lensfun-update-data at build time without granting network permissions (I'd rather not) but we can have everything to run the update by the maintainer and commit it to the flathub repository.

[paperdigits]

Yes, that would be delightful! I downloaded the tar file last night, and a bunch of my lenses are supported now. Thanks!

On May 5, 2020 12:42:08 PM PDT, "Hubert Figuière" notifications@github.com wrote:

I'll have a PR tonight with a slightly better solution (IMHO). We can't run lensfun-update-data at build time without granting network permissions (I'd rather not) but we can have everything to run the update by the maintainer and commit it to the flathub repository.

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/flathub/org.darktable.Darktable/issues/56#issuecomment-624266156

[eszlari]

Just a small recap from reading this all up:

• 0.3.2: db v1, can be compiled against darktable
• 0.3.95: db v2, can be compiled against darktable
• git HEAD: db v2, cannot be compiled against darktable (https://github.com/lensfun/lensfun/issues/1142)

[eszlari]

After some more investigation:

The flatpak moved from 0.3.2 to git with this commit (8 Jan 2018): https://github.com/flathub/org.darktable.Darktable/commit/04747010b2e4e219a9cc32161c69b15218c0bb88

This was unfortunately a mistake and not what upstream really supports: https://github.com/darktable-org/darktable/issues/2813

Upstream's Windows build uses 0.3.2: https://github.com/darktable-org/darktable/pull/2450 (I think the same is true for macOS)

Debian, Fedora and SUSE use 0.3.2 too: https://tracker.debian.org/pkg/lensfun https://src.fedoraproject.org/rpms/lensfun https://build.opensuse.org/package/show/openSUSE%3AFactory/lensfun

Only Arch (and therefore Manjaro) has 0.3.95: https://www.archlinux.org/packages/extra/x86_64/lensfun/

So the right thing to do would be to revert to 0.3.2 and the version_1.tar.bz2 tarball: http://wilson.bronger.org/lensfun-db/version_1.tar.bz2

Now how many people have used Darktable from Flathub since 8 Jan 2018 and saved image edits, I don't know. But overall, compatibility seems to me in favor of 0.3.2.

[eszlari]

I also opened a lensfun upstream issue about separating the database from the main repo: https://github.com/lensfun/lensfun/issues/1172

[paperdigits]

What are the ramifications of rolling back the library? I'd gather that we would mess up a bunch of people's edits, no? I don't think we should roll back, if that is the case.

On May 7, 2020 4:54:17 PM PDT, eszlari notifications@github.com wrote:

After some more investigation:

The flatpak moved from 0.3.2 to git with this commit (8 Jan 2018): https://github.com/flathub/org.darktable.Darktable/commit/04747010b2e4e219a9cc32161c69b15218c0bb88

This was unfortunately a mistake and not what upstream really supports: https://github.com/darktable-org/darktable/issues/2813

Upstream's Windows build uses 0.3.2: https://github.com/darktable-org/darktable/pull/2450 (I think the same is true for macOS)

Debian, Fedora and SUSE use 0.3.2 too: https://tracker.debian.org/pkg/lensfun https://src.fedoraproject.org/rpms/lensfun https://build.opensuse.org/package/show/openSUSE%3AFactory/lensfun

Only Arch (and therefore Manjaro) have 0.3.95: https://www.archlinux.org/packages/extra/x86_64/lensfun/

So the right thing to do would be to revert to 0.3.2 and the version_1.tar.bz2 tarball: http://wilson.bronger.org/lensfun-db/version_1.tar.bz2

Now how many people have used Darktable from Flathub since 8 Jan 2018 and saved image edits, I don't know. But overall, compatibility seems to me in favor of 0.3.2.

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/flathub/org.darktable.Darktable/issues/56#issuecomment-625554465

[junkyardsparkle]

What are the ramifications of rolling back the library? I'd gather that we would mess up a bunch of people's edits, no? I don't think we should roll back, if that is the case.

That isn't a stable release version, so their edits could end up messed up in any case, just sayin'. Probably better to deal with that now than later (if rolling back now would even cause issues, which may not be the case).

[paperdigits]

Understood, looks like its bring rolled back to 0.3.2. Thanks.

On May 8, 2020 7:46:09 PM PDT, junkyardsparkle notifications@github.com wrote:

What are the ramifications of rolling back the library? I'd gather that we would mess up a bunch of people's edits, no? I don't think we should roll back, if that is the case.

That isn't a stable release version, so their edits could end up messed up in any case, just sayin'.

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/flathub/org.darktable.Darktable/issues/56#issuecomment-626093241

[qosch]

Just wanted to add this for anyone who came across this in search for the location of the lensfun DB (e.g. to add a lens profile manually): For me (Manjaro, Flatpak) the file is located in /var/lib/flatpak/app/org.darktable.Darktable/x86_64/stable/*some_number*/files/share/lensfun

[hfiguiere]

This is absolutely discouraged.

[qosch]

How would I add a lens profile instead? What are the side effects except that an update would remove it?

[hfiguiere]

File a bug upstream (Darktable) to be able to provide profiles as a user...

Otherwise we just updated the database from lensfun. (it still building as I write this).

[paperdigits]

No please don't file a bug against darktable, this isn't a darktable issue. This is a lensfun issues. It isn't even really a lenafun issue because lensfun has a path where you can put new lenses, something like /var/lib/lensfun/ but I don't know what that path is inside the flatpak sandbox.

On December 30, 2020 8:09:46 AM PST, "Hubert Figuière" notifications@github.com wrote:

File a bug upstream (Darktable) to be able to provide profiles as a user...

Otherwise we just updated the database from lensfun. (it still building as I write this).

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/flathub/org.darktable.Darktable/issues/56#issuecomment-752675238

[hfiguiere]

No please don't file a bug against darktable, this isn't a darktable issue. This is a lensfun issues. It isn't even really a lenafun issue because lensfun has a path where you can put new lenses, something like /var/lib/lensfun/ but I don't know what that path is inside the flatpak sandbox.

If your process, to add user installable data, is to add it in a path that require root access, then this process is definitly wrong. So it's an upstream problem. Period.

[paperdigits]

I have no comment on what is deemed right or wrong, I'm just telling you how it works so we can avoid opening spurious upstream issues.

There are actually several paths that lensfun looks at to find new profiles and they look to be user adjustable at build time: https://lensfun.github.io/manual/latest/dbsearch.html

lfDatabase::UserLocation is at ~/.local/share/lensfun/updates/version_x, where x denotes the database format version (Database format versions).

lfDatabase::UserLocation is at ~/.local/share/lensfun.

On December 30, 2020 8:27:23 AM PST, "Hubert Figuière" notifications@github.com wrote:

No please don't file a bug against darktable, this isn't a darktable issue. This is a lensfun issues. It isn't even really a lenafun issue because lensfun has a path where you can put new lenses, something like /var/lib/lensfun/ but I don't know what that path is inside the flatpak sandbox.

If your process, to add user installable data, is to add it in a path that require root access, then this process is definitly wrong. So it's an upstream problem. Period.

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/flathub/org.darktable.Darktable/issues/56#issuecomment-752681068

[hfiguiere]

I have no comment on what is deemed right or wrong, I'm just telling you how it works so we can avoid opening spurious upstream issues.

So you want a quid pro quo.

Trying to get things better is also making design choices or influencing them. More often than not it is a "I didn't think of it" situation.

[paperdigits]

I don't think that qualifies as a quid pro quo, but that's beside the point.

The directories where lensfun looks for user provided profiles is configurable at build time, so why not just point one of those variables to a place where flatpak can read the files?

On December 30, 2020 8:52:33 AM PST, "Hubert Figuière" notifications@github.com wrote:

I have no comment on what is deemed right or wrong, I'm just telling you how it works so we can avoid opening spurious upstream issues.

So you want a quid pro quo.

Trying to get things better is also making design choices or influencing them. More often than not it is a "I didn't think of it" situation.

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/flathub/org.darktable.Darktable/issues/56#issuecomment-752688212

[eszlari]

If lensfun doesn't respect XDG_DATA_HOME this would be a bug to report to them. In the meantime, --persist=.local/share/lensfun could be added.