search

darktable-org / darktable

darktable is an open source photography workflow application and raw developer
https://www.darktable.org
GNU General Public License v3.0
9.81k stars 1.14k forks source link

[3.2.1] Lighttable: All thumbnails and zoomed images are rendered blank white if built with LTO #6001

Closed kanyck closed 3 years ago

kanyck commented 4 years ago

After upgrading on 3.2.1 all thumbnails got blank white. W or Ctrl-W are generating white squares, too. While when enering darkroom the picture is there.

To Reproduce

  1. Start dt
  2. See white square instead of thumbnails

Platform (please complete the following information):

maboleth commented 4 years ago

Have you tried to delete your cache dir and see if newly generated thumbnails are ok?

kanyck commented 4 years ago

The ~/.cache/darktable dir is absent. I ran darktable-generate-cache, it worked out for some time, but the dir hasn't appeared. Subsequent runs of darktable-generate-cache went very fast, yet with no result either. It seems that dt created entries in the database but no files on disk. An that doesn't explain why full preview is blank, too.

johnny-bit commented 4 years ago

Does image thumbnail update after entering darkroom or is it still pure white? What camera do you have and does issue present itself when using test image with same camera from raw.pixls.us?

kanyck commented 4 years ago

It is still pure white. No matter if I enter/exit darkroom, change thumbnail size, etc. Of course I fiddled with it before opening an issue. Nikon D7200. I can check that, yet I'm pretty sure about the result)) I've been using dt since v0.6 IIRC and have thousands of pics in the database from 3 cams I had since then. It's the first time dt behaves this way, it's obviously a regression in 3.2.1.

johnny-bit commented 4 years ago

I've checked both 3.2.1. and current master with reference D7200 RAWs from raw.pixls.us and couldn't reproduce your issue, which to me means the issue might be present on your system.

I too have gentoo os but no amdgpu.

it's also weird that you don't have .cache/darktable - have you played with XDG Cache dir setting?

can you try running darktable with clean darktable config, library and data (basically new .config/darktable dir), seeing if that changes anything for display, and if it doesn't - can you post results of darktable -d opencl when displaying lighttable, darktable -d lighttable, or even darktable -d all ?

kanyck commented 4 years ago

it's also weird that you don't have .cache/darktable - have you played with XDG Cache dir setting?

Lately -- definitely I did not...

Okay. I found the mipmaps. They're situated in /tmp/kanyck/.cache/darktable for some reason. There are lots of thumbnails there and they are all blank.

kanyck commented 4 years ago

Also I checkeddarktable -d opencl and darktable -d lighttable and found nothing unusual. dt works with the thumbs normally, except they're generated blank.

johnny-bit commented 4 years ago

That looks like you've indeed played with XDG cache dir ;)

anyway - try the things I mentioned (maybe even -d cache), we need to see where the problem comes from.

wt., 18 sie 2020 o 12:03 kanyck notifications@github.com napisał(a):

it's also weird that you don't have .cache/darktable - have you played with XDG Cache dir setting?

Lately -- definitely I did not...

Okay. I found the mipmaps. They're situated in /tmp/kanyck/.cache/darktable for some reason. There are lots of thumbnails there and they are all blank.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/darktable-org/darktable/issues/6001#issuecomment-675386759, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRRKFOWK3WSGUDKRGSYHTDSBJGVTANCNFSM4QBOY2VA .

-- Pozdrawiam, Hubert Kowalski

kanyck commented 4 years ago

You've puzzled me... There is XDG_CACHE_HOME=/tmp/kanyck/.cache set in env, indeed. But I've no idea what it comes from... Nothing in /etc/env.d or ~/.*. /etc/environment is empty. And I can't recall I ever fiddled with it. Moreover, I've ~/.cache mounted as tmpfs, which appears to be pointless in this case))) Miracle. UPD: Nevermind. There is /etc/profile.d/0~xdg_cache_home.sh dated 2017. Doesn't belong to any package..

However. I asked myself how our systems might be different. Mine is built with LTO. I disabled LTO for dt and voila! Thumbs and full view are back! So 3.2.1 has a regression that is triggered by LTO.

johnny-bit commented 4 years ago

LTO is a common source of problems and darktable is very much a platform of linked libraries which may very well not play nice with LTO. You might want to raise a bug with gentoo to not allow building dt with LTO flag :)

Interestingly enough the build LTO option is in rawspeed options and not general for darktable, so you might want to raise bug against rawspeed.

also - please consider closing this bug since the issue might be outside of the scope of darktable.

wt., 18 sie 2020 o 12:39 kanyck notifications@github.com napisał(a):

You've puzzled me... There is XDG_CACHE_HOME=/tmp/kanyck/.cache set in env, indeed. But I've no idea what it comes from... Nothing in /etc/env.d or ~/.*. /etc/environment is empty. And I can't recall I ever fiddled with it. Moreover, I've ~/.cache mounted as tmpfs, which appears to be pointless in this case))) Miracle.

However. I asked myself how our systems might be different. Mine is built with LTO. I disabled LTO for dr and voila! Thumbs and full view are back! So 3.2.1 has a regression that is triggered by LTO.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/darktable-org/darktable/issues/6001#issuecomment-675403145, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRRKFIYRV5CJ53B5GRWXTLSBJK6TANCNFSM4QBOY2VA .

-- Pozdrawiam, Hubert Kowalski

parafin commented 4 years ago

Please be more specific about LTO - how exactly do you enable and disable it? Because there is a cmake option which affects only rawspeed, but you can also use custom cflags, which Gentoo makes use of. So which is it?

parafin commented 4 years ago

Also which compiler are you using? If you build with emerge, then just posting emerge --info output would be best, otherwise output of cmake configuration should suffice. I think we need to learn more about this bug before considering closing it. It may be a bug in the darktable code which is triggered by optimisations.

kanyck commented 4 years ago

@johnny-bit LTO used to be a common source of problem in gcc 4.7 ages. Now in 2020 it's a well matured technology. Please consider looking at https://cloveros.ga -- the whole distro is built with LTO (and other riser's stuff BTW))) as well as https://github.com/InBetweenNames/gentooLTO -- the progect aimed at making LTO standard for Gentoo builds. So if an app fails with LTO (as well as different compiler versions, gcc <-> clang switch,-O3, etc.) -- that means there is a bug in the code, most probably some of undefined behavior type. Today the probablity that the bug is in gcc/linker is negligible and is probably the last option to check.

@parafin I use lto-overlay from gentooLTO project mentioned above to set up cflags in conjunction with standard Gentoo ebuild. Until 3.2.1 this approach works for dt just fine -- I only had to turn -fipa-pta off. Now I'm still on gcc 9.3.0-r1, so I postponed ltoize upgrade that demands gcc 10.

$ emerge --info darktable
Portage 2.3.103 (python 3.7.8-final-0, default/linux/amd64/17.1/desktop, gcc-9.3.0, glibc-2.31-r6, 5.6.0-pf6 x86_64)
=================================================================
                         System Settings
=================================================================
System uname: Linux-5.6.0-pf6-x86_64-AMD_Athlon-tm-_X4_760K_Quad_Core_Processor-with-gentoo-2.6
KiB Mem:     8173368 total,   5224704 free
KiB Swap:   12582904 total,  12582904 free
Timestamp of repository poly-c: Sat, 15 Aug 2020 11:05:26 +0000
Timestamp of repository gentoo: Mon, 17 Aug 2020 07:05:44 +0000
Head commit of repository gentoo: 60516daaa61d970655050ebae80db6a53d9227c6

sh bash 5.0_p17
ld GNU ld (Gentoo 2.33.1 p2) 2.33.1
distcc 3.3.3 x86_64-pc-linux-gnu [disabled]
ccache version 3.7.10 [disabled]
app-shells/bash:          5.0_p17::gentoo
dev-java/java-config:     2.3.1::gentoo
dev-lang/perl:            5.30.3::gentoo
dev-lang/python:          2.7.18-r100::lto-overlay, 3.7.8-r2::gentoo, 3.8.5::gentoo
dev-util/ccache:          3.7.10::gentoo
dev-util/cmake:           3.16.5::gentoo
sys-apps/baselayout:      2.6-r1::gentoo
sys-apps/openrc:          0.42.1::gentoo
sys-apps/sandbox:         2.18::gentoo
sys-devel/autoconf:       2.13-r1::gentoo, 2.69-r4::gentoo
sys-devel/automake:       1.16.1-r1::gentoo
sys-devel/binutils:       2.33.1-r1::gentoo
sys-devel/gcc:            9.3.0-r1::gentoo
sys-devel/gcc-config:     2.3.1::gentoo
sys-devel/libtool:        2.4.6-r6::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 5.6::gentoo (virtual/os-headers)
sys-libs/glibc:           2.31-r6::gentoo
Repositories:

personal
    location: /usr/local/portage/localovr
    masters: gentoo
    priority: 0

calculate
    location: /var/lib/layman/calculate
    masters: gentoo
    priority: 50

compiz-reloaded
    location: /var/lib/layman/compiz-reloaded
    masters: gentoo
    priority: 50

jorgicio
    location: /var/lib/layman/jorgicio
    masters: gentoo
    priority: 50

jps-gentoo
    location: /var/lib/layman/jps-gentoo
    masters: gentoo
    priority: 50

lmiphay
    location: /var/lib/layman/lmiphay
    masters: gentoo
    priority: 50

lto-overlay
    location: /var/lib/layman/lto-overlay
    masters: gentoo mv
    priority: 50

mv
    location: /var/lib/layman/mv
    masters: gentoo
    priority: 50

poly-c
    location: /var/lib/layman/poly-c
    masters: gentoo
    priority: 50

stuff
    location: /var/lib/layman/stuff
    masters: gentoo
    priority: 50

gentoo
    location: /usr/portage
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/gentoo.git
    sync-user: portage:portage
    priority: 1000

Installed sets: @custom
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O3 -fno-semantic-interposition -fipa-pta -flto=4 -fno-plt -fgraphite-identity -floop-nest-optimize -fuse-linker-plugin -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -O3 -fno-semantic-interposition -fipa-pta -flto=4 -fno-plt -fgraphite-identity -floop-nest-optimize -fuse-linker-plugin -pipe"
DISTDIR="/var/gentoo/distfiles"
EMERGE_DEFAULT_OPTS=" --jobs=2 --load-average=3.5"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs clean-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="ru_RU.UTF-8"
LC_ALL=""
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en ru ru_RU"
MAKEOPTS="-j3 -l3.5"
PKGDIR="/var/cache/binpkgs"
PORTAGE_BINHOST="http://packages.gentooexperimental.org/packages/amd64-stable"
PORTAGE_BUNZIP2_COMMAND="pbzip2 -p2 -d"
PORTAGE_BZIP2_COMMAND="pbzip2 -p2"
PORTAGE_COMPRESS="pbzip2"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="X a52 aac acpi alsa amd64 bash-completion berkdb bluray branding bzip2 cairo cdda cddb cdparanoia cdr cli colord crypt dbus directfb djvu dri dts dv dvd dvdr elogind emboss encode exif faac ffmpeg flac fontconfig gdbm gif gimp git gnome-keyring gpm gsm gstreamer gtk gtk2 iconv icu infinality java jpeg lame lcms libglvnd libnotify libsamplerate libtirpc lto lz4 lzma lzo mad matroska mercurial mng mp3 mp4 mpeg multilib musepack ncurses nls nptl nsplugin ogg opencl opengl openmp osc pam pango pch pcre pdf png policykit ppds python qt5 readline samba sdl seccomp session sound speex spell split-usr sqlite ssl startup-notification subversion svg syslog tcpd theora tiff truetype udev udisks unicode upower usb v4l2 vaapi vorbis wavpack wmf wxwidgets x264 xcb xcomposite xinerama xml xv xvid zlib" ABI_X86="64" ADA_TARGET="gnat_2018" ALSA_CARDS="hda-intel" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx fma3 fma4 mabm mcx16 mlzcnt msahf mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 xop" ELIBC="glibc" INPUT_DEVICES="evdev" KERNEL="linux" L10N="en ru ru-RU" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2 php7-3" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_7" PYTHON_TARGETS="python2_7 python3_7" QEMU_SOFTMMU_TARGETS="i386 x86_64" QEMU_USER_TARGETS="i386 x86_64" RUBY_TARGETS="ruby25" USERLAND="GNU" VIDEO_CARDS="amdgpu radeonsi" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

=================================================================
                        Package Settings
=================================================================

media-gfx/darktable-3.2.1::gentoo was built with the following:
USE="colord gnome-keyring gphoto2 lto nls opencl openexr openmp -cups -doc -flickr -geolocation -graphicsmagick -jpeg2k -kwallet -tools -webp" ABI_X86="(64)" CPU_FLAGS_X86="sse3" L10N="ru -de -es -fr -he -it -pl -pt-BR -sl"
CFLAGS="-march=native -O3 -fno-semantic-interposition -fno-plt -fgraphite-identity -floop-nest-optimize -fuse-linker-plugin -pipe -Wl,-O1 -Wl,--as-needed -msse3"
CXXFLAGS="-march=native -O3 -fno-semantic-interposition -fno-plt -fgraphite-identity -floop-nest-optimize -fuse-linker-plugin -pipe -Wl,-O1 -Wl,--as-needed -msse3"
github-actions[bot] commented 4 years ago

This issue did not get any activity in the past 30 days and will be closed in 365 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

johnny-bit commented 3 years ago

@kanyck - Since quite some time dt release build uses -O3 option successfully, I wonder if you could check whether current master still exhibits LTO problems for you

kanyck commented 3 years ago

@johnny-bit At the moment my dt CFLAGS are CFLAGS="-march=native -O3 -fno-semantic-interposition -fno-plt -fgraphite-identity -floop-nest-optimize -fuse-linker-plugin -pipe -Wl,-O1 -Wl,--as-needed -msse3" I may check git version with LTO enabled if database structure hasn't changed and I'm able to roll back to stable version after that. I've already lost all my edits by jumping back and forth and don't feel like repeat this experience)))

johnny-bit commented 3 years ago

For testing run your dt like darktable --configdir /tmp as to not modify anything in terms of db/lib/config/etc

also - in case of funups the lib/db are being backed up always before schema updates so you're covered that way ;)

kanyck commented 3 years ago

Backups won't help if you did some edits then try to roll back))) Besides, I don't know what happened (maybe I managed to get some surplus/inconsistences in the db somehow), but I did experienced db damages after updates few times, like edits from one pic (partially) applied to another. Since then I'm very careful with all this)) However, loosing old edits is a good chance to redo the most interesting pics in much better way -- both dt and I do evolve... Anyway, I'm gonna do some testing probably this weekend. Will keep you posted.

johnny-bit commented 3 years ago

TBH - I use files from raw.pixls.us and CC0 licenced playraws from discuss.pixls.us as a testbench files :)

kanyck commented 3 years ago

@johnny-bit ATM build fails for me.

johnny-bit commented 3 years ago

I tried building dt using your flags here (without -fno-semantic-interposition dute to #9303) and with those pipe can crash, but i haven't seen "white squares".

Using flags from #9303 (which are tiiiiiiiiiiiny bit less agressive) I can run dt no problem and not experience the issue here.

can you confirm?

(meaning:

CFLAGS='-march=native -O3 -fipa-pta -flto=4 -fno-plt -fgraphite-identity -floop-nest-optimize -fuse-linker-plugin -pipe -Wl,-O1 -Wl,--as-needed' - crash a bit CFLAGS='-march=native -O3 -flto=3 -fno-plt -fgraphite-identity -floop-nest-optimize -fuse-linker-plugin -pipe -Wl,-O1 -Wl,--as-needed' - rock solid

kanyck commented 3 years ago

-ipa-ptais known to cause issue in dt, see https://github.com/darktable-org/darktable/issues/3680

CFLAGS="-march=native -O3 -flto=3 -fno-plt -fgraphite-identity -floop-nest-optimize -fuse-linker-plugin -pipe -Wl,-O1 -Wl,--as-needed -msse3"-- seems to be just fine. I think this issue may be closed. Thanks!