unable to add network folder to library

[sans-c]

Describe the bug

Trying to import files from a network share (samba) does not work. Nothing happens.

Steps to reproduce

1. in lighttable when I click the "add to library"-button, a window opens. So far, so good.
2. hit the +-icon in the upper left corner (places) and navigate to the mount location of the network folder.
3. select the folder and open it. It now appears under the "folders" list.
4. now click on that folder, nothing happens.

Expected behavior

I expect to see importable files and/or importable folders to show up.

Logfile | Screenshot | Screencast

darktable -d all produces output of course but nothing at all happens when I try to import. No errors, no nothing.

Commit

No response

Where did you obtain darktable from?

self compiled

darktable version

4.7.0+242~ge56db18483

What OS are you using?

Linux

What is the version of your OS?

Endeavour

Describe your system?

I am using gnome and wayland (yes, I know..) Everything is as up to date as the official arch repositories.

Are you using OpenCL GPU in darktable?

Yes

If yes, what is the GPU card and driver?

No response

Please provide additional context if applicable. You can attach files too, but might need to rename to .txt or .zip

ansel 0.0.0+729~ge2c4a0a60 (latest as of today) imports the files just fine.

The share is a samba share. I tried both, adding smb-share:server=nas,share=images as well as the mout-point in /run/user/... to darktable with the same effect. Nothing happens.

[advancingu]

Same issue here under Ubuntu 20.04 LTS with Darktable 4.6.0 and /var/run/user/.../gvfs/... used as import source folder.

After selecting files to import and clicking "add to library", I can see the import progress counter increment quickly but none of the files actually get imported.

This used to be slow but work correctly in previous versions.

[advancingu]

Probably related to https://github.com/darktable-org/darktable/pull/15894

[sans-c]

New input: The issue is not present in 4.6 so this is a regression.

What I noticed though: I tried to add the networkfolder by adding its mount-path to places. It looks something like /run/user/1000/gvfs/smb-share:server=...

I can remove places by rightclicking on them. However I cannot remove places like the one above. They are permanently stuck and rightclicking on them gives: "you can't delete the selected places". Both in 4.6 and in master.

[Marc-AntoineGelinas]

Hi, Windows 11 user here. Also DarkTable 4.6.0.

I'm having to same issue whenever I try to recursively add a collection. But I have 2 different use cases that don't work. From my local computer drive, if I try to import I simply get a message popup "x images have been imported" (not verbatim), but nothing actually gets added just like @sans-c. This is for a local drive.

When I try to import from my NAS which I mapped as an SMB share, if I try to recursively add lets say Z:/Photos/2023, which contains multiple folders, which contains individual photos, I will get the message pop up saying "error loading Z:/Photos/2023/image_X.cr3" and will increment on all the pictures. The problem here is that for some reason DarkTable completely ignores the folders under 2023/ and tries to get a picture which should be in the folder that is omitted.

Adding each folder one by one without using the recursive one. But yeah I'd really rather have the recursive which used to work before I upgrade from 4.3 I believe.

[sans-c]

Issue is still present in 4.6.1 "stable". Import is functionally broken in 2 consecutive stable releases.

[ptilopteri]

works fine for me over smb with 4.7.0+630~gb0ebf97c5 on openSUSE Tumbleweed 20240221

check to see if you have odd or special chars in path or filename ie: not basic ascii or spaces, etc

[sans-c]

works fine for me over smb with 4.7.0+630~gb0ebf97c5 on openSUSE Tumbleweed 20240221

check to see if you have odd or special chars in path or filename ie: not basic ascii or spaces, etc

So none of the issues I've desribed above are present for you? You can mount an SMB-share, you can add folders, import works, regardles of wether the recursive option is on or off and you can remove folders from the list by rightclicking?

[ptilopteri]

I observe no problems. Just loaded 204 images on smb mount drive containing 14 directories, worked a duplicate image and saved it to a new directory in that smb mount. I self compiled my dt.

fwiw: I do not recall ever having issues over smb. not that I test that with every compile. but I do frequent compiles.

[sans-c]

I observe no problems. Just loaded 204 images on smb mount drive containing 14 directories, worked a duplicate image and saved it to a new directory in that smb mount. I self compiled my dt.

fwiw: I do not recall ever having issues over smb. not that I test that with every compile. but I do frequent compiles.

tried 4.7.0+630~gb0ebf97c59 - import does not work. 4.6.1 does have the same issues. The share is accessible, DT even shows the files while adding "places". Selecting a folder does not show any files. Regardless of any other settings (recursive, new pictures, non-raw...). So yes - state of affairs is still the same for me.

[ptilopteri]

what about odd chars in file or path name? have seen comments related lately

[sans-c]

No "odd characters", whatever that might be. There are dashes and underscores, alphanumeric characters and that is it.

As for darktable -d common/all, see my first post.

darktable -d all produces output of course but nothing at all happens when I try to import. No errors, no nothing.

Most of the other output is opencl-related. I can see no errors or anything with regards to file access. If you are looking for anything specific, let me know. I'd like to refrain from posting a complete log.

[Marc-AntoineGelinas]

Hi, Windows 11 user here. Also DarkTable 4.6.0.

I'm having to same issue whenever I try to recursively add a collection. But I have 2 different use cases that don't work. From my local computer drive, if I try to import I simply get a message popup "x images have been imported" (not verbatim), but nothing actually gets added just like @sans-c. This is for a local drive.

When I try to import from my NAS which I mapped as an SMB share, if I try to recursively add lets say Z:/Photos/2023, which contains multiple folders, which contains individual photos, I will get the message pop up saying "error loading Z:/Photos/2023/image_X.cr3" and will increment on all the pictures. The problem here is that for some reason DarkTable completely ignores the folders under 2023/ and tries to get a picture which should be in the folder that is omitted.

Adding each folder one by one without using the recursive one. But yeah I'd really rather have the recursive which used to work before I upgrade from 4.3 I believe.

After seeing your updates related to the recursive import working/not working I went ahead and updated to the Nightly darktable-4.7.0+630.gb0ebf97c59-win64. I didn't try 4.6.1.

I removed a film roll I had previously imported with the hierarchy of /2024//.cr3. I then recursively tried to add everything from the parent folder /2024/ and all subfolders imported correctly. As for odd characters all I have that is special are dashes "-".

[ptilopteri]

if the log file is too large, which I doubt, post it to a file sharing service and provide the url.

[ptilopteri]

if the log contains sensitive information, alter that info before posting.

[sans-c]

This is what happens... I started darktable, tried what I described in the first post and this is the output.

darktable 4.7.0+630~gb0ebf97c59
Copyright (C) 2012-2024 Johannes Hanika and other contributors.

Compile options:
  Bit depth              -> 64 bit
  Debug                  -> DISABLED
  SSE2 optimizations     -> ENABLED
  OpenMP                 -> ENABLED
  OpenCL                 -> ENABLED
  Lua                    -> ENABLED  - API version 9.3.0
  Colord                 -> ENABLED
  gPhoto2                -> ENABLED
  GMIC                   -> ENABLED  - Compressed LUTs are supported
  GraphicsMagick         -> ENABLED
  ImageMagick            -> DISABLED
  libavif                -> ENABLED
  libheif                -> ENABLED
  libjxl                 -> ENABLED
  OpenJPEG               -> ENABLED
  OpenEXR                -> ENABLED
  WebP                   -> ENABLED

See https://www.darktable.org/resources/ for detailed documentation.
See https://github.com/darktable-org/darktable/issues/new/choose to report bugs.

     0.0003 application_directory: /usr/bin
     0.0003 darktable.datadir: /usr/share/darktable
     0.0003 darktable.plugindir: /usr/lib/darktable
     0.0004 darktable.localedir: /usr/share/locale
     0.0004 darktable.configdir: /home/redacted/.config/darktable
     0.0004 darktable.cachedir: /home/redacted/.cache/darktable
     0.0005 darktable.sharedir: /usr/share
     0.0005 darktable.tmpdir: /tmp
     0.0005 new_xdg_data_dirs: /usr/local/share/:/usr/share/
     0,0922 [dt_worker_threads] using 6 worker threads
     0,0959 [dt_get_sysresource_level] switched to 1 as `default'
     0,0959   total mem:       11690MB
     0,0959   mipmap cache:    1461MB
     0,0959   available mem:   5845MB
     0,0959   singlebuff:      91MB
     0.1304 [opencl_init] opencl library 'libOpenCL' found on your system and loaded, preference 'default path'
     0.1812 [opencl_init] found 1 platform
[opencl_init] found 1 device

[dt_opencl_device_init]
   DEVICE:                   0: 'Intel(R) HD Graphics 520'
   PLATFORM, VENDOR & ID:    Intel(R) OpenCL Graphics, Intel(R) Corporation, ID=32902
   CANONICAL NAME:           intelropenclgraphicsintelrhdgraphics520
   DRIVER VERSION:           23.48.27912
   DEVICE VERSION:           OpenCL 3.0 NEO 
   DEVICE_TYPE:              GPU, unified mem
   GLOBAL MEM SIZE:          10539 MB
   MAX MEM ALLOC:            4096 MB
   MAX IMAGE SIZE:           16384 x 16384
   MAX WORK GROUP SIZE:      256
   MAX WORK ITEM DIMENSIONS: 3
   MAX WORK ITEM SIZES:      [ 256 256 256 ]
   ASYNC PIXELPIPE:          NO
   PINNED MEMORY TRANSFER:   NO
   AVOID ATOMICS:            NO
   MICRO NAP:                250
   ROUNDUP WIDTH & HEIGHT    16x16
   CHECK EVENT HANDLES:      128
   TILING ADVANTAGE:         0.000
   DEFAULT DEVICE:           NO
   KERNEL BUILD DIRECTORY:   /usr/share/darktable/kernels
   KERNEL DIRECTORY:         /home/redacted/.cache/darktable/cached_v3_kernels_for_IntelROpenCLGraphicsIntelRHDGraphics520_234827912
   CL COMPILER OPTION:       
   CL COMPILER COMMAND:      -w   -DINTEL=1 -I"/usr/share/darktable/kernels"
   KERNEL LOADING TIME:       0.0468 sec
[opencl_init] OpenCL successfully initialized. internal numbers and names of available devices:
[opencl_init]       0   'Intel(R) OpenCL Graphics Intel(R) HD Graphics 520'
     0.2286 [opencl_init] FINALLY: opencl is AVAILABLE and ENABLED.
[opencl_init] opencl_scheduling_profile: 'default'
[opencl_init] opencl_device_priority: '*/!0,*/*/*/!0,*'
[opencl_init] opencl_mandatory_timeout: 400
[dt_opencl_update_priorities] these are your device priorities:
[dt_opencl_update_priorities]       image   preview export  thumbs  preview2
[dt_opencl_update_priorities]       0   -1  0   0   -1
[dt_opencl_update_priorities] show if opencl use is mandatory for a given pixelpipe:
[dt_opencl_update_priorities]       image   preview export  thumbs  preview2
[dt_opencl_update_priorities]       0   0   0   0   0
[opencl_synchronization_timeout] synchronization timeout set to 200
   UNIFIED MEM SIZE:         2923 MB reserved for 'intelropenclgraphicsintelrhdgraphics520'
[dt_opencl_update_priorities] these are your device priorities:
[dt_opencl_update_priorities]       image   preview export  thumbs  preview2
[dt_opencl_update_priorities]       0   -1  0   0   -1
[dt_opencl_update_priorities] show if opencl use is mandatory for a given pixelpipe:
[dt_opencl_update_priorities]       image   preview export  thumbs  preview2
[dt_opencl_update_priorities]       0   0   0   0   0
[opencl_synchronization_timeout] synchronization timeout set to 200
     0,4379 [dt_worker_threads] using 6 worker threads
     1,5072 [lib_load_module] failed to open `midi': libportmidi.so.2: cannot open shared object file: No such file or directory
    50,1497 Session fullpipe cache report. hits/run=0,00, hits/test=0,000
 [opencl_summary_statistics] device 'Intel(R) OpenCL Graphics Intel(R) HD Graphics 520' (0): NOT utilized

[sans-c]

Mind telling me, what exactly you are looking for?

To my eyes, there is nothing unusual - once again, nothing happens in the log when I am in the import dialogue...

dtlog.txt

[sans-c]

once again, nothing happens in the log when I am in the import dialogue...

So yes...same as in my initial post. Everyting still stands true.

[sans-c]

Was it between seconds 6,75 and 40,19?

Yes. More specifically it probably was between 11,7900 and 40,1956 when I "accidentally" hovered over a single photo which was displayed in the lighttable.

[sans-c]

When I go to add to library there are SQL queries

There are when adding files from a "normal" folder. Trying the network share results in what I described above and what is shown in the log.

So I was looking around a bit and there are old issues and commits. What we see here seems to be the same thing still, just 3 years later. See here for exmaple

https://github.com/darktable-org/darktable/issues/9740

https://github.com/darktable-org/darktable/pull/10937

So when you asked about "odd characters", you probably meant the mount path, not the structure of the actual networkfolder? The former does have "odd characters". Once again, see the initial post.

smb-share:server=nas,share=images ...

Note that this is standard behaviour without any modificiation on my side. Just a plain gnome installation. And as I said before, ansel has absolutely no trouble importing these folders. I don't know if changes to the import have been made there - at least I am not aware of any.

[ptilopteri]

so my smb shares are assigned link paths, ie: /home//

have you considered making such an adjustment

keep in mind that dt was developed an a linux animal and is no as such "supported' with other operating systems, ie: windows

[ptilopteri]

s/"ie: /home//"/"ie: /home//remote-box-1/"

[sans-c]

so my smb shares are assigned link paths, ie: /home//

I have no idea what that is supposed to mean, sorry.

have you considered making such an adjustment

No. I mount network shares via the standard mechanism my desktop environment/file manager provides. I am not using some alien tech but a very commonly used software.

keep in mind that dt was developed an a linux animal and is no as such "supported' with other operating systems, ie: windows

I am using Linux and merely trying to work with a samba network share. Something which ansel (a fork of DT) can do and something which DT itself was probably able to do before some bug crept in.

I appreciate the help you are trying to provide but "have you considered pulling the plug" and "have you considered some fancy workaround because a very common, standard way of doing things does not work" does not strike me as very helpful. :[

[wpferguson]

I think the problem here is a result of the way the file system is mounted. If you rely on the GUI to mount the file system it's going to mount it as a virtual file system in/var/run/user/xxxx/gvfs/..... IIRC there were issues in the past using fuse and other automagick mounts.

I create a mount point and mount my network file systems using the mount command and I haven't had any issues.

I'm not sure what the difference is programmatically accessing a mounted file system versus a virtual file system, but I think that's the issue.

[sans-c]

I think the problem here is a result of the way the file system is mounted.

That is probably the cause, yes. But the issue is, that there is probably a bug in darktable. I've yet to come across another program which cannot deal with shares mounted in this way properly. The only issue I am having right now is with darktable. And it doesn't even complain - it just does nothing. If the stance is "there is no issue, the issue is the way, the share is mounted", then DT should at least tell you what you are supposed to do and not silently fail.

As I said above, there might be "workarounds", but I hope the goal is to fix what I consider a bug. And if any more input is needed or "experiments" need to be run, let me know and I will assist if I can.

[sans-c]

can you bisect?

A build takes like 20 minutes for me and I wont have much time/need my machine for the next few days..impractical. But in case this bug will be nagging us for much longer, I might finally be able to do that.

And yes, I said that 4.6.0 doesn't have this issue. I don't think I made that up but I am not sure what was wrong there. I feel like import didn't work properly in 4.6.0, too but I clearly wrote that this issue wasn't there... Well, I think we are making progress nonetheless. :)

[sans-c]

Can you build/test before this commit and then with it? 70de82c

I just did that. Your suspicions and those of @advancingu were correct.

https://github.com/darktable-org/darktable/commit/9e3864a9e5d35d6fe6977a19a8a97ec1891c9031 works https://github.com/darktable-org/darktable/commit/70de82c422bedade2a9606ca6245a67bf4b41709 does not

4.6.0 consequently does work too. I just tried adding folders as desrcibed in the first post and checked if there are files displayed and if the filters (recursive, new files...) work. I did not actually went on to import anything. I can do some testing once the original issue is resolved.

[sans-c]

Using current master, can you mount the samba drive not using the virtual file system? To test/confirm the issue?

Yes. This works.

[TurboGit]

@sans-c : Are you sure it was working on 4.6.0?

I have mounted my Google Drive via Nautilus on GNOME and so I have a GVFS mounter drive in /var/run/user/1000/gvfs/.

Using 4.6.0 I can indeed add the location into the import dialog but I cannot see any of the jpeg in this location. And I have huge amount of error in the console:

(darktable:2945786): GLib-GIO-CRITICAL **: 10:26:00.974: file ../../../gio/gfileinfo.c: line 1633 (g_file_info_get_is_hidden): should not be reached

Same with current master but without any error in the console.

So to me it doesn't look like a regression compared to 4.6.0.

[sans-c]

@sans-c : Are you sure it was working on 4.6.0?

Yes. I tested it for the third time now.

But it is definitely buggy. Suppose I have a folder structure like images/august/01 with pictures in 01. If I add the 01-folder to places and consequently to folders, I can see all the files and they import just fine. regardless of wether recursive is checked or not.

If i add images or images/august/ to places, choose that folder and check "recursive", then I can see all the errors your mention in the debug output but I can see the files. However, when I import, DT says it is importing but complains about errors all the time. It finshes by saying that some amount of pictures were imported but they were actually not.

So yes, the import is broken in multiple ways. It is just that it is more broken in 4.6.1 because there I cannot import any pictures from the network share, no matter what I try.

While the approach in 4.6.0 can be cumbersome if you try to import multiple nested folders at once, you can at least get your pictures into the library.

Edit: Funny thing: the images on the share that were added with 4.6.0 work just fine when I open that library in master.

[TurboGit]

@sans-c : #16379 should be improving the support. This is still wip, but would be good if you could test.

[sans-c]

@sans-c : #16379 should be improving the support. This is still wip, but would be good if you could test.

I've seen some errors (very few though in comparison to the number of images imported, all clumped together and maybe even after the import was finished) in the log but the import worked with the recursive option, pretty much as expected. Let me know if you need more input. I probably wont be around for today anymore - not sure yet.

[victoryforce]

the import worked with the recursive option, pretty much as expected.

@sans-c And your original problem is solved too, did I understand correctly? Can we close this issue?

[sans-c]

This is still wip

Can we close this issue?

I guess if it is wip, we shouln't close it for good? As I said previously, my (quick and superficial) tests worked. I didn't get around to test a lot more since then.

[victoryforce]

This is still wip

Can we close this issue?

I guess if it is wip, we shouln't close it for good?

It was wip for a few hours on February 25. The wip tag was removed when the fix was merged into the darktable codebase.

As I said previously, my (quick and superficial) tests worked. I didn't get around to test a lot more since then.

OK, I'll close the issue then.