search

darktable-org / darktable

darktable is an open source photography workflow application and raw developer
https://www.darktable.org
GNU General Public License v3.0
9.75k stars 1.14k forks source link

DT 2.6 macOS: no supported devices found #2431

Closed mtersch closed 5 years ago

mtersch commented 5 years ago

Describe the bug In darktable 2.6 on macOS Sierra (10.12.6) camera (Nikon D610) is not detected. Camera is detected in darktable 2.4 / gphoto2 2.5.20 / macOS Photos-app.

To Reproduce Steps to reproduce the behavior:

  1. Open darktable 2.6
  2. Attach Nikon D610 through USB, and turn on camera
  3. Message 'no supported devices found'
  4. Click 'scan for devices'
  5. Still message 'no supported devices found'

Expected behavior

  1. Open darktable 2.6
  2. Attach Nikon D610 through USB, and turn on camera
  3. Click 'scan for devices'
  4. Camera 'Nikon D610' listed, with options 'import from camera' and 'tethered shoot' (= behaviour in darktable 2.4.4) Alternatively:
  5. Attach Nikon D610 through USB, and turn on camera
  6. Open darktable 2.6
  7. Camera 'Nikon D610' listed, with options 'import from camera' and 'tethered shoot'

Platform (please complete the following information):

Additional context Bug seems to be introduced in darktable 2.6; observed correct behaviour (importing from camera, and tethered shooting) in:

mtersch commented 5 years ago

Issue seems to be macOS-specific:

parafin commented 5 years ago

I just checked with Canon 400D on macOS Mojave 10.14.4 in darktable 2.6.2 from DMG - it works out-of-the-box. On older macOS'es it was necessary to kill PTPcamera process before darktable could see the camera, which is mentioned in FAQ on DT website: https://www.darktable.org/about/faq/ So I'm not sure that there is anything to fix for us.

mtersch commented 5 years ago

Reinstalled 2.6.1 from DMG, issue remains. Checked for PTPCamera process (in Terminal: both 'ps aux | grep PTP' and 'ps aux | grep [Cc]amera' returned nothing) => proposed fix (kill PTPCamera process) can't be used. If it works on Mojave 10.14.4 then it must be some interaction between Sierra (10.12.6) and darktable 2.6; at least downgrading to 2.4.4 offers a workaround. Still curious what the interaction could be; libgphoto2 in the 2.6.1 DMG is the same as in gphoto2 through HomeBrew (which correctly identifies the camera; both are version 2.5.22). Anything I can try/check to figure this out?

parafin commented 5 years ago

Sorry, I don't have any ideas.

Jopp-gh commented 5 years ago

I reported this issue too and got the same not-working workaround. After that my report was trashed which made me really frustrated.

I'm happy to have not the newest Os as older environments (Mac 10.11) works perfectly for me. Upgrading my work station just because of some applications? Hmm, a clear no.

mtersch commented 5 years ago

Some more attempts to find information, dumping them in this reply for your reference: (spoiler alert: apparently something's wrong with libusb-1.0.0.dylib and using an older version seems to fix the issue)

Tried out 'darktable -d camctl' on commandline (then wait for darktable to start, switch on camera, and hit 'scan for devices'): 2.4.4: 4.906687 [camera_control] creating new context 0x7f8ff7ed2530 5.134980 [camera_control] loaded 2525 camera drivers. 5.238112 [camera_control] loaded 4 port drivers. 5.317850 [camera_control] 0 cameras connected 5.640104 [camera_control] registering listener 0x7f8ff6bac8b0 wait time 0.159967s 37.729863 [camera_control] loaded 5 port drivers. 37.742211 [camera_control] 1 cameras connected 38.294073 [camera_control] device Nikon DSC D610 on port usb:253,003 initialized 2.6.1: 4,773459 [camera_control] creating new context 0x7f8e28a08110 4,981019 [camera_control] loaded 2601 camera drivers. 5,115990 [camera_control] loaded 4 port drivers. 5,172801 [camera_control] 0 cameras connected 5,553998 [camera_control] registering listener 0x7f8e28a1e280 22,625656 [camera_control] loaded 5 port drivers. 22,638337 [camera_control] 1 cameras connected 22,642205 [camera_control] failed to initialize camera Nikon DSC D610 on port usb:253,003 22,642221 [camera_control] failed to initialize device Nikon DSC D610 on port usb:253,003, probably causes are: locked by another application, no access to udev etc.

'killall PTPCamera' or 'sudo killall PTPCamera' doesn't change a thing ('ps aux' doesn't show any PTPCamera process anyway).

Searching for issues with libgphoto2 on MacOS led me to https://github.com/mejedi/mac-gphoto-enabler => I will investigate this a bit more over the next few days.

Group ownership on /Applications/darktable (for 2.4.4) is 'admin', for 2.6.1 (extracted to Downloads) is 'staff' => changing to 'admin' using 'chgrp -R admin darktable\ 2.6.1' doesn't help...

Strange... darktable.app/Contents/Resources/lib/libusb-1.0.0.dylib has same version number in 2.4.4 and 2.6.1, but different size! What happens if I copy the 'old' 1.0.0 (183468 bytes) over the 'new' 1.0.0 (109952 bytes)...? Success! (Even stranger: the version of libusb-1.0.0.dylib in gphoto2 installed through HomeBrew has yet another size (85720 bytes)!)

Apparently the issue is somewhere in libusb; copying libusb from 2.4.4 into 2.6.1 works around the issue (for me).

Minor PS to johans3: please remember parafin is volunteering to do the macOS build, not contractually required to provide support or anything similar (nor am I). Try to be helpful/constructive in your comments, and accept the fact that sometimes people do not have the time and/or knowledge to fix your problem.

parafin commented 5 years ago

Could you please show the logs with both libusb versions after setting LIBUSB_DEBUG=4 in environment?

Jopp-gh commented 5 years ago

@mtersch Thanks for remembering me but I know this and didn't force anybody. My report was less detailed but relevant for all Mac users, not just for myself. I criticize just the fact that my report wasn't taken too seriously. I'm glad that the issue in discussion doesn't break the application but it's nevertheless a bug. And not a small one

A workflow gives me the opportunity to start and finish all relevant steps till my task is accomplished. In this case, importing, sorting, evaluating, retouching, exporting and what else I need to retouch endless film rolls.

parafin commented 5 years ago

The point is that I'm the only one doing any fixes for macOS port of darktable and I can't fix what I can't debug. Also there's a problem of how much time I'm willing to put into it. So if you want something fixed, you have to step up and do some actual debugging, at least investigate the root cause and produce meaningful debug logs. Maybe your original bug report shouldn't have been closed, but it wouldn't have changed anything at all if it wasn't. It could have been easily re-opened if you have produced any additional information helpful to resolve the issue. But I admit I sometimes write off macOS bug reports to particular system (mis-)configuration - macOS can be quirky. This issue doesn't seem to be that simple though since it's reproducible on more than one system.

mtersch commented 5 years ago

Seems like we're getting somewhere...

The log files are in a GitHub repository at https://github.com/mtersch/dt261-issue-no-device.git I went through the '261libusbfrom244' and 261libusbfrom261' error logs, and noticed the following line in the 'from261' version that wasn't in the 'from244' version: libusb: error [darwin_claim_interface] QueryInterface: unknown error (0x80000004)

Some searching online (libusb darwin_claim_interface QueryInterface error 0x80000004) leads me to this post on the Apple Developer Forums: https://forums.developer.apple.com/thread/110920 Which in turn points to this post: https://forums.developer.apple.com/thread/107369

Apparently some interface changed in macOS 10.14, and you would have to explicitly use an older version of this interface for macOS 10.13 and before.

parafin commented 5 years ago

That's strange since libusb is careful about using the correct version of interfaces: https://github.com/libusb/libusb/blob/v1.0.22/libusb/os/darwin_usb.h Maybe the issue is the reverse - 10.12 doesn't support some older interface version. Could you please try the 2 attached libusb libraries? libusb.zip

mtersch commented 5 years ago

See the new logs in my repo; both are working properly.

PS: used a smartphone (Moto G5+) in PTP mode, camera battery is in the charger right now.

parafin commented 5 years ago

Strange, because 10.7 variant should be the same as in 2.6.2 DMG, I just rebuilt libusb with the same settings. Not sure what happened here, but if it indeed works, then next dt release will have this issue solved. But please re-test with Nikon camera, maybe device makes a difference.

mtersch commented 5 years ago

Did a fresh download of 2.6.2 DMG (from https://github.com/darktable-org/darktable/releases/download/release-2.6.2/darktable-2.6.2.dmg as mentioned on the 'install' page); libusb-1.0.0.dylib in that version is same size (109952 bytes) as in 2.6.1 DMG, but 'diff' tells me that there's something different between these files (is build date or something similar included in .dylib file? using 'strings' to extract all text and then doing 'diff' shows no differences).

What's more interesting: the libusb-1.0.0.dylib-10.7 you provided is smaller (90292 bytes) than the 'stock' version in the 2.6.2 DMG. This means at least the files themselves are different.

When I test 2.6.2 (from DMG, using Nikon D610) with the 'stock' libusb-1.0.0.dylib it doesn't work (dt262.stdout/stderr.log in my repo, still 'QueryInterface: unknown error'); when I replace the 'stock' libusb-1.0.0.dylib with your '-10.7' version it does work (dt262libusb107.stdout/stderr.log).

Looks like the DMG builds are done (accidentally?) using different settings.

For now we have a workaround: use libusb from darktable 2.4.4 or the 10.7 version in the ZIP you provided. Let me know when you have a fresh build, so I can test it on 10.12 Sierra.

mtersch commented 5 years ago

For the technically challenged visitors a step-by-step explanation of the workaround:

  1. Download darktable for macOS from the 'install' page (https://www.darktable.org/install/#macos)
  2. Install darktable the usual way by opening the DMG file, and dragging the darktable icon onto the Applications icon 2b. Open darktable, to get rid of the 'this app was downloaded from the web' message, then close it again
  3. Download @parafin's ZIP file with 2 versions of libusb (https://github.com/darktable-org/darktable/files/3111429/libusb.zip), open it, and drag the file 'libusb-1.0.0.dylib-10.7' to your Desktop or Downloads folder
  4. Rename the 'libusb-1.0.0.dylib-10.7' file to 'libusb-1.0.0.dylib' (without the '-10.7' part)
  5. Use Finder to open the Applications folder, right-click or Ctrl-click the darktable icon, and select 'Show Package Contents'
  6. Navigate to 'Contents' -> 'Resources' -> 'lib'
  7. Copy the file you renamed to 'libusb-1.0.0.dylib' into this folder (replacing the version that's already there)

Now darktable 2.6.2 should be able to detect your camera on macOS 10.12 Sierra (and possibly other versions before 10.14 Mojave).

(If you prefer the commandline, you could just unzip 'libusb.zip' in Downloads, and do a 'cp ~/Downloads/libusb/libusb-1.0.0.dylib-10.7 /Applications/darktable.app/Contents/Resources/lib/libusb-1.0.0.dylib'.)

parafin commented 5 years ago

I won't rebuild DMG of old releases, so you will have to wait for next release (I don't know when it's going to be). In the meantime I will investigate how this happened, so let's keep this issue open for now.

Jopp-gh commented 5 years ago

I confirm. My camera is detected if I rename and use "libusb-1.0.0.dylib-10.9" and kill the "PTPCamera" Daemon. I made also a screenshot, and between clicking "find device" and getting the dt-window to import my pictures from my camera memo card elapse just a few seconds. Buuuut....

Now I was curious to see if I could repeat this behavior, and quit Dt.

After re-launching Dt, switching my eos d600 on, killing the "PTPCamera" Daemon and pressing the button "check for devices" in Dt, all I got was a twirling beach-ball and Dt hung for around 5 minutes or more. I've to add that with another "usblib", dt tends to hang also without checking for new devices.

The default "libusb" shipped with version 2.6.2 didn't hang dt but does nothing at all :(

mtersch commented 5 years ago

@johans3 Hmmm... just when we thought we got it figured out... :-( You could use the commandline option for logging camera detection info (open Terminal, and run '/Applications/darktable.app/Contents/MacOS/darktable -d camctl'; add ' > stdout.log 2> stderr.log' at the end to write the messages to the files 'stdout.log' (informational messages) and 'stderr.log' (error messages)). For a lot more information you could run 'export LIBUSB_DEBUG=4' first to switch on debugging for libusb, and then run the previous command. (You definitely want to make it '/Applications/darktable.app/Contents/MacOS/darktable -d camctl > stdout.log 2> stderr.log' to have some logfiles; maybe @parafin has time to look at them?)

@parafin I've got a decent workaround now, and maybe can help @johans3 a bit in debugging his setup (25 years of Linux experience should help a bit on the cmdline :-) ); just drop a note in this discussion when 2.6.3 has arrived and then I'll test it with Sierra. Thanks for all the help so far!

Jopp-gh commented 5 years ago

"libusb-1.0.0.dylib-10.9" renamed to "libusb-1.0.0.dylib"works sometimes (for the first 4 times only - [crazy, crazy])on Mac 10.11, IF.. I follow this steps:

1) plug in and switch on my camera 2) kill the "PTPCamera" Daemon 3) launch Darktable - my camera is already available on startup under the label "import"

If I kill the "PTPCamera" Daemon meanwhile Dt is already running, dt hangs and after awhile the process checking for connected cameras gives up and turns off for the current session.

This is the error log (process:2762): GLib-GObject-CRITICAL **: 21:49:30.027: g_object_set: assertion 'G_IS_OBJECT (object)' failed

(darktable-bin:2762): GLib-GObject-WARNING **: 21:49:30.420: invalid cast from 'GtkMenuBar' to 'GtkWindow'

(darktable-bin:2762): Gtk-CRITICAL **: 21:49:30.420: gtk_window_add_accel_group: assertion 'GTK_IS_WINDOW (window)' failed

(darktable-bin:2762): GLib-GObject-CRITICAL **: 21:50:36.629: g_object_unref: assertion 'G_IS_OBJECT (object)' failed

This is the Info log

2,291418 [camera_control] creating new context 0x7ffb5141e590 2,336657 [camera_control] loaded 2601 camera drivers. 2,441040 [camera_control] loaded 5 port drivers. 2,461630 [camera_control] 1 cameras connected 2,534548 [camera_control] registering listener 0x7ffb4b7ab7a0 6,970382 [camera_control] gphoto2 error: PTP Timeout 7,002034 [camera_control] failed to initialize camera Canon EOS 600D on port usb:253,008 7,002048 [camera_control] failed to initialize device Canon EOS 600D on port usb:253,008, probably causes are: locked by another application, no access to udev etc. 7,011705 [camera_control] loaded 5 port drivers. 7,024992 [camera_control] 1 cameras connected 11,533566 [camera_control] gphoto2 error: PTP Timeout 11,565038 [camera_control] failed to initialize camera Canon EOS 600D on port usb:253,008 11,565074 [camera_control] failed to initialize device Canon EOS 600D on port usb:253,008, probably causes are: locked by another application, no access to udev etc. 11,581276 [camera_control] loaded 5 port drivers. 11,595432 [camera_control] 1 cameras connected 16,103613 [camera_control] gphoto2 error: PTP Timeout 16,134906 [camera_control] failed to initialize camera Canon EOS 600D on port usb:253,008 16,134943 [camera_control] failed to initialize device Canon EOS 600D on port usb:253,008, probably causes are: locked by another application, no access to udev etc. 16,261886 [camera_control] loaded 5 port drivers. 16,280357 [camera_control] 1 cameras connected 20,800679 [camera_control] gphoto2 error: PTP Timeout 20,832208 [camera_control] failed to initialize camera Canon EOS 600D on port usb:253,008 20,832242 [camera_control] failed to initialize device Canon EOS 600D on port usb:253,008, probably causes are: locked by another application, no access to udev etc. 21,109617 [camera_control] loaded 5 port drivers. 21,123226 [camera_control] 1 cameras connected 25,630507 [camera_control] gphoto2 error: PTP Timeout 25,661889 [camera_control] failed to initialize camera Canon EOS 600D on port usb:253,008 25,661906 [camera_control] failed to initialize device Canon EOS 600D on port usb:253,008, probably causes are: locked by another application, no access to udev etc. 25,845551 [camera_control] loaded 5 port drivers. 25,865469 [camera_control] 1 cameras connected 30,375161 [camera_control] gphoto2 error: PTP Timeout 30,408005 [camera_control] failed to initialize camera Canon EOS 600D on port usb:253,008 30,408038 [camera_control] failed to initialize device Canon EOS 600D on port usb:253,008, probably causes are: locked by another application, no access to udev etc. 30,583745 [camera_control] loaded 5 port drivers. 30,602247 [camera_control] 1 cameras connected 35,109933 [camera_control] gphoto2 error: PTP Timeout 35,141746 [camera_control] failed to initialize camera Canon EOS 600D on port usb:253,008 35,141789 [camera_control] failed to initialize device Canon EOS 600D on port usb:253,008, probably causes are: locked by another application, no access to udev etc. 35,692347 [camera_control] loaded 5 port drivers. 35,707478 [camera_control] 1 cameras connected 40,215827 [camera_control] gphoto2 error: PTP Timeout 40,263769 [camera_control] failed to initialize camera Canon EOS 600D on port usb:253,008 40,263804 [camera_control] failed to initialize device Canon EOS 600D on port usb:253,008, probably causes are: locked by another application, no access to udev etc. 40,278256 [camera_control] loaded 5 port drivers. 40,291932 [camera_control] 1 cameras connected 44,799626 [camera_control] gphoto2 error: PTP Timeout 44,831141 [camera_control] failed to initialize camera Canon EOS 600D on port usb:253,008 44,831182 [camera_control] failed to initialize device Canon EOS 600D on port usb:253,008, probably causes are: locked by another application, no access to udev etc. 44,858479 [camera_control] loaded 5 port drivers. 44,874185 [camera_control] 1 cameras connected 49,382267 [camera_control] gphoto2 error: PTP Timeout 49,413906 [camera_control] failed to initialize camera Canon EOS 600D on port usb:253,008 49,413920 [camera_control] failed to initialize device Canon EOS 600D on port usb:253,008, probably causes are: locked by another application, no access to udev etc. 49,491955 [camera_control] unregistering listener 0x7ffb4b7ab7a0 <\code>

Jopp-gh commented 5 years ago

@mtersch

That helped. Does 'export LIBUSB_DEBUG=4' need yet another command to set the debugging mode off again?

mtersch commented 5 years ago

@johans3 The setting 'LIBUSB_DEBUG=4' will stay active until you close Terminal, then it is automatically unset again. (This also means you will have to set it again the next time you want to turn on debugging information.) By the way, all this setting does is turning on debugging information; it does not change the behaviour of darktable and/or libusb. And looking at your logs it seems that libusb is correctly detecting the camera in your case, but can't access it because PTPCamera has locked it ('1 cameras connected' / 'error: PTP Timeout' / 'failed to initialize device Canon EOS 600D on port usb:253,008, probably causes are: locked by another application, no access to udev etc.').

Apparently the PTPCamera process is controlled from within the 'Image Capture' application on macOS; according to Image Capture Help - topic 'Transfer images', section 'Automatically transfer items from your device to your computer whenever they’re connected' - you can enable/disable automatic image transfer (and thereby the PTPCamera daemon) by selecting the connected camera in Image Capture, folding out the 'Connecting this camera opens' section in the bottom left corner, and selecting 'No application' or one of the listed applications. On my MacBook setting this to 'No application' means PTPCamera is only started when I manually start Image Capture or the Photos app. (Too bad that selecting 'darktable' for this option doesn't work...)

You could apply a little hack(!!) to kill PTPCamera each time you start up darktable - edit the startup script to include the 'killall PTPCamera' command:

  1. Use Finder to open the Applications folder, right-click or Ctrl-click the darktable icon, and select 'Show Package Contents'
  2. Navigate to 'Contents' -> 'MacOS'
  3. Open the 'darktable' file with a text editor
  4. On line 18 (just before the line 'if test -n "$GTK_DEBUG_LAUNCHER"; then') add a new line containing 'killall PTPCamera' (without the single quotes)
  5. Save the 'darktable' file

Now as part of starting up the darktable executable file this should automatically kill the PTPCamera daemon, and your camera should be properly detected. Perhaps you could check the following cases:

  • first switching on the camera, then starting darktable
  • first starting up darktable, then switching on the camera (and selecting 'scan for devices')

Disclaimer: this is a bit of a hack, and properly solving the problem will probably involve rewriting parts of darktable to use the PTPCamera daemon instead of libgphoto2/libusb, or changing the 'Connecting this camera opens:' setting in Image Capture.

By the way, maybe we should move the second problem (darktable hanging up when killing PTPCamera after starting up darktable) into it's own issue?

mtersch commented 5 years ago

Minor update on 'too bad selecting darktable doesnt work'... apparently I was messing about with v2.4 and v2.6 libraries which caused darktable to crash; cleaning up the old library ($HOME/.config/darktable/) solved this. This means that in Image Capture you can select darktable as the app to open when your camera is connected; the next time you connect your camera, darktable will automatically open (and should detect your camera).

Jopp-gh commented 5 years ago

Even if there's no default photo application specified in "image capture" at launch, even if "image capture" isn't running when I connect my camera and even if I kill "ptpcamera" using a launch Deamon which runs all the time before to start Dt.. Dt continues to hang after the second and successive times I switch on my camera.

That's crazy behaviour. Anyway I'd welcome if the developer of Dt would integrate the "killall PTPCamera" and other shell commands in their code for Mac machines if we press the button "checking for new cameras/devices" and at startup of Dt too. Why not using PTPCamera as universal bridge in first place? I guess PTPCamera is available just on Mac.. :(

parafin commented 5 years ago

I don't think automatically killing a process is a good idea, what if user is currently using some other mentioned Mac app to transfer photos at that time? As for why not use PTPCamera - that's a question for gphoto developers and you already guessed the most likely answer - it's Mac-specific, while gphoto is cross-platform with focus on Linux.

Jopp-gh commented 5 years ago

To give it a context let's execute killing when: 1 at startup of Dt (if we have Dt why using other software in the same moment) 2) when we press the button "check for devices" ditto:if we check for new cameras in Dt why should we need another software to download pictures in the same moment?)

Depends about the context

Jopp-gh commented 5 years ago

@mtersch Thanks for the wealth of info. I had tried most of this, except modifying the startup setup of Dt, ("darktable" in darktable.app/Contents/MacOs/) adding commands like this: killall PTPCamera sleep 6 But I put this code after the $GTK_DEBUG_LAUNCHER Else it didn't work.

Same result. The very first time Dt starts (after log in/screen sleep)with my cam already mounted. If I quit Dt, switch my cam off, and switch my cam on again, start Dt..

Then dt hangs up till I switch my cam off again. Dt's cache must store some values which interfere with a successful mount after the first launch, IMO

Jopp-gh commented 5 years ago

udev udev is a generic device manager running as a daemon on a Linux system (also Mac) and listening (via a netlink socket) to uevents the kernel sends out if a new device is initialized or a device is removed from the system. From:

https://en.wikipedia.org/wiki/Udev

This article talks about udev, and how to add rules to this device manager:

http://reactivated.net/writing_udev_rules.html

parafin commented 5 years ago

Closing this issue. Next release should have correct libusb. I won't be adding PTPCamera killing anywhere, it's up to a user to implement workarounds. Or maybe try convincing gphoto people to do it in their library;)) But I doubt they think it's a good idea either.

mtersch commented 4 years ago

Reopening one last time - to confirm that issue has been resolved in 2.6.3.1 (2.6.3+1~gd57792b17) as downloaded 2019-11-23 from https://github.com/darktable-org/darktable/releases/download/release-2.6.3/darktable-2.6.3.1.dmg (link available on https://www.darktable.org/install/#macos).

Properly working on MacBook Pro 13" (late 2011, MacOS Sierra 10.12.6) connected to Nikon D610. Scenarios tested: 1) start up Darktable, connect & switch on camera, and click 'scan for devices' => camera shows up correctly in Darktable. 2) connect & switch on camera, start up Darktable => camera shows up correctly in Darktable.

Thanks for all the effort @parafin & @johans3 put into this; let's put this one to sleep forever. :-)

kurdybacha commented 4 years ago

It seems the issue is back on 10.15.6 :( killall PTPCamera and restarting Darktable solves it temporarily.

ldmbouge commented 6 months ago

April 2024.... And I have this exact issue on Sonoma with the same Nikon D610 model. Killing ptpcamerad does not help as the process auto-restarts right away.

zisoft commented 6 months ago

@ldmbouge: This issue is five years old, please open a new issue with a detailed description.