kbhomes / libmtp-zune

Adding Zune support to libmtp.
GNU Lesser General Public License v2.1
35 stars 19 forks source link

Building and Installing

See the "INSTALL" file.

Initiator and Responder

libmtp implements an MTP initiator, which means it initiate MTP sessions with devices. The devices responding are known as MTP responders. libmtp runs on something with a USB host controller interface, using libusb to access the host controller.

If you're more interested in the MTP responders, gadgets like MP3 players, mobile phones etc, look into MeeGo:s Buteo Sync: http://wiki.meego.com/Buteo - these guys are creating a fully open source MTP responder.

Heritage

libmtp is based on several ancestors:

Contacting and Contributing

See the project page at http://libmtp.sourceforge.net/ We always need your help. There is a mailinglist and a bug report system there.

People who want to discuss MTP devices in fora seem to hang out on the forums at AnythingbutiPod: http://www.anythingbutipod.com/forum/

Compiling programs for libmtp

libmtp has support for the pkg-config script by adding a libmtp.pc entry in $(prefix)/lib/pkgconfig. To compile a libmtp program, "just" write:

gcc -o foo pkg-config --cflags --libs libmtp foo.c

This also simplifies compilation using autoconf and pkg-config: just write e.g.

PKG_CHECK_MODULES(MTP, libmtp) AC_SUBST(MTP_CFLAGS) AC_SUBST(MTP_LIBS)

To have libmtp LIBS and CFLAGS defined. Needless to say, this will only work if you have pkgconfig installed on your system, but most people have nowadays.

If your library is installed in e.g. /usr/local you may have to tell this to pkgconfig by setting the PKG_CONFIG_PATH thus:

export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig

Documentation

Read the API documentation that can be generated with doxygen. It will be output in doc/html if you have Doxygen properly installed. (It will not be created unless you have Doxygen!)

For information about the Media Transfer Protocol, see: http://en.wikipedia.org/wiki/Media_Transfer_Protocol

The official 1.0 specification for MTP was released by the USB Implementers Forum in may, 2008. Prior to this, only a proprietary Microsoft version was available, and quite a few devices out there still use some aspects of the Microsoft version, which deviates from the specified standard. You can find the official specification here: http://www.usb.org/developers/devclass_docs/MTP_1.0.zip

The Examples

In the subdirectory "examples" you find a number of command-line tools, illustrating the use of libmtp in very simple terms.

Please do not complain about the usability or documentation of these examples, they look like they do for two reasons:

  1. They are examples, not tools. If they were intended for day-to-day usage by commandline freaks, I would have called them "tools" not "examples".

  2. The MTP usage paradigm is that a daemon should hook the device upon connection, and that it should be released by unplugging. GUI tools utilizing HAL (hald) and D-Bus do this much better than any commandline program ever can. (See below on bugs.) Specificationwise this is a bug, however it is present in many, many devices.

That said, if you want to pick up and maintain the examples, please volunteer.

FAQ: Common Problems

Some MTP devices have strange pecularities. We try to work around these whenever we can, sometimes we cannot work around it or we cannot test your solution.

New Devices

If you happen upon a device which libmtp claims it cannot autodetect, please submit the vendor ID and device ID (these can be obtained from the "lsusb" and "lsusb -n" commands run as root) as a bug, patch or feature request on the Sourceforge bug tracker at our homepage. If it gives a sensible output from "mtp-detect" then please attach the result as well as it teach us some stuff about your device. If you've done some additional hacking, join our mailinglist and post your experiences there.

If you want to be able to hack some more and you're not afraid of C hacking, add an entry for your device's vendor/product ID and a descriptive string to the database in the file src/music-players.h.

If you want to poke around to see if your device has some special pecularities, you can test some special device flags (defined in src/device-flags.h) by inserting them together with your device entry in src/music-players.h. Flags can be tested in isolation or catenated with "|" (binary OR). If relatives to your device use a certain flag, chances are high that a new device will need it too, typically from the same manufacturer.

The most common flag that needs to be set is the DEVICE_FLAG_UNLOAD_DRIVER that detach any Linux kernel drivers that may have attached to the device making MTP access impossible. This is however not expected to really work: this is a problem being tracked as of now (2007-08-04). See the "last resort" solutions below if you really need to get your dual-mode device to work with MTP.

Another flag which is easy to identify is the DEVICE_FLAG_NO_ZERO_READS, which remedies connection timeouts when getting files, and some timeouts on e.g. successive "mtp-connect" calls.

If your device is very problematic we are curious of how it works under Windows, so we enjoy reading USB packet sniffs that reveal the low-level traffic carried out between Windows Media Player and your device. This can be done using e.g.:

There are other USB monitors as well, some more expensive alternatives use hardware and even measure electronic characteristics of the traffic (which is far too much detail for us).

Device sniffs are an easy read since the PTP/MTP protocol is nicely structured. All commands will have a structure such as this in the log, we examplify with a object list request:

PTP REQEUST: 000120: Bulk or Interrupt Transfer (UP), 03.09.2007 12:49:25.9843750 +0.0 Pipe Handle: 0x863ce234 (Endpoint Address: 0x2) Send 0x20 bytes to the device: 20 00 00 00 01 00 05 98 23 00 00 00 27 03 00 10 ......?#...'... Length TYPE CMD Trans# Param1

00 00 00 00 02 DC 00 00 00 00 00 00 00 00 00 00 .....Ü.......... Param2 Param3 Param4 Param5

[OPTIONAL] DATA PHASE: 000121: Bulk or Interrupt Transfer (UP), 03.09.2007 12:49:26.0 +0.0156250 Pipe Handle: 0x863ce214 (Endpoint Address: 0x81) Get 0x1a bytes from the device: 1A 00 00 00 02 00 05 98 23 00 00 00 01 00 00 00 .......?#....... Length TYPE CMD Trans# DATA

27 03 00 10 02 DC 04 00 00 30 '....Ü...0

RESPONSE: 000122: Bulk or Interrupt Transfer (UP), 03.09.2007 12:49:26.0 +0.0 Pipe Handle: 0x863ce214 (Endpoint Address: 0x81) Get 0xc bytes from the device: 0C 00 00 00 03 00 01 20 23 00 00 00 ....... #... Length TYPE CODE Trans#

If you want to compare the Windows behaviour with a similar operation using libmtp you can go into the src/libusb-glue.c file and uncomment the row that reads:

//#define ENABLE_USB_BULK_DEBUG

(I.e. remove the two //.)

This will make libmtp print out a hex dump of every bulk USB transaction. The bulk transactions contain all the PTP/MTP layer data, which is usually where the problems appear.

Notes to assist with debugging new devices:

In debugging new hardware, we highly recommend that you only use the example mtp-* applications that come with libmtp, as other applications may have their own bugs that may interfere with your new device working correctly. Using another application instead of those that come with libmtp just adds another point of failure.

For debugging, there are 3 main options:

  1. Use the env variable: LIBMTP_DEBUG to increase the verboseness of the debugging output for any application using libmtp. Relevant codes are:
    • 0x00 [0000 0000] : no debug (default)
    • 0x01 [0000 0001] : PTP debug
    • 0x02 [0000 0010] : Playlist debug
    • 0x04 [0000 0100] : USB debug
    • 0x08 [0000 1000] : USB data debug // Codes are hex and binary respectively. Simple add them togther // to get your desired level of output.

(Assuming bash) eg: $ export LIBMTP_DEBUG=12 $ mtp-detect // To get USB debug and USB data debug information.

$ export LIBMTP_DEBUG=2 $ mtp-detect // To get Playlist debug information.

Also note, an application may also use the LIBMTP_debug() API function to achieve the same options as listed above.

  1. Use "strace" on the various mtp-* commands to see where/what is falling over or getting stuck at.
    • On Solaris and FreeBSD, use "truss" or "dtrace" instead on "strace".
    • On Mac OS X, use "ktrace" or "dtrace" instead of "strace".
    • On OpenBSD and NetBSD, use "ktrace" instead of "strace".

This will at least help pinpoint where the application is failing, or a device is reporting incorrect information. (This is extremely helpful with devices that have odd disconnection requirements).

The use of these tools may also pinpoint issues with libusb as implemented by each OS vendor or issues with the MTP implementation on the new device as well, so please be prepared for either case.

  1. Use "gdb" or similar debugger to step through the code as it is run. This is time consuming, and not needed just to pinpoint where the fault is.

The use of gdb or another debugger may also miss or actually cause command and data timing issues with some devices, leading to false information. So please consider this a last resort option.

Also please read the "It's Not Our Bug!" section below, as it does contain some useful information that may assist with your device.

Dual-mode devices does not work - last resort:

Some devices that are dual-mode are simply impossible to get to work under Linux because the usb-storage(.ko) kernel module hook them first, and refuse to release them, even when we specify the DEVICE_FLAG_UNLOAD_DRIVER flag. (Maybe it DOES release it but the device will immediately be probed at the USB mass storage interface AGAIN because it enumerates.)

Here is what some people do:

  1. Plug in the device.
  2. USB-mass storage folder will open automatically.
  3. Unmount the device.
  4. Run mtp-detect. It will most likely fail the first time.
  5. Run mtp-detect again, it might work this time, or fail. Keep running till it works. 99% it works by the third try.
  6. Once mtp-detect gives you an "Ok", open either Rhythmbox or Gnomad2, everything should work.

Linux: Try this, if you have a recent 2.6.x Linux kernel, run (as root) something like:

rmmod usb_storage ; mtp-detect

You can run most any command or a client like gnomad2 or Amarok immediately after the rmmod command. This works sometimes. Another way:

Now none of you USB disks, flash memory sticks etc will be working (you just disabled them all). However you can try your device, and it might have started working because there is no longer a USB mass storage driver that tries to hook onto the mass storage interface of your device.

If not even blacklisting works (check with "lsmod | grep usb-storage"), there is some problem with something else and you may need to remove or rename the file /lib/modules//kernel/drivers/usb/storage/usb-storage.ko manually.

If you find the PerfectSolution(TM) to this dilemma, so you can properly switch for individual devices whether to use it as USB mass storage or not, please tell us how you did it. We know we cannot use udev, because udev is called after-the-fact: the device is already configured for USB mass storage when udev is called.

On Mac OS there is another ugly hack:

  1. Open up a terminal window
  2. Type: sudo mv /System/Library/Extensions/IOUSBMassStorageClass.kext /System/Library/Extensions/IOUSBMassStorageClass.kext.disabled

and when prompted enter your password.

  1. Restart.

To reverse this change, just reverse the filenames:

sudo mv /System/Library/Extensions/ IOUSBMassStorageClass.kext.disabled /System/Library/Extensions/ IOUSBMassStorageClass.kext

and restart.

Calendar and contact support:

The Creative Zen series can read VCALENDAR2 (.ics) files and VCard (.vcf) files from programs like for example Evolution with the following limitations/conditions:

Syncing in with Evolution and Creative Devices

Evolution can easily export .ics an .vcf files, but you currently need some command-line hacking to get you stuff copied over in one direction host -> device. The examples/ directory contains a script created for the Creative Zen Microphoto by Nicolas Tetreault.

Lost symbols

Shared libraries can be troublesome to users not experienced with them. The following is a condensed version of a generic question that has appeared on the libmtp mailing list from time to time.

PTP: Opening session Queried Creative Zen Vision:M gnomad2: relocation error: gnomad2: undefined symbol: LIBMTP_Get_Storageinfo (...) Are these type of errors related to libmtp or something else?

The problem is of a generic nature, and related to dynamic library loading. It is colloquially known as "dependency hell". (http://en.wikipedia.org/wiki/Dependency_hell)

The gnomad2 application calls upon the dynamic linker in Linux to resolve the symbol "LIBMTP_Get_Storageinfo" or any other symbol (ELF symbol, or link point or whatever you want to call them, a symbol is a label on a memory address that the linker shall resolve from label to actual address.) For generic information on this subject see the INSTALL file and this Wikipedia page:

http://en.wikipedia.org/wiki/Library_(computing)

When Linux /lib/ld-linux.so.X is called to link the symbols compiled into gnomad2 (or any other executable using libmtp), it examines the ELF file for the libmtp.so.X file it finds first and cannot resolve the symbol "LIBMTP_Get_Storageinfo" (or whichever symbol you have a problem witj) from it, since it's probably not there. There are many possible causes of this symbol breakage:

1) You installed precompiled libmtp and gnomad2 packages (RPMs, debs whatever) that do not match up. Typical cause: your gnomad2 package was built against a newer version of libmtp than what's installed on your machine. Another typical cause: you installed a package you found on the web, somewhere, the dependency resolution system did not protest properly (as it should) or you forced it to install anyway, ignoring some warnings.

2) You compiled libmtp and/or gnomad2 from source, installing both or either in /usr/local/lib and /usr/local/bin. This means at compile-time gnomad2 finds the libmtp library in /usr/local/lib but at runtime, it depends on the Linux system wide library loader (/lib/ld-linux.so.X) in order to resolve the symbols. This loader will look into the file /etc/ld.so.conf and/or the folder /etc/ld.so.conf.d in order to find paths to libraries to be used for resolving the symbols. If you have some older version of libmtp in e.g. /usr/lib (typically installed by a package manager) it will take precedence over the new version you just installed in /usr/local/lib and the newly compiled library in /usr/local/lib will not be used, resulting in this error message.

3) You really did install the very latest versions (as of writing libmtp 0.1.5 and gnomad2 2.8.11) from source and there really is no pre-installed package of either on your machine. In that case I'm totally lost, I have no idea what's causing this.

Typical remedies:

1) If you don't want to mess around with your system and risk these situations, only use pre-packaged software that came with the distribution or its official support channels. If it still breaks, blame your distribution, they're not packaging correctly. Relying on properly packaged software and not installing things yourself is the Linux solution to the "dependency hell" problem.

2) Read about dynamically linked library handling until the stuff I wrote about in the previous list sounds like music to your ears, inspect your /lib, /usr/lib, /usr/local/lib, /etc/ld.so.conf and the /etc/ld.so.conf.d, remove all pre-packed versions using RPM, APT, YaST or whatever your distribution uses, compile libmtp and gnomad2 (or whatever) from source only and you will be enlighted.

I don't know if this helps you, it's the best answer we can give.

API is obscure - I want plain files!

PTP/MTP devices does not actually contain "files", they contain objects. These objects have file names, but that is actually just a name tag on the object.

Folders/directories aren't really such entities: they are just objects too, albeit objects that can act as parent to other objects. They are called "associations" and are created in atomic fashion and even though there is an MTP command to get all the associations of a certain object, this command is optional so it is perfectly possible (and most common, actually) to create devices where the "folders" (which are actually associations) have no idea whatsoever of what files they are associated as parents to (i.e. which files they contain). This is very easy for device manufacturers to implement, all the association (i.e. finding out which files are in a certain folder) has to be done by the MTP Initiator / host computer.

Moving a file to a new folder is for example very simple in a "real" file system. In PTP/MTP devices it is often not even possible, some devices may be able to do that, if they support command 0x1019 "Move Object", but actually the only reliable way of executing file movement is to upload the file to the host, download it with the new parent, then delete the old file. We have played with the idea of implementing this time consuming function as a fallback in case the device does not support command 0x1019, perhaps one day we will do that. (Some devices also support command 0x101a "Copy Object".)

Then the issue that in PTP/MTP it is legal for two files to have exactly the same path as long as their object IDs differ. A folder/association can contain two files with the exact same name. (And on the Creative devices this even works, too, though most devices implicitly fail at this.) Perhaps one could add some custom hook for handling that, so they become /Foo.mp3 and /Foo.mp3(1) or something similar, but it's really a bit kludgy.

Playlists and albums aren't really files, thinking about them as files like the hacks in libgphoto2 is really backwards. They are called associations and are more like a symbolic link that links in a star-shaped pattern to all the files that are part of the album/playlist. Some devices (Samsung) thought that was too complicated and have a different way of storing playlists in an UTF-16 encoded .spl-like file instead! This is why playlists/albums must have their own structs and functions.

Plain file access also assumes to be able to write files of an undetermined size, which is simply not possible in a transactional file system like PTP/MTP. (See further below.)

I Want Streaming!

Streaming reads is easy. Just connect the output file descriptor from LIBMTP_Get_File_To_File_Descriptor() (and a similar function for tracks) wherever you want.

People have connected this to TCP sockets for streaming web servers etc, works like a charm. Some devices will even survive if the callback functions return non-zero and cancel the download. Some devices will lock up and even require a reset if you do that. Devices are poorly implemented so that's life. If you want to stream off a device, the best idea is always to stream the entire file and discard the stuff at the end you don't want. It will incur a delay if you e.g. want to skip between tracks, sadly.

Then we get to the complicated things: streaming WRITES...

There is a function: LIBMTP_Send_File_From_File_Descriptor() (and similar for tracks) which will write a file to a device from a file descriptor, which may be a socket or whatever.

HOWEVER: this requires a piece of metadata with the .filesize properly set first.

This is not because we think it is funny to require that, the protocol requires it. The reason is that PTP/MTP is a transactional file system and it wants to be able to deny file transfer if the file won't fit on the device, so the transaction never even starts, it's impossible to start a transaction without giving file length.

People really want streaming so I tried a lot of hacks to see if they would work, such as setting file size to 0xffffffffU or something other unnaturally big and then aborting the file transfer when the stream ends. It doesn't work: either the device crashes or the file simply disappears since the device rolls back all failed transactions.

So this is an inherent limitation of the PTP/MTP protocol.

I want to remote control my device!

I have both good and bad news for you.

The good news is that the MTP protocol has well-defined commands to play back content on a device. Operation 0xD411 (PTP_DPC_MTP_PlaybackObject) will start playing back a file on the device (whatever that may mean if this is not a music or video file), and operation 0xD403 can set the playback volume to save your ears. Then there are operations to determine how far into the current file you currently are, so as to support say progress bars.

Since these commands have been around since the dawn of the MTP protocol and since it was developed in cooperation with Creative Technology, this is probably a requested feature from the Creative people who already had support for playback on their devices using the PDE protocol back then.

Anyway, here are the bad news: [logs]$ grep d411 * mtp-detect-trekstor-vibez.txt: 0xd411: Playback Object

Aha there is only one known device in the world which actually supports playback on the device. So either you go buy the Trekstor Vibez, or you can forget about this. You could always try asking your hardware vendor of choice to go implement this.

Since none of the core developers of libmtp has the Trekstor device, this is not yet implemented in libmtp.

I make MTP devices!

If you are a device vendor there is a lot you can do for libmtp:

Vendors do need help from libmtp too, especially we want to help vendors improve their MTP stacks, because they all suffer from the same problem: the lack of a proper conformance test has made many devices incompliant with the MTP specification as it is published today: most devices are just compliant with the Windows MTP stack, and don't work out-of-the-box with libmtp. We need someone on the inside to help in bug reporting vendors MTP stacks internally so these issues are raised. A good way to go toward better MTP compliance is to test with an alternative implementation of the stack. In e.g. IETF standardization it is compulsory for an RFC to have atleast two independent implementations for it to reach the status as standard.

Being compliant with libmtp is also more and more important for vendors: libmtp is being deployed in some embedded systems like set-top-boxes etc. It will be very irritating for customers if a device will not dock properly with some home entertainment equipment just because it is based on Linux and libmtp and not the Windows MTP stack.

Autodetect with gudev

Previously you would use HAL to detect devices being plugged in. Nowadays we use udev directly, or though the GNOME libgudev library. LIBMTPs default udev rules export the proper properties to detect any MTP device automatically, here is a verbose example derived from gnomad2:

define G_UDEV_API_IS_SUBJECT_TO_CHANGE

include <gudev/gudev.h>

const char const gudev_subsystems[] = { "usb", NULL }; GUdevClient gudev_client; guint uevent_id; guint uevent_bus_hooked = 0; guint uevent_device_hooked = 0;

static void uevent_cb(GUdevClient client, const char action, GUdevDevice device, void data) { guint64 devicenum; guint vendor; guint model; guint busnum; guint devnum; guint mtpdevice;

devicenum = (guint64) g_udev_device_get_device_number(device); g_print("%s event for %s (%"G_GINT64_MODIFIER"x)", action, g_udev_device_get_sysfs_path (device), devicenum);

/ get device info / vendor = get_property_as_int(device, "ID_VENDOR_ID", 16); model = get_property_as_int(device, "ID_MODEL_ID", 16); busnum = get_property_as_int(device, "BUSNUM", 10); devnum = get_property_as_int(device, "DEVNUM", 10); mtpdevice = get_property_as_int(device, "ID_MTP_DEVICE", 10);

if (vendor == 0 || model == 0) { g_print("couldn't get vendor or model ID for device at (%x:%x)\n", busnum, devnum); return; } else { g_print("vendor = %x, model = %x, bus = %x, device = %x\n", vendor, model, busnum, devnum); }

if (mtpdevice) { g_print("device is MTP compliant\n");

if (g_str_equal(action, "add") &&
   uevent_bus_hooked == 0 &&
   uevent_device_hooked == 0) {
  g_print(MTP device plugged in!\n");
  uevent_bus_hooked = busnum;
  uevent_device_hooked = devnum;
  scan_jukebox(NULL);
} else if (g_str_equal (action, "remove") &&
       uevent_bus_hooked == busnum &&
       uevent_device_hooked == devnum) {
  g_print("MTP device removed!\n");
  uevent_bus_hooked = 0;
  uevent_device_hooked = 0;
}

} }

(...) /*

SKETCH OF AN OVERVIEW

Draft agenda for a talk on MTP devices submitted for the Android builders summit, might come to recycle this: